Sentiment Analysis with Core ML on iOS 11

Introduction

What’s Machine Learning, a term that’s pretty hyped at the moment? Machine Learning allows computers to learn and make decisions without being explicitly programmed how to do that. This is accomplieshed by algorithms that iteratively learn from the data provided. It’s a very complex topic and an exciting field for researchers, data scientists and academia. However, lately, it’s starting to be also a must know skill for good tech people in general. Regular users expect from their apps to be smarter, to learn from their previous decisions and give reccomendations for their future actions. For example, when we are listening songs in Youtube generated playlists, we expect the next song to be tailored to our musical taste. We expect Google to filter out and not bother us with all the spam emails. We expect Siri to know what we exactly mean with our spoken phrases. Machine learning is all the magic behind, that makes this work.

That’s why we, as software engineers, must be aware of the capabilities of machine learning and how it might improve our applications. Apple is also expecting us to catch up with these technologies, by announcing Core ML. Core ML is a brand new framework from Apple that enables integration of already trained learning models into the iOS apps. Developers can use trained models from popular deep learning frameworks, like Caffe, Keras, SKLearn, LibSVM and XgBoost. Using coremltools, provided by Apple, you can convert trained models from the frameworks above to iOS Core ML model, that can be easily integrated in the app. Then the predictions happen on the device, using both the GPU and CPU (depending on what’s more appropriate at the moment). This means, you don’t need internet connection and using an external web service (such as api.ai for example), to provide intelligence to your apps. Also, the predictions are pretty fast. So, it’s a pretty powerful framework, but with lot of restrictions, as we will see below.


Sentiment Analysis for Movie Reviews

In this post, we will be building an app that will clasify movie reviews to positive and negative. Users will be able to add movies and provide reviews for them. The app will automatically group them based on already trained data sat. The sub-field of Artificial Intelligence that does this is called Sentiment Analysis. Sentiment Analysis is the process of computationally identifying and categorizing opinions expressed in a piece of text, especially in order to determine whether the writer’s attitude towards a particular topic is positive, negative, or neutral.
Finding an appropriate trained model to convert to Core ML can be tricky. Apple’s coremltools are still in <1 version, which means they are still incomplete and can’t support a lot of trained models. After trying out few Caffe and Keras models, I’ve finally found something that works in this great post. The model here is trained with Tf-Idf weighted word count extraction, described in a previous post. The training and testing is done with the SKLearn framework. The resulting model is Linear Support Vector machine (LinearSVM), which is trained with Tf-idf vectorized data set. The model from this post was also failing to be converted, so with some inspiration from another excellent pioneering post on Core ML, I’ve managed to modify the scripts and get something that Core ML understands. It was definitely the most time consuming part of this post. Here’s the resulting script:

def make_Corpus(root_dir):
    polarity_dirs = [os.path.join(root_dir,f) for f in os.listdir(root_dir)]
    corpus = []    

    for polarity_dir in polarity_dirs:
        sentiment = 'bad' if polarity_dir == 'txt_sentoken/neg' else 'good'
        reviews = [os.path.join(polarity_dir,f) for f in os.listdir(polarity_dir)]
        for review in reviews:
            reviewInfo = [sentiment]
            doc_string = "";
            with open(review) as rev:
                for line in rev:
                    doc_string = doc_string + line
            reviewInfo.append(doc_string)
            if not corpus:
                corpus = [reviewInfo]
            else:
                corpus.append(reviewInfo)
    return corpus

root_dir = 'txt_sentoken'
corpus = make_Corpus(root_dir)
corpus = np.array(corpus)
X = corpus[:, 1]
y = corpus[:, 0]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=22)

vectorizer = TfidfVectorizer()
vectorized = vectorizer.fit_transform(X)
words = open('words_ordered.txt', 'w')
for feature in vectorizer.get_feature_names():
    words.write(feature.encode('utf-8') + '\n')
words.close()
model = LinearSVC()
model.fit(vectorized, y)
coreml_model = sklearn.convert(model)
coreml_model.save('MovieReviews.mlmodel')

First, you need the movie reviews polarity dataset, which can be downloaded from here. Then, we go through both the positive and negative reviews directories and we append the sentiment as a first element in an array and the review as a second. We create a training and testing set with the function train_test_split, which splits arrays or matrices into random train and test subsets. Afterwards, we are creating TfidfVectorizer, which converts a collection of raw documents to a matrix of TF-IDF features. We are also creating a new text file words_ordered.txt with all the words of the vectorizer (we will need this later). This vectorized data is used to create the LinearSVC model, which is then converted to the Core ML format, using sklearn converter from coremltools.
If you open this model in Xcode, you will see some basic information about the model, the Swift generated code that you can use in your app and the input and output parameters.

1.png
The input format defines what the model expects to receive in order to provide prediction based on the training set. We might have expected a string here, the logical thing would be: I sent a new review text to the model and it returns me whether the sentiment is good or bad. However, here we have MLMultiArray, which is multi-dimensional array used as an input to most Core ML models (you can also send images to the models, but that’s not applicable to our app). This array has a dimension of 39659. How do we create such an input?
If you examine the movie review dataset and the words_ordered.txt file, you will see that it actually has 39659 words. It is trained and tested with Tf-Idf vectorizer, so what we need here is to compute the tf-idf weight factor for every word of the review the user has entered and put it at the exact place in the MLMultiArray as it is ordered in the words_ordered.txt file. All the other entries in the multi array (words that don’t appear in the review) will be zeros, so they don’t influence the result.
But now we have another big problem. We need to compute tf-idf, which requires word countings of all the occurencies of the words in the review in all other 1000 positive and 1000 negative training reviews. If we do this every time the user types a review, we will put our users to sleep with our slowness and inefficiency. What we need is to pre-compute the word occurencies and the index in the word ordering for every word and put them in a dictionary, so they can be accessed in constant time, whenever they are needed. Pre-computing will speed up the process a lot, and the resulting output would be something like this (words.json):

{
     "movie" : { "index" : 123, "count" : 50 },
     ...
}

In order to do this, we will do some more python scripting. We will go through all the text files in the positive and negative datasets and count each word in every file. We will also go through the ordered words array in words_ordered.txt, to get the index of every word in the multi dimensional array:

import os
import re
import json
import sys
sys.stdout=open('words.json','w')
from collections import Counter
from glob import iglob

wordsRaw = open('words_ordered.txt', 'r')
words_array = []
for line in wordsRaw:
	words_array.append(line.rstrip())

frequency = {}

def removegarbage(text):
    text=re.sub(r'\W+',' ',text)
    text=text.lower()
    return text

folderpaths=['txt_sentoken/pos/', 'txt_sentoken/neg/']
counter=Counter()

for folderpath in folderpaths:
	for filepath in iglob(os.path.join(folderpath,'*.txt')):
    		with open(filepath,'r') as filehandle:
        		counter.update(removegarbage(filehandle.read()).split())

for word,count in counter.most_common():
	frequency[word] = count

result = {}

index = 0
for word in words_array:
	info = {}
	info["count"] = frequency[word]
	info["index"] = index
	result[word] = info
	index += 1

print(json.dumps(result))

That’s everything we need to get started with coding the iOS app, which is the simpler part in this case. We will store the movies locally, in UserDefaults. For our app, we will only need the title of the movie, as well as the positive and negative reviews for the movie. The MovieManager class takes care of this, it provides methods for adding/listing movies, as well as adding and retrieving reviews for a particular movie. Here’s a snippet of its most relevant methods (pretty standard stuff):

2.png
The most interesting part here is the addReview(toMovieTitle movieTitle: String, review: String, sentiment: ReviewSentiment) method. How do we determine the sentiment here? The sentiment(forReview review: String) method does that. It receives a string input, entered by the user, calls the convert method (which we will see see later) and then sends the newly created multi-dimensional array to the MovieReviews Core ML model. The model tries to make a prediction and if it fails, we will be good and assume it’s a positive review. If the prediction is successful, we check which polarity has bigger class probability and use that as a sentiment:

3.png
The convert method takes the user review and the word countings which we loaded from the words.json as an input and returns the MLMultiArray with tf-idf weight factors. We do this by creating MLMultiArray and filling everything with zeros. Then, we get the words from the sentence, by removing the punctuation characters and whitespaces. You can also do this with NSLinguisticTagger, see the previous post for more details. Then we go through the separated words and we are trying to get the word count and index from the pre-computed dictionary wordCountings. We use this information to compute the tf-idf factor and update the multi-array index with the new value.

4.png
If we now try the app, add any movie (let’s say Harry Potter) and open the movie details, we can start adding reviews. Let’s first try with few positive ones, like “Unique and amazing, one of the best movies ever”. You will see that our model will clasify it in the positive section, along with other similar reviews, like “Excellent movie, I really enjoyed watching it.”, which is what we’ve expected.

5.png

Let’s now add some negative reviews, like “This movie sucks, it’s weird and boring”. The model will correctly classify this as a negative review:

6.png
Of course, it doesn’t always work perfect. Machine learning and training a model is a really hard to get right and the margin of error is always present. You can try with different examples, and you will see that it might produce a wrong sentiment.
This leads us to the biggest problem with Core ML, the models can’t be trained additionaly, when the user starts using the app. For example, provided a review is classified in the wrong polarity, if the user could provide input that this is the wrong answer, the Core ML model won’t be able to learn this and fix it for future similar requests. Core ML only makes predictions on previously trained models, it’s not a machine learning framework itself. Let’s hope that this will enabled in future versions. For other Core ML cons, please check this good summary.

Simulator Screen Shot 10 Jul 2017, 23.52.22.png
In any case, Core ML provides iOS software engineers a great tool to get started with Machine Learning. The framework is still in an early phase and it will improve a lot, along with our know-how of it and machine learning in general. As usual, you can find the complete source code for this post on my GitHub account.

Advertisements

2 thoughts on “Sentiment Analysis with Core ML on iOS 11

    1. Yes, you are right, that’s currently a big problem with Core ML. For text analysis like this one, the trained model is not that big, but models for image classification, especially the ones Apple provided, are quite big (over 200 MB).

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s