Text recognition using Vision and Core ML

We will use a convolutional neural network model, which is trained with a lot of grayscale images, with different fonts. The accuracy of the model with a testing set is approximately 86-90%. The model is converted to Core ML using Apple’s coremltools. Let’s include the model in the iOS application.

You can see some information about the model, like the name, type, size, author, description and license. In the model evaluation parameters section, you can see what the model expects as an input and what it returns as an output. In our case, it expect 28×28 grayscale image. It returns a dictionary of the probabilities (each letter or number as key and the probability that the character is the value we are looking for). It also return the Class, which is the alphanumeric character with the highest probability.

When we integrate the model in Xcode, an automatically generated Swift model class is available. We can use the class to load the model and make predictions.

Next, let’s implement the @IBAction of the button for picking image. Here, we will display an action sheet, with two options – picking an image from the photo album or directly taking it from the camera.

When the user picks an image, the imagePickerController(_ picker:, didFinishPickingMediaWithInfo info:) delegate method is called, with the selected image. Here, we will need to first check if the image has the proper orientation (up). Afterwards, we are displaying the image in the image view and we start the text detection.

Before we take a look at the detectText(image:) implementation, let’s define the Forward Pipe operator. We will need to process the image with few filters and since these are all pure functions, we can use this operator to make our code more readable.

Now we are ready to look at the detectText(image:) method.

First, we are adjusting the colors of the image and we are converting it to a grayscale image. After that, we are creating a VNImageRequestHandler with the converted image. This class is used to perform requests to the Vision framework. Next, we create a request for detecting text as rectangles, with the VNDetectTextRectanglesRequest. An important note here is to set the reportCharacterBoxes property of the request to true, we will explain soon why we need the characterBoxes. In the completion handler, after the standard error and empty results checking, we go through the found results. These results now are detected, cropped images, consisting of one section of letters or numbers each. For example, all letters in one row will be in one result object. They are objects of type VNTextObservation, which describes a text area detected by the request we have created before.

To get each letter of the detected area, we will use the characterBoxes property.  These are the bounding boxes of the individual characters. Now that we know the position of every letter in the image, we can crop the original image to smaller images consisting of one letter. Before we send the image to Core ML for classification, we need to pre-process it first. Based on the training data, we need to insert some padding for better classification. Also, we have to resize the image to the expected 28×28 size and convert it to grayscale. The preProcess(image:) method does that. While iterating through the recognized areas, we keep track of the numberOfWords and numberOfCharacters values, since those will be needed for keeping the order of the images.

Next, let’s see the method for classifying the images. It expects an image that needs to be classified, the number of the word it is contained in, as well as the number of the character in the word.

In the method, we are creating VNCoreMLRequest with the Core ML model we have created at the beginning. We are executing the request with the VNImageRequestHandler. In the callback of the request, we are receiving results (if any), of type VNClassificationObservation. We are interested in the top result. However, you can use the other results as well, if you want to provide the user option to select another one that potentially might be the one they are looking for. Next, we are creating a dictionary with the wordNumber, characterNumber and the classified result. We are sending this dictionary to the handleResult() method.

The handleResult method first checks if the required values are in the provided dictionary. Then, we are using a dictionary called textMetadata, which stores all the found words. The keys to this data structure are the numbers of the words, and as values we have another dictionary, with all the characters and their positions in that word. Since the classification tasks are running in background threads, we are locking this chunk of code with the objc_sync_enter method, to avoid any concurrency issues. After we are done, we are taking the main thread, to remove the activity indicator and to display the text.

This method goes through the contents of the textMetadata and forms the words, based on the sorted word and character numbers.

That’s everything we need to do in order to have a basic text recognition app. If you run the app and select an image, you will see that the results are accurate in general, although there might be an error in some characters.

One problem currently is that Vision does not detect spaces so well. For example, in the following examples, you can see that it handles everything as one word.

However, if the words are in a separate line, they will be split correctly. There is also room for improvement of the Core ML model. Frequently, it mixes R with B (and other similar letters).

Core ML impressions

The biggest problem with Core ML, is that the models can’t be trained additionally, after the user starts using the app. For example, provided a character is classified as wrong letter, if the user could provide that as an input, the Core ML model won’t be able to learn that and fix it for future similar requests. Core ML only makes predictions on previously trained models, it’s not a machine learning framework itself. Let’s hope that this will enabled in future versions.  One workaround would be to ask the user whether a prediction was correct and send that answer to your backend. When you have enough such corrections, you can re-train your model and submit an app update. Not that elegant, but it can be tried out until there is better way.

That also prevents us from providing a customised user experience. If we want to learn the preferences of the users and based on that give recommendations (like for example in a music app), that’s currently not possible with Core ML.

Another issue with Core ML is the size, the more test data you add, the bigger the size of the model. No one would install an app bigger than 100 – 200 MBs.

In any case, Core ML provides iOS software engineers a great tool to get started with Machine Learning. The main role of Core ML currently is to bridge the gap between the academia (that does the process of researching, designing algorithms and training datasets) and the developers (that don’t have much machine learning expertise, but know how to bring production-ready apps to the real world). The framework is still in an early phase and it will improve a lot, along with our know-how of it and machine learning in general.

Source code

You can check the full source code here.  You can find more details about the image filtering methods there, since we haven’t covered them in the post.