Introduction

This year, WWDC was full of new and exciting features. The biggest one was probably SwiftUI, Apple’s new UI framework. However, there were a lot of other cool announcements, especially in the machine learning and augmented reality areas. Text recognition is now supported directly in the Vision framework. Previously, you would have to create your own machine learning model and combine it with the text detection feature from Vision. To see how things worked before iOS 13, please check my post Text recognition using Vision and Core ML.

In this post, we build a brand new text recognizer, using the new feature from Vision. We will do this in SwiftUI, which is (as you will see) very fun to use.

Implementation

To follow along, you will need Xcode 11 and iOS 13 (beta at the time of writing). To get started, create a new project called TextRecognizerSwiftUI. Make sure to have “Use SwiftUI” selected.

This will create a project setup to use SwiftUI. You can have a look at the created files. Apart from the AppDelegate, now we also have a SceneDelegate, which actually creates the window now. To do this, it creates a UIHostingController with a root view, called ContentView. The hosting controller is a ViewController that is capable of hosting the new SwiftUI views.

func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
     // Use a UIHostingController as window root view controller
     let window = UIWindow(frame: UIScreen.main.bounds)
     window.rootViewController = UIHostingController(rootView: ContentView())
     self.window = window
     window.makeKeyAndVisible()
}

Next, let’s build the content view. Our app will have one text view, which will display the detected text. It will also have a button, which will allow us to pick a document from which we will extract the text. Translated in code, this will look like this:

struct ContentView : View {

    @ObjectBinding var recognizedText: RecognizedText = RecognizedText()

    var body: some View {
        NavigationView {
            VStack {
                Text(recognizedText.value)
                .lineLimit(nil)
                Spacer()
                NavigationButton(destination: ScanningView(recognizedText: $recognizedText.value)) {
                    Text("Scan document")
                }
            }
        }
    }
}

There are already many great resources for SwiftUI, so I will not go into much detail about this structure. In a nutshell, in SwiftUI, we specify one body view, in which we declaratively specify the components which are part of that view.  At the root is our navigation view (since we will show another screen). Then, we have a vertical stack, which consists of the text view for the recognized text, a spacer and a navigation button. When the button is tapped, the scanning view is presented.

The most interesting part here is the @ObjectBinding property wrapper for the recognized text. @ObjectBinding is the source of truth for this view. It contains the state for the view. When this value changes, all the immutable views are re-drawn, in a smart and efficient way. @ObjectBinding is used when we want to have an external state (value which can be modified by other views as well). For simpler cases, when the scope of change is only that view, you can use the @State property wrapper.

In order to have @ObjectBinding property wrapper, we need to have a class that implements the BindableObject protocol. This protocol has only one requirement, which is implementing the didChange method. Here’s the implementation for our RecognizedText class.

import Combine
import SwiftUI

final class RecognizedText: BindableObject {

    let didChange = PassthroughSubject()

    var value: String = "Scan document to see its contents" {
        didSet {
            didChange.send(self)
        }
    }

}

To implement this protocol, we use the PassthroughSubject type from Apple’s new Combine framework. This type is simply a subject that passes values. In our case, when the value of the recognized text changes, we notify this change.

By using @ObjectBinding, we are in charge of notifying when the data is changed and this is the mechanism to do it.

Next, let’s see the ScanningView. One specific thing here is that we are using a VNDocumentCameraViewController, which is a VisionKit standard view controller, and not a SwiftUI view. This means, we need to build a wrapper in the ScanningView, to make this controller usable in SwiftUI.

The way to do this is to implement the UIViewControllerRepresentable protocol. For this, we need to implement several methods. But first, we need to specify the type of the view controller that we are wrapping, in this case VNDocumentCameraViewController.

Next, we can optionally specify a coordinator. We will need a custom coordinator in order to specify the delegate for the VNDocumentCameraViewController (more on that later). Then, in the makeUIViewController method, we create the view controller that needs to be presented by SwiftUI.

struct ScanningView: UIViewControllerRepresentable {

    @Binding var recognizedText: String

    typealias UIViewControllerType = VNDocumentCameraViewController

    func makeCoordinator() -> Coordinator {
        return Coordinator(recognizedText: $recognizedText)
    }

    func makeUIViewController(context: UIViewControllerRepresentableContext) -> VNDocumentCameraViewController {
        let documentCameraViewController = VNDocumentCameraViewController()
        documentCameraViewController.delegate = context.coordinator
        return documentCameraViewController
    }

    func updateUIViewController(_ uiViewController: VNDocumentCameraViewController, context: UIViewControllerRepresentableContext) {

    }

Note how we specified a @Binding property wrapper for the recognized text. This is an unowned binding (remember that we defined the @ObjectBinding in the ContentView). With this binding in place, whenever the text changes here, it will magically change also in the ContentView.

Next, let’s see the Coordinator implementation. Our coordinator will basically implement the VNDocumentCameraViewControllerDelegate and do the text recognition.

class Coordinator: NSObject, VNDocumentCameraViewControllerDelegate {

        @Binding var recognizedText: String
        private let textRecognizer: TextRecognizer

        init(recognizedText: Binding) {
            self.$recognizedText = recognizedText
            textRecognizer = TextRecognizer(recognizedText: recognizedText)
        }

        public func documentCameraViewController(_ controller: VNDocumentCameraViewController, didFinishWith scan: VNDocumentCameraScan) {
            var images = [CGImage]()
            for pageIndex in 0 ..< scan.pageCount {
                let image = scan.imageOfPage(at: pageIndex)
                if let cgImage = image.cgImage {
                    images.append(cgImage)
                }
            }
            textRecognizer.recognizeText(from: images)
            controller.navigationController?.popViewController(animated: true)
        }

    }

When the delegate method of the document camera view controller is called, we take the images and call our text recognizer with those images. Note how we again here specify the @Binding for the recognized text.

Finally, let's see the TextRecognizer, as well as the new text recognition API. The new type of request is called VNRecognizeTextRequest. In this request, we can specify the recognition level. There are two types of levels: fast and accurate. Fast is recommended for realtime scanning, for example in AR apps. Accurate is recommended for documents and already taken images. It takes a bit longer than the fast recognition level, since it uses neural networks to determine the characters. We will use the accurate one for our use-case.

public struct TextRecognizer {

    @Binding var recognizedText: String

    private let textRecognitionWorkQueue = DispatchQueue(label: "TextRecognitionQueue",
                                                         qos: .userInitiated, attributes: [], autoreleaseFrequency: .workItem)

    func recognizeText(from images: [CGImage]) {
        self.recognizedText = ""
        var tmp = ""
        let textRecognitionRequest = VNRecognizeTextRequest { (request, error) in
            guard let observations = request.results as? [VNRecognizedTextObservation] else {
                print("The observations are of an unexpected type.")
                return
            }
            // Concatenate the recognised text from all the observations.
            let maximumCandidates = 1
            for observation in observations {
                guard let candidate = observation.topCandidates(maximumCandidates).first else { continue }
                tmp += candidate.string + "\n"
            }
        }
        textRecognitionRequest.recognitionLevel = .accurate
        for image in images {
            let requestHandler = VNImageRequestHandler(cgImage: image, options: [:])

            do {
                try requestHandler.perform([textRecognitionRequest])
            } catch {
                print(error)
            }
            tmp += "\n\n"
        }
        self.recognizedText = tmp
    }

}

Finally, we go through the images and we create a VNImageRequestHandler for each image. Then, we perform the text recognition request with the created handler. At the end, we update the recognized text binding, which automatically updates the recognized text everywhere in the app.

Conclusion

In this post, we’ve built an app that recognizes text from an image, using the new text recognition API from iOS 13. We’ve implemented this feature using the SwiftUI framework. SwiftUI looks very cool and powerful and it will probably be the future of building iOS apps.

You can find the source code for this app here.

What are your thoughts on SwiftUI and text recognition? Do you have any suggestions how to improve the code? Please share you thoughts in the comments section.