Listing the tasks performed by the app

The app will analyze each captured frame to perform the following tasks:

• Feature extraction: We will describe an object of interest with Speeded-Up Robust Features (SURF), which is an algorithm used to find distinctive keypoints in an image that are both scale-invariant and rotation invariant. These keypoints will help us to make sure that we are tracking the right object over multiple frames because the appearance of the object might change from time to time. It is important to find keypoints that do not depend on the viewing distance or viewing angle of the object (hence, the scale and rotation invariance).
• Feature matching: We will try to establish a correspondence between keypoints using the Fast Library for Approximate Nearest Neighbors (FLANN) to see whether a frame contains keypoints similar to the keypoints from our object of interest. If we find a good match, we will mark the object on each frame.
• Feature tracking: We will keep track of the located object of interest from frame to frame using various forms of early outlier detection and outlier rejection to speed up the algorithm.
• Perspective transform: We will then reverse any translations and rotations that the object has undergone by warping the perspective so that the object appears upright in the center of the screen. This creates a cool effect in which the object seems frozen in a position while the entire surrounding scene rotates around it.

An example of the first three steps, namely, the feature extraction, matching, and tracking is shown in the following screenshot:

The screenshot contains a template image of our object of interest on the left and a handheld printout of the template image on the right. Matching features in the two frames are connected with blue lines, and the located object is outlined in green on the right.

The last step is to transform the located object so that it is projected onto the frontal plane, as depicted in the following photograph:

The image looks roughly like the original template image, appearing close-up, while the entire scene seems to warp around it.

Let's first plan the application that we are going to create in this chapter.