Tracking hand gestures in real time

Hand gestures are analyzed by the recognize function; this is where the real magic takes place. This function handles the entire process flow, from the raw grayscale image to a recognized hand gesture. It returns the number of fingers and the illustration frame. It implements the following procedure:

1. It extracts the user's hand region by analyzing the depth map (img_gray), and returns a hand region mask (segment), like this:

def recognize(img_gray: np.ndarray) -> Tuple[int,np.ndarray]:
    # segment arm region
    segment = segment_arm(img_gray) 

2. It performs contour analysis on the hand region mask (segment). Then, it returns the largest contour found in the image (contour) and any convexity defects (defects), as follows:

# find the hull of the segmented area, and based on that find the
# convexity defects
contour, defects = find_hull_defects(segment)

3. Based on the contour found and the convexity defects, it detects the number of extended fingers (num_fingers) in the image. Then, it creates an illustration image (img_draw) using (segment) image as a template, and annotates it with contour and defect points, like this:

img_draw = cv2.cvtColor(segment, cv2.COLOR_GRAY2RGB)
num_fingers, img_draw = detect_num_fingers(contour,
                                             defects, img_draw)

4. Finally, the estimated number of extended fingers (num_fingers), as well as the annotated output image (img_draw), are returned, as follows:

return num_fingers, img_draw

In the next section, let's learn how to accomplish hand region segmentation, which we used at the beginning of the procedure.