Using a bilateral filter for edge-aware smoothing

A strong bilateral filter is ideally suitable for converting an RGB image into a color painting or a cartoon, because it smoothens the flat regions while keeping the edges sharp. The only drawback of this filter is its computational cost—it is orders of magnitude slower than other smoothing operations, such as a Gaussian blur.

The first measure to take when we need to reduce the computational cost is to perform an operation on an image of low resolution. In order to downscale an RGB image (imgRGB) to a quarter of its size (that is, reduce the width and height to half), we could use cv2.resize:

    img_small = cv2.resize(img_rgb, (0, 0), fx=0.5, fy=0.5) 

A pixel value in the resized image will correspond to the pixel average of a small neighborhood in the original image. However, this process may produce image artifacts, which is also known as aliasing. While image aliasing is a big problem on its own, the negative effect might be enhanced by subsequent processing, for example, edge detection.

A better alternative might be to use the Gaussian pyramid for downscaling (again to a quarter of the original size). The Gaussian pyramid consists of a blur operation that is performed before the image is resampled, which reduces any aliasing effects:

    downsampled_img = cv2.pyrDown(rgb_image)

However, even at this scale, the bilateral filter might still be too slow to run in real time. Another trick is to repeatedly (say, five times) apply a small bilateral filter to the image instead of applying a large bilateral filter once:

    for _ in range(num_bilaterals):
        filterd_small_img = cv2.bilateralFilter(downsampled_img, 9, 9, 7)

The three parameters in cv2.bilateralFilter control the diameter of the pixel neighborhood (d=9) and the standard deviation of the filter in the color space (sigmaColor=9) and coordinate space (sigmaSpace=7).

So, the final code to run the bilateral filter that we use is as follows:

1. Downsample the image using multiple pyrDown calls:

    downsampled_img = rgb_image
    for _ in range(num_pyr_downs):
        downsampled_img = cv2.pyrDown(downsampled_img)

2. Then, apply multiple bilateral filters:

    for _ in range(num_bilaterals):
        filterd_small_img = cv2.bilateralFilter(downsampled_img, 9, 9, 7)

3. And finally, upsample it to the original size:

    filtered_normal_img = filterd_small_img
    for _ in range(num_pyr_downs):
        filtered_normal_img = cv2.pyrUp(filtered_normal_img)

The result looks like a blurred color painting of a creepy programmer, as follows:

The next section shows you how to detect and emphasize prominent edges.