How it works…

Let's now look at how we can achieve multiprocessing in Python. Our imports include the multiprocessing library, shortened to mp, as it is quite lengthy otherwise; the logging and sys libraries for thread status messages; the time library to slow down execution for our example; and the randint method to generate times that each thread should wait for:

from __future__ import print_function
import logging
import multiprocessing as mp
from random import randint
import sys
import time

Before creating our processes, we set up a function that they will execute. This is where we put the task each process should execute before returning to the main thread. In this case, we take a number of seconds for the thread to sleep as our only argument. To print a status message that allows us to differentiate between the processes, we use the current_process() method to access the name property for each thread:

def sleepy(seconds):
    proc_name = mp.current_process().name
    logger.info("{} is sleeping for {} seconds.".format(
        proc_name, seconds))
    time.sleep(seconds)

With our worker function defined, we create our logger instance, borrowing code from the previous recipe, and set it to only record to the console.

logger = logging.getLogger(__file__)
logger.setLevel(logging.DEBUG)
msg_fmt = logging.Formatter("%(asctime)-15s %(funcName)-7s "
                            "%(levelname)-8s %(message)s")
strhndl = logging.StreamHandler(sys.stdout)
strhndl.setFormatter(fmt=msg_fmt)
logger.addHandler(strhndl)

We now define the number of workers we want to spawn and create them in a for loop. Using this technique, we can easily adjust the number of processes we have running. Inside of our loop, we define each worker using the Process class and set our target function and the required arguments. Once the process instance is defined, we start it and append the object to a list for later use:

num_workers = 5
workers = []
for w in range(num_workers):
    p = mp.Process(target=sleepy, args=(randint(1, 20),))
    p.start()
    workers.append(p)

By appending the workers to a list, we can join them in sequential order. Joining, in this context, is the process of waiting for a process to complete before execution continues. If we do not join our process, one of them could continue to the end of the script and complete the code before other processes complete. While that wouldn't cause huge problems in our example, it can cause the next snippet of code to start too early:

for worker in workers:
    worker.join()
    logger.info("Joined process {}".format(worker.name))

When we execute the script, we can see the processes start and join over time. Since we stored these items in a list, they will join in an ordered fashion, regardless of the time it takes for one worker to finish. This is visible below as Process-5 slept for 14 seconds before completing, and meanwhile, Process-4 and Process-3 had already completed: