Chapter 3. Parallelism in Python

We mentioned threads, processes, and in general, parallel programming in the previous two chapters. We talked, at a very high level and very much in abstract terms, about how you can organize code so that some portions run in parallel, potentially on multiple CPUs or even multiple machines.

In this chapter, we will look at parallel programming in more detail and see which facilities Python offers us to make our code use more than one CPU or CPU core at the time (but always within the boundaries of a single machine). The main goal here will be speed for CPU-intensive problems, and responsiveness for I/O-intensive code.

The good news is that we can write parallel programs in Python using just modules in the standard library and nothing else. This is not to say that no external libraries and tools might be relevant—quite the opposite. It is just that the Standard Library is enough for what we will try and do in this chapter.

In this chapter, we will cover the following topics:

• Multiple threads
• Multiple processes
• Multiprocess queues