Coroutines

In Python, the key to being able to suspend the execution of a function midway through is the use of coroutines, as we will see in this section. In order to understand coroutines, one needs to understand generators, and in order to understand those, one needs to have a grasp of iterators!

Most Python programmers are familiar with the concept of iterating some sort of collection (for example, strings, lists, tuples, file objects, and so on):

>>> for i in range(3):
... print(i)
... 
0
1
2
>>> for line in open('exchange_rates_v1.py'):
... print(line, end='')
... 
#!/usr/bin/env python3
import itertools
import time
import urllib.request
…

The reason why we can iterate all sorts of objects and not just lists or strings is the iteration protocol. The iteration protocol defines a standard interface for iteration: an object that implements __iter__ and __next__ (or __iter__ and next in Python 2.x) is an iterator and, as the name suggests, can be iterated over, as shown in the following code snippet:

class MyIterator(object):
 def __init__(self, xs):
 self.xs = xs

 def __iter__(self):
 return self

 def __next__(self):
 if self.xs:
 return self.xs.pop(0)
 else:
 raise StopIteration

for i in MyIterator([0, 1, 2]):
 print(i)

Running the preceding code prints the following output:

0
1
2

Again, the reason why we can loop over any instance of MyIterator is because it implements the iterator protocol by virtue of its __iter__ and __next__ methods; the former returns the object we iterate, and the latter method returns the inpidual elements of the sequence one by one.

To better see how the protocol works, we can unroll the loop manually as the following piece of code shows:

itrtr = MyIterator([3, 4, 5, 6])

print(next(itrtr))
print(next(itrtr))
print(next(itrtr))
print(next(itrtr))

print(next(itrtr))

Running the preceding code prints the following output:

3
4
5
6
Traceback (most recent call last):
 File "iteration.py", line 32, in <module>
 print(next(itrtr))
 File "iteration.py", line 19, in __next__
 raise StopIteration
StopIteration

We instantiate MyIterator, and then, in order to get its values, we call next() on it multiple times. Once the sequence is exhausted, next() throws a StopIteration exception. The for loop in Python, for instance, uses the same mechanism; it calls next() on its iterator and catches the StopIteration exception to know when to stop.

A generator is simply a callable that generates a sequence of results rather than returning a result. This is achieved by yielding (by way of using the yield keyword) the inpidual values rather then returning them, as we can see in the following example (generators.py):

def mygenerator(n):
 while n:
 n -= 1
 yield n

if __name__ == '__main__':
 for i in mygenerator(3):
 print(i)

The preceding commands, when executed, give the following output:

2
1
0

It is the simple presence of yield that makes mygenerator a generator and not a simple function. The interesting behavior in the preceding code is that calling the generator function does not start the generation of the sequence at all; it just creates a generator object, as the following interpreter session shows:

>>> from generators import mygenerator
>>> mygenerator(5)
<generator object mygenerator at 0x101267b48>

In order to activate the generator object, we need to call next() on it, as we can see in the following snippets (in the same interpreter session):

>>> g = mygenerator(2)
>>> next(g)
1
>>> next(g)
0
>>> next(g)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration

Each next() call produces a value from the generated sequence until the sequence is empty, and that is when we get the StopIteration exception instead. This is the same behavior that we saw when we looked at iterators. Essentially, generators are a simple way to write iterators without the need for defining classes with their __iter__ and __next__ methods.

As a side note, you should keep in mind that generators are one-shot operations; it is not possible to iterate the generated sequence more than once. To do that, you have to call the generator function again.

The same yield expression used in generator functions to produce a sequence of values can be used on the right-hand side of an assignment to consume values. This allows the creation of coroutines. A coroutine is simply a type of function that can suspend and resume its execution at well-defined locations in its code (via yield expressions).

It is important to keep in mind that coroutines, despite being implemented as enhanced generators, are not conceptually generators themselves. The reason is that coroutines are not associated with iteration. Another way of looking at the difference is that generators produce values, whereas coroutines consume values.

Let's create some coroutines and see how we can use them. There are three main constructs in coroutines, which are stated as follows:

• yield(): This is used to suspend the execution of the coroutine
• send(): This is used to pass data to a coroutine (and hence resume its execution)
• close(): This is used to terminate a coroutine

The following code illustrates how we can use these in a silly coroutine (coroutines.py):

def complain_about(substring):
 print('Please talk to me!')
 try:
 while True:
 text = (yield)
 if substring in text:
 print('Oh no: I found a %s again!'
 % (substring))
 except GeneratorExit:
 print('Ok, ok: I am quitting.')

We start off by defining our coroutine; it is just a function (we called it complain_about) that takes a single argument: a string. After printing a message, it enters an infinite loop enclosed in a try except clause. This means that the only way to exit the loop is via an exception. We are particularly interested in a very specific exception: GeneratorExit. When we catch one of these, we simply clean up and quit.

The body of the loop itself is pretty simple; we use a yield expression to fetch data (somehow) and store it in the variable text. Then, we simply check whether substring is in text, and if so, we whine a bit.

The following snippet shows how we can use this coroutine in the interpreter:

>>> from coroutines import complain_about
>>> c = complain_about('Ruby')
>>> next(c)
Please talk to me!
>>> c.send('Test data')
>>> c.send('Some more random text')
>>> c.send('Test data with Ruby somewhere in it')
Oh no: I found a Ruby again!
>>> c.send('Stop complaining about Ruby or else!')
Oh no: I found a Ruby again!
>>> c.close()
Ok, ok: I am quitting.

The execution of complain_about('Ruby') creates the coroutine, but nothing else seems to happen. In order to use the newly created coroutine, we need to call next() on it, just like we had to do with generators. In fact, we see that it is only after calling next() that we get Please talk to me! printed on the screen.

At this point, the coroutine has reached the text = (yield) line, which means that it suspends its execution. The control goes back to the interpreter so that we can send data to the coroutine itself. We do that using the its send() method, as the following snippet shows:

>>> c.send('Test data')
>>> c.send('Some more random text')
>>> c.send('Test data with Ruby somewhere in it')
Oh no: I found a Ruby again!

Each call of the send() method advances the code to the next yield; in our case, to the next iteration of the while loop and back to the text = (yield) line. At this point, the control goes back to the interpreter.

We can stop the coroutine by calling its close() method, which results in a GeneratorExit exception being risen inside the coroutine. The only thing that a coroutine is allowed to do at this point is catch the exception, do some cleaning up, and exit. The following snippet shows how to close the coroutine:

>>> c.close()
Ok, ok: I am quitting.

If we were to comment out the try...except block, we would not get the GeneratorExit exception back, but the coroutine will simply stop working, as the following snippet shows:

>>> def complain_about2(substring):
... print('Please talk to me!')
... while True:
... text = (yield)
... if substring in text:
... print('Oh no: I found a %s again!'
... % (substring))
... 
>>> c = complain_about2('Ruby')
>>> next(c)
Please talk to me!
>>> c.close()
>>> c.send('This will crash')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration
>>> next(c)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration

We see from the preceding example that once we close a coroutine, the object stays around, but it is not all that useful; we cannot send data to it, and we cannot use it by calling next().

When using coroutines, most people find having to call next() on the coroutine rather annoying and end up using a decorator to avoid the extra call, as the following example shows:

>>> def coroutine(fn):
... def wrapper(*args, **kwargs):
... c = fn(*args, **kwargs)
... next(c)
... return c
... return wrapper
... 
>>> @coroutine
... def complain_about2(substring):
... print('Please talk to me!')
... while True:
... text = (yield)
... if substring in text:
... print('Oh no: I found a %s again!'
... % (substring))
... 
>>> c = complain_about2('JavaScript')
Please talk to me!
>>> c.send('Test data with JavaScript somewhere in it')
Oh no: I found a JavaScript again!
>>> c.close()

Coroutines can be arranged in rather complex hierarchies, with one coroutine sending data to multiple other ones and getting data from multiple sources as well. They are particularly useful in network programming (for performance) and in system programming, where they can be used to reimplement most Unix tools very efficiently in pure Python.