Exploring the limitations of generators

We noted that there are some limitations of generator expressions and generator functions. The limitations can be observed by executing the following command snippet:

>>> from ch02_ex4 import *
>>> pfactorsl(1560)
<generator object pfactorsl at 0x1007b74b0>
>>> list(pfactorsl(1560))
[2, 2, 2, 3, 5, 13]
>>> len(pfactorsl(1560))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'generator' has no len() 

In the first example, we saw the generator function, pfactors1, created a generator. The generator is lazy, and doesn't have a proper value until we consume the results yielded by the generator. in itself isn't a limitation; lazy evaluation is an important reason why generator expressions fit with functional programming in Python.

In the second example, we materialized a list object from the results yielded by the generator function. This is handy for seeing the output and writing unit test cases.

In the third example, we saw one limitation of generator functions: there's no len(). Because the generator is lazy, the size can't be known until after all of the values are consumed.

The other limitation of generator functions is that they can only be used once.

For example, look at the following command snippet:

>>> result = pfactorsl(1560)
>>> sum(result)
27
>>> sum(result)
0

The first evaluation of the sum() method performed evaluation of the generator, result. All of the values were consumed. The second evaluation of the sum() method found that the generator was now empty. We can only consume the values of a generator once.

Generators have a stateful life in Python. While they're very nice for some aspects of functional programming, they're not quite perfect.

We can try to use the itertools.tee() method to overcome the once-only limitation. We'll look at this in depth in Chapter 8, The Itertools Module. Here is a quick example of its usage:

import itertools
from typing import Iterable, Any
def limits(iterable: Iterable[Any]) -> Any:
    max_tee, min_tee = itertools.tee(iterable, 2)
    return max(max_tee), min(min_tee)  

We created two clones of the parameter generator expression, max_tee and min_tee. This leaves the original iterator untouched, a pleasant feature that allows us to do very flexible combinations of functions. We can consume these two clones to get maximum and minimum values from the iterable.

Once consumed, an iterable will not provide any more values. When we want to compute multiple kinds of reductions—for example, sums and counts, or minimums and maximums—we need to design with this one-pass-only limitation in mind.