Generators

Difficulty: Advanced

Generators are a special type of iterator in Python that allow you to produce a sequence of values lazily -- one at a time, on demand -- rather than computing and storing them all in memory at once. This makes generators extremely memory-efficient for working with large or infinite sequences.

A generator function looks like a regular function but uses the `yield` keyword instead of (or in addition to) `return`. When called, a generator function returns a generator object without executing the function body. Each call to next() on the generator resumes execution until the next yield, which produces a value and suspends the function's state. When the function exits, StopIteration is raised automatically.

Generator expressions provide a concise syntax similar to list comprehensions but with parentheses instead of brackets: (x2 for x in range(10)). They create generator objects that produce values lazily, making them ideal for large datasets where you don't need all values at once.

The send() method allows you to send a value back into a generator, which becomes the result of the yield expression inside the generator. This enables two-way communication and turns generators into coroutines. The first call must be send(None) or next() to advance the generator to its first yield.

Python's itertools module provides a collection of fast, memory-efficient tools for working with iterators. Functions like chain(), islice(), count(), cycle(), and combinations() build on the generator concept and are indispensable for advanced iteration patterns.

Code examples

Basic Generator with yield

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for num in countdown(5):
    print(num)

Each iteration calls next() on the generator, which resumes countdown() until the next yield. The function's local state (n) is preserved between calls.

Generator Expression vs List Comprehension

squares_list = [x2 for x in range(5)]
print(type(squares_list))
print(squares_list)

squares_gen = (x2 for x in range(5))
print(type(squares_gen))
print(list(squares_gen))

List comprehensions build the entire list in memory. Generator expressions produce values lazily, using minimal memory regardless of sequence length.

Using next() and StopIteration

def simple_gen():
    yield "first"
    yield "second"
    yield "third"

g = simple_gen()
print(next(g))
print(next(g))
print(next(g))

Each next() call resumes the generator until the next yield. Calling next() a fourth time would raise StopIteration.

Using send() for Two-Way Communication

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value

gen = accumulator()
print(next(gen))
print(gen.send(10))
print(gen.send(20))
print(gen.send(5))

next(gen) advances to the first yield (total=0). Each send(value) resumes the generator, assigns value to the yield expression, and runs until the next yield returns the updated total.

Key points

• Generators are iterators -- they implement __iter__ and __next__ automatically, so you can use them in for loops and with next().
• Generator expressions use parentheses: (expr for x in iterable). When passed as the sole argument to a function, the extra parentheses can be omitted: sum(x2 for x in range(10)).
• A generator can only be iterated once. After exhaustion, calling next() raises StopIteration.

Concepts covered

yield, generator expressions, send(), next(), itertools