Generators & Iterators

Difficulty: Advanced

Question

Explain generators in Python. How are they different from regular functions and lists?

Answer

Generators are functions that use 'yield' to produce a sequence of values lazily (one at a time) instead of computing all values at once and returning a list.

Key differences from regular functions: - Use yield instead of return - Maintain their state between calls - Produce values on demand (lazy evaluation) - Memory efficient: generate items one at a time

A generator function returns a generator object that implements the iterator protocol (__iter__ and __next__).

Code examples

Generator vs List: Memory Comparison

import sys

# List: stores ALL values in memory
nums_list = [x  2 for x in range(1_000_000)]
print(f"List size: {sys.getsizeof(nums_list):,} bytes")

# Generator: computes values on demand
nums_gen = (x  2 for x in range(1_000_000))
print(f"Generator size: {sys.getsizeof(nums_gen):,} bytes")

# Generator function
def squares(n):
    for i in range(n):
        yield i  2  # pauses here, resumes on next()

for sq in squares(5):
    print(sq, end=' ')

The list stores 1M values in memory (~8MB). The generator object is only 200 bytes because it computes values one at a time.

yield: Pause and Resume

def countdown(n):
    print("Starting countdown")
    while n > 0:
        yield n  # pause, return n
        n -= 1
    print("Done!")

gen = countdown(3)
print(next(gen))  # Starting countdown, then 3
print(next(gen))  # 2
print(next(gen))  # 1
# next(gen) would print 'Done!' then raise StopIteration

# Practical: Reading large files line by line
def read_large_file(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

yield pauses the function and remembers its state. next() resumes from where it left off. This is perfect for processing data that doesn't fit in memory.

Generator Pipeline Pattern

def read_lines(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

def filter_comments(lines):
    for line in lines:
        if not line.startswith('#'):
            yield line

def parse_csv(lines):
    for line in lines:
        yield line.split(',')

def get_column(rows, col):
    for row in rows:
        yield row[col]

# Compose the pipeline - nothing executes until iteration
pipeline = get_column(
    parse_csv(
        filter_comments(
            read_lines('data.csv')
        )
    ),
    col=0
)

for value in pipeline:
    print(value)

Generator pipelines are composable and memory-efficient. Each stage processes one item at a time. The entire pipeline uses O(1) memory regardless of file size.

Key points

• Generators can only be iterated once - after exhaustion they produce nothing
• yield from delegates to a sub-generator (PEP 380)
• Generator expressions: (x for x in range(10)) - like list comprehension but lazy

Concepts covered

Generators, yield, Iterator Protocol, __iter__, __next__, Generator Expression