Python

Python Generators👨‍💻

Generators are functions that produce a sequence of values lazily — one at a time, on demand, without building the entire sequence in memory. Every time you call next() on a generator, it runs until it hits a yield, hands back the value, and freezes its entire stack frame. The next call picks up exactly where it left off, local variables and all.

This matters because real programs deal with data that doesn't fit in RAM: server logs, database result sets, streaming API responses, sensor feeds. A list comprehension that processes 10 million rows allocates 10 million objects. A generator expression processes one row at a time. Same logic, constant memory. Once you internalize this, you'll reach for generators whenever data flows through your program rather than sitting in it.

Key Takeaways

1A generator function uses `yield` instead of `return` — calling it returns a generator iterator, it does not execute the function body
2Generator state is frozen between `yield` calls: local variables, instruction pointer, and exception state are all preserved on the frame object
3Generator expressions (`(x for x in iterable)`) are the lazy counterpart to list comprehensions — same syntax, parentheses instead of brackets
4`yield from` delegates to a sub-generator transparently, forwarding `send()`, `throw()`, and `close()` calls — essential for composing generator pipelines
5`send(value)` injects a value back into the generator at the point of the last `yield`, turning generators into coroutines that accept input
6Async generators (`async def` + `yield`) combine `await` and `yield` to produce values lazily from async data sources like paginated APIs or WebSocket streams

Master python generators

Take the Python Fundamentals course with hands-on lessons and challenges.

Examples

Processing large CSV files without loading them into memory

python

Each function yields one row at a time. The pipeline chains without intermediate lists, so a 4 GB CSV uses the same memory as a 4 KB one. This is the bread-and-butter use case for generators in data processing.

Paginated API client with automatic page fetching

python

yield from flattens each page of results into a single stream. The caller doesn't know or care about pagination — it just iterates. The generator only fetches the next page when the current one is exhausted.

Generator expressions vs list comprehensions — memory comparison

python

The generator expression is 40,000x smaller than the list for the same logical operation. Pass generator expressions directly to any(), all(), sum(), min(), max(), and ''.join() to avoid allocating throwaway lists.

send() — building a running average coroutine

python

send() pushes a value into the generator at the yield point. The generator processes it and yields the next result. You must call next() once first to advance to the initial yield — this is called 'priming'. This pattern predates asyncio and is still useful for stateful stream processors.

yield from — composing sub-generators transparently

python

yield from delegates iteration to another generator or iterable. It's not just syntactic sugar for a for-loop — it correctly propagates send(), throw(), and return values through the chain, which matters when generators are used as coroutines.

Async generators — streaming paginated API responses

python

Async generators combine await (for I/O) with yield (for lazy production). They're consumed with async for. This is the cleanest way to stream data from paginated APIs, message queues, or WebSocket connections without buffering everything in memory.

Common Mistakes

Mistake:

Calling a generator function and expecting it to run — `read_csv("data.csv")` returns a generator object, it doesn't read anything

Fix:

A generator function call creates the iterator but executes zero lines of code. You must consume it: `for row in read_csv("data.csv")`, or `list(read_csv(...))`, or `next(gen)`. The body runs only when you pull values.

Mistake:

Trying to iterate over a generator twice — the second loop silently produces nothing

Fix:

Generators are single-pass. Once exhausted, they're done. If you need multiple passes, either recreate the generator by calling the function again, or collect into a list first with `data = list(gen)`. Prefer recreating when memory is the concern.

Mistake:

Using `return value` in a generator and expecting the caller to receive it via iteration

Fix:

A `return value` in a generator raises `StopIteration(value)`. Normal for-loops discard it. Only `yield from` captures the return value: `result = yield from sub_generator()`. Use `yield` for values the caller should see, `return` for a final result passed to a delegating generator.

Mistake:

Forgetting to prime a generator before calling `send()` — results in `TypeError: can't send non-None value to a just-started generator`

Fix:

Always call `next(gen)` once before `send()`. This advances the generator to the first `yield` expression, where it can receive a value. A common pattern is a decorator that auto-primes: call `next()` in the wrapper and return the primed generator.

Best Practices

Default to generator expressions over list comprehensions when the result is consumed once — sum(), any(), all(), min(), max(), and for-loops all work with generators directly
Structure data pipelines as chains of small generator functions, each responsible for one transformation — this keeps each stage testable and composable
Use `yield from` instead of `for x in sub: yield x` — it's faster, handles send/throw/close correctly, and captures the sub-generator's return value
Always close generators that hold external resources (files, connections) by using them inside `with contextlib.closing(gen)` or by letting a for-loop exhaust them naturally
Prefer async generators over collecting `await` results into lists when streaming from network sources — they keep memory bounded and allow the caller to apply backpressure

Summary

Generators are Python's mechanism for lazy, memory-efficient iteration. A `yield` in a function body turns it into a generator that produces values on demand while preserving its full stack frame between calls. Generator expressions provide the same laziness in a compact syntax. `yield from` enables clean composition of sub-generators and pipelines. `send()` and `close()` turn generators into stateful coroutines that accept input. Async generators extend the pattern to I/O-bound work. In practice, reach for generators whenever you process data that flows through your program — file streams, API pages, database cursors, transformation pipelines — rather than data that sits in memory all at once.

Practice Python with hands-on challenges

Learn python generators hands-on in your IDE

Interactive lessons and challenges on Stanza, practice in VS Code, Cursor, or the web.

Related Concepts

Python Decorators Python List Comprehensions Python Asyncio Python Context Managers

Related Cheatsheets

Python Syntax Cheatsheet

Master Python with Stanza

Interactive lessons and challenges, right in your code editor.

Check the free courses. No credit card.