|$ curl https://forge-ai.dev/api/markdown?path=docs/python/generators
$cat docs/python-—-generators-&-iterators.md
updated Recently·16 min read·published
Python — Generators & Iterators
◆Python◆Intermediate
Introduction
Generators are Python's primary tool for lazy evaluation. Unlike lists that hold all elements in memory, generators produce values one at a time on demand. This makes them indispensable for large datasets, streaming data, infinite sequences, and memory-efficient pipelines.
intro.py
Python
| 1 | # A list holds everything in memory |
| 2 | squares_list = [x ** 2 for x in range(10_000_000)] |
| 3 | # Memory: ~80 MB (list of 10M ints) |
| 4 | |
| 5 | # A generator produces values lazily |
| 6 | squares_gen = (x ** 2 for x in range(10_000_000)) |
| 7 | # Memory: ~120 bytes (just the generator object) |
| 8 | |
| 9 | # Generators are single-use iterators |
| 10 | for val in squares_gen: |
| 11 | if val > 100: |
| 12 | break |
| 13 | print(val, end=" ") # 0 1 4 9 16 25 36 49 64 100 |
✓
best practice
Use generators when you don't need all values at once. They transform memory from O(n) to O(1). Every generator is an iterator, but not every iterator is a generator.
Generator Functions
A generator function uses yield instead of return. When called, it returns a generator object without executing any code. Execution starts when next() is called and pauses at each yield.
generator_functions.py
Python
| 1 | def countdown(n: int): |
| 2 | """Count down from n to 1.""" |
| 3 | while n > 0: |
| 4 | yield n |
| 5 | n -= 1 |
| 6 | |
| 7 | # Calling the function creates a generator object |
| 8 | gen = countdown(5) |
| 9 | print(gen) # <generator object countdown at 0x...> |
| 10 | |
| 11 | # Each next() call resumes execution until next yield |
| 12 | print(next(gen)) # 5 |
| 13 | print(next(gen)) # 4 |
| 14 | print(next(gen)) # 3 |
| 15 | print(next(gen)) # 2 |
| 16 | print(next(gen)) # 1 |
| 17 | print(next(gen)) # StopIteration raised |
| 18 | |
| 19 | # For loop handles StopIteration automatically |
| 20 | for num in countdown(5): |
| 21 | print(num, end=" ") # 5 4 3 2 1 |
| 22 | |
| 23 | # Manual next() equivalent |
| 24 | gen = countdown(3) |
| 25 | while True: |
| 26 | try: |
| 27 | print(next(gen)) |
| 28 | except StopIteration: |
| 29 | break |
| 30 | |
| 31 | # Generator with state |
| 32 | def fibonacci(limit: int): |
| 33 | """Generate Fibonacci numbers up to limit.""" |
| 34 | a, b = 0, 1 |
| 35 | while a < limit: |
| 36 | yield a |
| 37 | a, b = b, a + b |
| 38 | |
| 39 | fib = fibonacci(100) |
| 40 | print(list(fib)) # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] |
| 41 | |
| 42 | # Multiple yields in one function |
| 43 | def multi_yield(): |
| 44 | yield "first" |
| 45 | yield "second" |
| 46 | yield "third" |
| 47 | |
| 48 | print(list(multi_yield())) # ['first', 'second', 'third'] |
| 49 | |
| 50 | # Generator with return value (Python 3.3+) |
| 51 | def with_return(n: int): |
| 52 | total = 0 |
| 53 | for i in range(n): |
| 54 | total += i |
| 55 | yield i |
| 56 | return total # stored in StopIteration.value |
| 57 | |
| 58 | gen = with_return(5) |
| 59 | for val in gen: |
| 60 | print(val, end=" ") # 0 1 2 3 4 |
| 61 | # The return value is in StopIteration: |
| 62 | # gen.value -> 10 (but can't access after for loop exits) |
ℹ
info
Unlike return which terminates the function, yield suspends it. The local variable state is preserved between calls. This is the key difference that makes generators powerful.
Generator Expressions
Generator expressions are the lazy sibling of list comprehensions. Same syntax, but with parentheses instead of brackets. They produce values on demand and never materialize the full sequence.
generator_expressions.py
Python
| 1 | # Generator expression: (expr for item in iterable) |
| 2 | gen = (x ** 2 for x in range(10)) |
| 3 | |
| 4 | # Nothing computed yet |
| 5 | print(gen) # <generator object <genexpr> at 0x...> |
| 6 | |
| 7 | # Pull values one at a time |
| 8 | print(next(gen)) # 0 |
| 9 | print(next(gen)) # 1 |
| 10 | print(next(gen)) # 4 |
| 11 | |
| 12 | # Consume all remaining (small data only!) |
| 13 | rest = list(gen) # [9, 16, 25, 36, 49, 64, 81] |
| 14 | |
| 15 | # With condition |
| 16 | evens = (x for x in range(20) if x % 2 == 0) |
| 17 | print(list(evens)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] |
| 18 | |
| 19 | # Nested for clauses |
| 20 | pairs = ((x, y) for x in range(3) for y in range(3)) |
| 21 | print(list(pairs)) |
| 22 | # [(0,0), (0,1), (0,2), (1,0), (1,1), (1,2), (2,0), (2,1), (2,2)] |
| 23 | |
| 24 | # Memory comparison |
| 25 | big_list = [i for i in range(10_000_000)] # ~80 MB |
| 26 | big_gen = (i for i in range(10_000_000)) # ~120 bytes |
| 27 | |
| 28 | # Generator expression vs list comprehension |
| 29 | import sys |
| 30 | list_size = sys.getsizeof([x for x in range(1000)]) |
| 31 | gen_size = sys.getsizeof((x for x in range(1000))) |
| 32 | print(list_size) # ~9000 bytes |
| 33 | print(gen_size) # ~120 bytes (always constant) |
| 34 | |
| 35 | # Gotcha: generator expressions exhaust after one iteration |
| 36 | gen = (x for x in range(5)) |
| 37 | print(list(gen)) # [0, 1, 2, 3, 4] |
| 38 | print(list(gen)) # [] — empty, already consumed |
| 39 | |
| 40 | # Passing generator expression directly (no extra parens needed) |
| 41 | total = sum(x ** 2 for x in range(100)) |
| 42 | print(total) # 328350 |
✓
best practice
Generator expressions are syntactically lighter than generator functions for simple transforms. Use them with sum(), max(), min(), any(), and all() to avoid creating intermediate lists.
Lazy Evaluation
Lazy evaluation means delaying computation until the result is actually needed. Generators compute each value only when requested, enabling efficient processing of large or infinite data streams.
lazy_evaluation.py
Python
| 1 | # Lazy pipeline — no intermediate lists |
| 2 | def read_large_file(path: str): |
| 3 | """Yield lines one at a time (never load entire file).""" |
| 4 | with open(path) as f: |
| 5 | for line in f: |
| 6 | yield line |
| 7 | |
| 8 | def parse_json_lines(lines): |
| 9 | for line in lines: |
| 10 | yield json.loads(line) |
| 11 | |
| 12 | def filter_active(records): |
| 13 | for rec in records: |
| 14 | if rec.get("active"): |
| 15 | yield rec |
| 16 | |
| 17 | # Pipeline chained lazily — O(1) memory |
| 18 | lines = read_large_file("data.ndjson") |
| 19 | records = parse_json_lines(lines) |
| 20 | active = filter_active(records) |
| 21 | |
| 22 | # Only one record in memory at a time |
| 23 | for rec in active: |
| 24 | process(rec) |
| 25 | |
| 26 | # Short-circuit evaluation with any/all |
| 27 | # Without list: stops early on first match |
| 28 | has_even = any(x % 2 == 0 for x in range(1_000_000_000)) |
| 29 | # After finding x=0 (even), stops immediately |
| 30 | |
| 31 | # With list: always computes ALL values first |
| 32 | has_even = any([x % 2 == 0 for x in range(1_000_000_000)]) |
| 33 | # Creates 1B-element list before checking — BAD |
| 34 | |
| 35 | # Lazy reading without generator — bad |
| 36 | def load_all(path): |
| 37 | with open(path) as f: |
| 38 | return f.readlines() # all lines in memory! |
| 39 | |
| 40 | # Lazy reading with generator — good |
| 41 | def lazy_lines(path): |
| 42 | with open(path) as f: |
| 43 | for line in f: |
| 44 | yield line # one line at a time |
ℹ
info
any() and all() with generator expressions short-circuit — they stop evaluating as soon as the result is determined. Using a list comprehension inside them defeats this optimization and doubles memory.
Infinite Generators
Because generators produce values lazily, they can represent infinite sequences. The generator never runs out of values — you control how many to consume.
infinite.py
Python
| 1 | # Infinite counter |
| 2 | def counter(start: int = 0, step: int = 1): |
| 3 | """Generate numbers forever.""" |
| 4 | current = start |
| 5 | while True: |
| 6 | yield current |
| 7 | current += step |
| 8 | |
| 9 | # Take what you need |
| 10 | c = counter() |
| 11 | print(next(c)) # 0 |
| 12 | print(next(c)) # 1 |
| 13 | print(next(c)) # 2 |
| 14 | |
| 15 | # Infinite sequence of natural numbers |
| 16 | naturals = counter(1) |
| 17 | first_10 = [next(naturals) for _ in range(10)] |
| 18 | print(first_10) # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] |
| 19 | |
| 20 | # Infinite alternating sequence |
| 21 | def alternating(): |
| 22 | """Yield True, False, True, False, ... forever.""" |
| 23 | while True: |
| 24 | yield True |
| 25 | yield False |
| 26 | |
| 27 | alt = alternating() |
| 28 | print([next(alt) for _ in range(6)]) |
| 29 | # [True, False, True, False, True, False] |
| 30 | |
| 31 | # Sieve of Eratosthenes — infinite primes |
| 32 | def primes(): |
| 33 | yield 2 |
| 34 | seen = [] |
| 35 | n = 3 |
| 36 | while True: |
| 37 | if all(n % p != 0 for p in seen): |
| 38 | seen.append(n) |
| 39 | yield n |
| 40 | n += 2 |
| 41 | |
| 42 | prime_gen = primes() |
| 43 | print([next(prime_gen) for _ in range(10)]) |
| 44 | # [2, 3, 5, 7, 11, 13, 17, 19, 23, 29] |
| 45 | |
| 46 | # Using islice to take from infinite generators |
| 47 | from itertools import islice |
| 48 | |
| 49 | def fibonacci_inf(): |
| 50 | a, b = 0, 1 |
| 51 | while True: |
| 52 | yield a |
| 53 | a, b = b, a + b |
| 54 | |
| 55 | fib = fibonacci_inf() |
| 56 | first_20 = list(islice(fib, 20)) |
| 57 | print(first_20) |
| 58 | # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, ...] |
| 59 | |
| 60 | # WARNING: never call list() on an infinite generator |
| 61 | # list(fibonacci_inf()) — hangs forever! |
✓
best practice
Always use itertools.islice() or a for loop with a break condition to consume finite portions of infinite generators. Never call list() on them.
yield from
yield from (Python 3.3+) delegates iteration to a sub-generator. It yields each value from another iterable, simplifying generator composition and eliminating manual iteration loops.
yield_from.py
Python
| 1 | # Without yield from — manual iteration |
| 2 | def chain_manual(*iterables): |
| 3 | for it in iterables: |
| 4 | for item in it: |
| 5 | yield item |
| 6 | |
| 7 | # With yield from — simpler and faster |
| 8 | def chain(*iterables): |
| 9 | for it in iterables: |
| 10 | yield from it |
| 11 | |
| 12 | print(list(chain([1, 2], [3, 4], [5, 6]))) |
| 13 | # [1, 2, 3, 4, 5, 6] |
| 14 | |
| 15 | # Flatten nested sequences recursively |
| 16 | def flatten(nested): |
| 17 | for item in nested: |
| 18 | if isinstance(item, (list, tuple)): |
| 19 | yield from flatten(item) |
| 20 | else: |
| 21 | yield item |
| 22 | |
| 23 | deep = [1, [2, [3, 4], 5], 6] |
| 24 | print(list(flatten(deep))) # [1, 2, 3, 4, 5, 6] |
| 25 | |
| 26 | # yield from with strings (iterable of chars) |
| 27 | def repeat_delegate(s: str, times: int): |
| 28 | for _ in range(times): |
| 29 | yield from s |
| 30 | |
| 31 | print("".join(repeat_delegate("ab", 3))) # ababab |
| 32 | |
| 33 | # yield from passes through return values |
| 34 | def inner(): |
| 35 | yield 1 |
| 36 | yield 2 |
| 37 | return "done" |
| 38 | |
| 39 | def outer(): |
| 40 | result = yield from inner() |
| 41 | print(f"inner returned: {result}") # inner returned: done |
| 42 | |
| 43 | list(outer()) # [1, 2] |
| 44 | |
| 45 | # yield from with generators |
| 46 | def square_gen(nums): |
| 47 | for n in nums: |
| 48 | yield n ** 2 |
| 49 | |
| 50 | def pipeline(): |
| 51 | yield from square_gen(range(5)) |
| 52 | |
| 53 | print(list(pipeline())) # [0, 1, 4, 9, 16] |
| 54 | |
| 55 | # yield from for simple delegation |
| 56 | def read_sections(paths): |
| 57 | for path in paths: |
| 58 | with open(path) as f: |
| 59 | yield from f # delegate file reading |
| 60 | |
| 61 | for line in read_sections(["a.txt", "b.txt"]): |
| 62 | process(line) |
ℹ
info
yield from isn't just syntactic sugar — it's actually faster than manual iteration and properly propagates .send(), .throw(), and .close() to sub-generators. Use it whenever you yield from another iterable.
send(), throw(), close()
Generators support two-way communication. send() passes a value into the generator (becoming the value of the yield expression). throw() injects an exception. close() terminates the generator.
send_throw_close.py
Python
| 1 | # Generator.send() — send values into a generator |
| 2 | def running_average(): |
| 3 | """Compute running average of sent values.""" |
| 4 | total = 0.0 |
| 5 | count = 0 |
| 6 | average = None |
| 7 | while True: |
| 8 | value = yield average # yield receives sent value |
| 9 | if value is None: # skip None values |
| 10 | continue |
| 11 | total += value |
| 12 | count += 1 |
| 13 | average = total / count |
| 14 | |
| 15 | avg = running_average() |
| 16 | next(avg) # prime the generator (advance to first yield) |
| 17 | print(avg.send(10)) # 10.0 |
| 18 | print(avg.send(20)) # 15.0 |
| 19 | print(avg.send(30)) # 20.0 |
| 20 | print(avg.send(40)) # 25.0 |
| 21 | |
| 22 | # Generator.throw() — inject exceptions |
| 23 | def echo(): |
| 24 | try: |
| 25 | while True: |
| 26 | val = yield |
| 27 | print(f"echo: {val}") |
| 28 | except ValueError: |
| 29 | print("value error caught") |
| 30 | except GeneratorExit: |
| 31 | print("generator closing") |
| 32 | |
| 33 | e = echo() |
| 34 | next(e) # prime |
| 35 | e.send("hello") # echo: hello |
| 36 | e.throw(ValueError) # value error caught |
| 37 | |
| 38 | # Generator.close() — stop the generator |
| 39 | def infinite(): |
| 40 | try: |
| 41 | while True: |
| 42 | yield 1 |
| 43 | finally: |
| 44 | print("cleanup: closing") |
| 45 | |
| 46 | g = infinite() |
| 47 | print(next(g)) # 1 |
| 48 | print(next(g)) # 1 |
| 49 | g.close() # cleanup: closing |
| 50 | |
| 51 | # close() raises GeneratorExit inside the generator |
| 52 | def auto_close(): |
| 53 | while True: |
| 54 | try: |
| 55 | yield |
| 56 | except GeneratorExit: |
| 57 | print("received GeneratorExit, cleaning up") |
| 58 | raise # must re-raise or StopIteration |
| 59 | |
| 60 | # Priming pattern — use a wrapper to avoid next() calls |
| 61 | @dataclass |
| 62 | class Averager: |
| 63 | gen: Generator = field(default_factory=running_average) |
| 64 | |
| 65 | def __post_init__(self): |
| 66 | next(self.gen) # prime on creation |
| 67 | |
| 68 | def add(self, value): |
| 69 | return self.gen.send(value) |
✓
best practice
Always prime a generator with next(gen) before calling send(). A common pattern is to wrap the generator in a class that handles priming automatically. Remember that yield from propagates send()/throw()/close() to sub-generators.
Generator Pipelines
Generator pipelines chain multiple generators together, where each generator is a processing stage. Data flows lazily from stage to stage with O(1) memory — no intermediate lists are created between stages.
pipelines.py
Python
| 1 | # Stage 1: source — yield raw data |
| 2 | def read_logs(path: str): |
| 3 | with open(path) as f: |
| 4 | for line in f: |
| 5 | yield line |
| 6 | |
| 7 | # Stage 2: filter — keep only relevant lines |
| 8 | def filter_by_level(lines, level: str): |
| 9 | for line in lines: |
| 10 | if f"[{level}]" in line: |
| 11 | yield line |
| 12 | |
| 13 | # Stage 3: parse — extract structured data |
| 14 | def parse_lines(lines): |
| 15 | for line in lines: |
| 16 | parts = line.strip().split(" | ") |
| 17 | yield { |
| 18 | "timestamp": parts[0], |
| 19 | "level": parts[1], |
| 20 | "message": parts[2], |
| 21 | } |
| 22 | |
| 23 | # Stage 4: transform — enrich records |
| 24 | def enrich(records): |
| 25 | for rec in records: |
| 26 | rec["length"] = len(rec["message"]) |
| 27 | yield rec |
| 28 | |
| 29 | # Stage 5: sink — consume results (terminal operation) |
| 30 | def output(records, max_records: int): |
| 31 | for i, rec in enumerate(records): |
| 32 | if i >= max_records: |
| 33 | break |
| 34 | print(f"[{rec['timestamp']}] {rec['message']}") |
| 35 | |
| 36 | # Assemble pipeline — no data flows yet |
| 37 | raw = read_logs("app.log") |
| 38 | filtered = filter_by_level(raw, "ERROR") |
| 39 | parsed = parse_lines(filtered) |
| 40 | enriched = enrich(parsed) |
| 41 | |
| 42 | # Data flows lazily on iteration — O(1) memory |
| 43 | output(enriched, max_records=10) |
| 44 | |
| 45 | # Pipeline as a list of stages |
| 46 | pipeline = [ |
| 47 | read_logs("app.log"), |
| 48 | filter_by_level(_, "WARN"), |
| 49 | parse_lines(_), |
| 50 | enrich(_), |
| 51 | ] |
| 52 | |
| 53 | # Build pipeline with functools.reduce |
| 54 | from functools import reduce |
| 55 | def pipe(data, stages): |
| 56 | return reduce(lambda d, fn: fn(d), stages, data) |
| 57 | |
| 58 | # Functions as pipeline stages |
| 59 | def grep(lines, pattern: str): |
| 60 | for line in lines: |
| 61 | if pattern in line: |
| 62 | yield line |
| 63 | |
| 64 | def take(lines, n: int): |
| 65 | for i, line in enumerate(lines): |
| 66 | if i >= n: |
| 67 | break |
| 68 | yield line |
| 69 | |
| 70 | data = pipe( |
| 71 | read_logs("data.txt"), |
| 72 | [lambda d: grep(d, "ERROR"), |
| 73 | lambda d: take(d, 5)], |
| 74 | ) |
| 75 | |
| 76 | for entry in data: |
| 77 | print(entry.strip()) |
✓
best practice
Generator pipelines are a form of streaming data processing. Each stage is a pure generator that never materializes a full collection. This pattern is the foundation of Unix pipes, Spark RDDs, and .NET LINQ. It keeps memory constant regardless of input size.
itertools Integration
The itertools module is the standard library's generator toolbox. It provides composable iterator building blocks that work seamlessly with your own generators.
itertools.py
Python
| 1 | import itertools |
| 2 | |
| 3 | # count(start, step) — infinite arithmetic progression |
| 4 | for i in itertools.count(10, 2.5): |
| 5 | if i > 20: |
| 6 | break |
| 7 | print(i) # 10.0, 12.5, 15.0, 17.5, 20.0 |
| 8 | |
| 9 | # cycle(iterable) — infinite repetition |
| 10 | colors = itertools.cycle(["red", "green", "blue"]) |
| 11 | for i, c in zip(range(6), colors): |
| 12 | print(c, end=" ") # red green blue red green blue |
| 13 | |
| 14 | # repeat(element, times) — repeat a value N times |
| 15 | list(itertools.repeat("x", 5)) # ['x', 'x', 'x', 'x', 'x'] |
| 16 | |
| 17 | # chain(*iterables) — concatenate sequences |
| 18 | list(itertools.chain([1, 2], [3, 4], [5])) |
| 19 | # [1, 2, 3, 4, 5] |
| 20 | |
| 21 | # chain.from_iterable — chain a single iterable of iterables |
| 22 | matrix = [[1, 2], [3, 4], [5]] |
| 23 | list(itertools.chain.from_iterable(matrix)) # [1, 2, 3, 4, 5] |
| 24 | |
| 25 | # islice(iterable, start, stop, step) — slice any iterator |
| 26 | list(itertools.islice(range(100), 10, 30, 3)) |
| 27 | # [10, 13, 16, 19, 22, 25, 28] |
| 28 | |
| 29 | # takewhile / dropwhile — conditional slicing |
| 30 | list(itertools.takewhile(lambda x: x < 5, count(0))) |
| 31 | # [0, 1, 2, 3, 4] |
| 32 | |
| 33 | list(itertools.dropwhile(lambda x: x < 5, [1, 4, 6, 3, 8])) |
| 34 | # [6, 3, 8] (drops prefix, yields rest) |
| 35 | |
| 36 | # accumulate — running reduction |
| 37 | list(itertools.accumulate([1, 2, 3, 4])) # [1, 3, 6, 10] |
| 38 | list(itertools.accumulate([1, 2, 3, 4], operator.mul)) |
| 39 | # [1, 2, 6, 24] |
| 40 | |
| 41 | # groupby — group adjacent elements (sort first!) |
| 42 | data = [(1, "a"), (1, "b"), (2, "c")] |
| 43 | for key, group in itertools.groupby(data, key=lambda x: x[0]): |
| 44 | print(key, list(group)) |
| 45 | |
| 46 | # permutations / combinations |
| 47 | list(itertools.permutations([1, 2, 3], 2)) |
| 48 | # [(1,2), (1,3), (2,1), (2,3), (3,1), (3,2)] |
| 49 | |
| 50 | list(itertools.combinations([1, 2, 3], 2)) |
| 51 | # [(1,2), (1,3), (2,3)] |
| 52 | |
| 53 | list(itertools.combinations_with_replacement([1, 2], 2)) |
| 54 | # [(1,1), (1,2), (2,2)] |
| 55 | |
| 56 | # product — Cartesian product |
| 57 | list(itertools.product([1, 2], ["a", "b"])) |
| 58 | # [(1,'a'), (1,'b'), (2,'a'), (2,'b')] |
| 59 | |
| 60 | # zip_longest — zip with fill value |
| 61 | list(itertools.zip_longest([1, 2], [3], fillvalue=0)) |
| 62 | # [(1, 3), (2, 0)] |
ℹ
info
itertools functions return generators, not lists. They compose beautifully — islice(chain.from_iterable(matrix), 10) gives you the first 10 flattened elements without creating any intermediate lists.
Performance Comparison
| Metric | List | Generator |
|---|---|---|
| Memory (1M items) | ~8 MB (list of ints) / ~80 MB (list of objects) | ~120 bytes (generator object) |
| Construction time | O(n) — materializes all elements | O(1) — creates lazy object |
| Iteration speed | Fast (C-level iteration) | Comparable (same C-level mechanics) |
| Indexing | O(1) random access | Not supported — sequential only |
| Reusable | Yes — iterate multiple times | No — single use, exhausts after iteration |
| Length known | Yes — len() built-in | No — must compute by consuming |
performance.py
Python
| 1 | # Benchmark: sum of squares of 1M numbers |
| 2 | import timeit |
| 3 | import sys |
| 4 | |
| 5 | # Setup |
| 6 | N = 10_000_000 |
| 7 | |
| 8 | # List comprehension — materializes full list |
| 9 | def sum_with_list(): |
| 10 | return sum([x ** 2 for x in range(N)]) |
| 11 | |
| 12 | # Generator expression — lazy evaluation |
| 13 | def sum_with_gen(): |
| 14 | return sum(x ** 2 for x in range(N)) |
| 15 | |
| 16 | # Memory measurement |
| 17 | list_size = sys.getsizeof([x for x in range(100_000)]) |
| 18 | gen_size = sys.getsizeof((x for x in range(100_000))) |
| 19 | |
| 20 | print(f"List memory: {list_size:,} bytes") |
| 21 | print(f"Gen memory: {gen_size:,} bytes") |
| 22 | print(f"Ratio: {list_size / gen_size:.0f}x") |
| 23 | |
| 24 | # Timing (list is sometimes faster due to less overhead) |
| 25 | # But memory difference is orders of magnitude |
| 26 | list_time = timeit.timeit(sum_with_list, number=10) |
| 27 | gen_time = timeit.timeit(sum_with_gen, number=10) |
| 28 | |
| 29 | print(f"List time: {list_time:.3f}s") |
| 30 | print(f"Gen time: {gen_time:.3f}s") |
| 31 | |
| 32 | # When to use each: |
| 33 | # - List: need random access, reuse, or len() |
| 34 | # - Generator: large data, streaming, infinite sequences |
| 35 | # - Both: single pass iteration over modest data |
| 36 | |
| 37 | # Memory chart: |
| 38 | # List: ████████████████████████████████ O(n) |
| 39 | # Gen: ░░ O(1) |
| 40 | |
| 41 | # Practical rule: default to generator; |
| 42 | # switch to list only when you need it. |
✓
best practice
Default to generators; switch to lists only when you need random access, multiple iterations, or len(). For single-pass operations over large data, generators are strictly better on memory and comparable on speed.
$Blueprint — Engineering Documentation·Section ID: PYTHON-GENS·Revision: 1.0