🔬 Python Code Lab — 🧠 Generator vs List: Measuring Real Memory Usage 🐍

A generator object looks small. But comparing its shallow size with a list does not show the memory cost of doing useful work. Let's create and fully consume the same sequence, then measure its peak allocation.

1. Same result, different storage

def list_total(n):
    return sum([i * i for i in range(n)])

def generator_total(n):
    return sum(i * i for i in range(n))

The list retains every squared integer before summation starts. The generator supplies values as sum requests them. range supplies the input without a prebuilt list. Generators preserve execution state between yielded values; this example does not retain the full output. Python generators.

2. Measure peak allocations

gc.collect()
tracemalloc.start()
try:
    result = generator_total(n)
    _, peak = tracemalloc.get_traced_memory()
finally:
    tracemalloc.stop()

Start a fresh trace before each workload. Read peak, not just current allocation after it returns: the temporary list may already have been freed. The notebook checks every result against n * (n - 1) * (2 * n - 1) // 6, repeats each case three times, and reports the median peak.

This measures allocations traced during the workload. It excludes pre-existing memory and is not total process RAM (RSS); interpreter memory, tracing overhead and some native allocations are outside the reported metric. tracemalloc documentation.

3. Results from this run

CPython 3.11.9, macOS ARM64. Units: 1 KiB = 1,024 bytes; 1 MiB = 1,048,576 bytes.

All paths produced the same sum. Tracing is process-wide, so notebook background activity can add noise; raw repeats are retained in the results file. The generator's peak stayed small across these tested sizes; that does not prove every generator uses constant memory. Retained inputs, buffers and growing objects can change the result. These measurements do not compare execution speed.

4. The materialization trap

def materialized_total(n):
    return sum(list(i * i for i in range(n)))

This starts with a generator but immediately stores every result in a list. Peak memory returns to the list's range. Streaming only saves storage when downstream operations avoid collecting everything.

sys.getsizeof is useful for shallow inspection, but does not recursively include referenced objects. It also says nothing about a generator's future execution peak. getsizeof documentation.

5. Choose for the workload

Use a generator for one-pass processing that can consume values incrementally. Use a list when you need indexing, repeated traversal or retained results. An exhausted generator does not restart; create a new one to run the computation again.

The accompanying notebook includes four short code cells, correctness checks, raw measurements and a logarithmic comparison plot. Run it in a fresh kernel to reproduce the experiment on your own Python version.