Learn Generators in Python

Generators are a special type of iterable in Python, used to generate sequences of values on the fly instead of storing them in memory. They are efficient for handling large datasets or infinite sequences because they produce values lazily, meaning one at a time as needed.

What is a Generator ?

A generator is a function that: Uses the yield keyword instead of return. Returns a generator object, which can be iterated over to produce values.

Why Use Generators ?

Efficient: Instead of holding all values in memory (like lists), generators produce them one at a time. Faster when working with large data as they don’t pre-compute all values.
Lazy Evaluation: Values are computed only when required.

How Generators Work

yield vs return

Generator Object:

When a generator function is called, it doesn’t execute immediately. Instead, it returns a generator object.

Use next() to get the next value from the generator or loop through it with a for loop.

Generator Syntax: Creating a Generator

def generator_function():
    yield value

Example 1: Simple Generator

def simple_generator():
    yield 1
    yield 2
    yield 3

gen = simple_generator()
print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3

Real-Life Analogy

Imagine a vending machine:

Instead of giving you all the snacks at once, it gives you one snack when you press a button (lazy evaluation).

Each button press resumes from where it left off (maintaining state).

Examples of Generators in Python

1. Basic Generator

def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1

for number in count_up_to(5):
    print(number)
Output:
1
2
3
4
5

2. Infinite Generator Generators can produce infinite sequences.

def infinite_numbers():
    num = 0
    while True:
        yield num
        num += 1

gen = infinite_numbers()
for _ in range(5):
    print(next(gen))  # Output: 0, 1, 2, 3, 4

Real-Life Use Case: Generating unique IDs or timestamps.

3. Fibonacci Sequence

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
for _ in range(10):
    print(next(fib))
Output:
0, 1, 1, 2, 3, 5, 8, 13, 21, 34

4. Filtering Data with Generators Generators can filter large datasets efficiently.

def even_numbers(sequence):
    for number in sequence:
        if number % 2 == 0:
            yield number

numbers = range(10)
for even in even_numbers(numbers):
    print(even)
Output:

0, 2, 4, 6, 8

Using Generator Expressions

A generator expression is a compact way to create a generator, similar to list comprehensions but with parentheses.

Example

squared = (x ** 2 for x in range(5))
print(next(squared))  # Output: 0
print(next(squared))  # Output: 1

Compare:

List comprehension: [x ** 2 for x in range(5)] creates the entire list in memory.

Generator expression: (x ** 2 for x in range(5)) computes one value at a time.

Advantages of Generators

Memory Usage:

Lists store all elements in memory.
Generators compute elements on demand, making them suitable for large or infinite sequences.

Example

import sys
list_nums = [x ** 2 for x in range(1000)]
gen_nums = (x ** 2 for x in range(1000))

print(sys.getsizeof(list_nums))  # Output: Memory size of list
print(sys.getsizeof(gen_nums))  # Output: Memory size of generator

Key Methods for Generator

Real-Life Applications

1. Processing Large Files Generators are perfect for reading large files line by line without loading the entire file into memory.

def read_large_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line.strip()

for line in read_large_file("large_file.txt"):
    print(line)

2. Streaming Data Handle live data streams like sensor readings or API responses.

import time
def live_sensor_data():
    while True:
        yield f"Sensor Reading at {time.time()}"
        time.sleep(1)

for reading in live_sensor_data():
    print(reading)

33. Pipelining Use multiple generators together to process data step-by-step.

def numbers():
    for i in range(10):
        yield i

def square_numbers(nums):
    for n in nums:
        yield n ** 2

pipeline = square_numbers(numbers())
for value in pipeline:
    print(value)

Comparing Generators with Iterators

Key Takeaways

Generators are ideal for processing large or infinite sequences efficiently.
Use yield to pause and resume execution They shine in scenarios like:
Iterating over large datasets (e.g., files, APIs).
Producing infinite or dynamic sequences.
Creating pipelines for data transformation.