Python - Iterators and Generators

Iterators and Generators in Python

Introduction

Iterators and Generators are powerful features in Python that help manage sequences of data. Understanding these tools is essential for writing efficient, readable, and memory-conscious Python code. In this guide, we will cover the foundational principles, benefits, implementation patterns, and real-world examples of both iterators and generators.

Understanding Iterables and Iterators

What is an Iterable?

An iterable is any Python object capable of returning its elements one at a time. Common iterable types include lists, tuples, dictionaries, and sets.

What is an Iterator?

An iterator is an object that implements the iterator protocol, consisting of two methods:

• __iter__(): Returns the iterator object itself.
• __next__(): Returns the next value from the iterable. Raises StopIteration when no more items are available.

Example: Manual Iteration

numbers = [1, 2, 3]
it = iter(numbers)

print(next(it))  # Output: 1
print(next(it))  # Output: 2
print(next(it))  # Output: 3
# print(next(it))  # Raises StopIteration

Creating a Custom Iterator

Steps to Create a Custom Iterator

To create a custom iterator, define a class with the __iter__ and __next__ methods.

Example: Counter Iterator

class Counter:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        current = self.current
        self.current += 1
        return current

counter = Counter(1, 5)
for number in counter:
    print(number)

Built-in Iterables

Common Built-in Iterables

• Strings
• Lists
• Tuples
• Dictionaries
• Sets
• Files

Iteration Example

my_list = ['a', 'b', 'c']
for item in my_list:
    print(item)

Why Use Iterators?

Advantages

• Memory efficiency
• Cleaner syntax
• Lazy evaluation (fetch data only when needed)
• Better performance with large datasets

Use Case: Reading Large Files

with open('largefile.txt') as f:
    for line in f:
        process(line)

Generators in Python

What is a Generator?

A generator is a simpler way of creating iterators using functions and the yield keyword. Instead of returning a value, a generator function yields it and pauses its state until the next call.

Basic Generator Example

def simple_gen():
    yield 1
    yield 2
    yield 3

for value in simple_gen():
    print(value)

How Generators Work

Generator vs Return

Unlike return, which ends the function, yield pauses execution and maintains the function’s state.

Step-by-step Execution

def counter():
    print("First yield")
    yield 1
    print("Second yield")
    yield 2

gen = counter()
print(next(gen))  # First yield, 1
print(next(gen))  # Second yield, 2

Creating Generator Functions

Example: Fibonacci Generator

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

for number in fibonacci(5):
    print(number)

Use Case: Infinite Generator

def infinite_numbers():
    num = 0
    while True:
        yield num
        num += 1

gen = infinite_numbers()
for _ in range(5):
    print(next(gen))

Generator Expressions

Generator Comprehension Syntax

Similar to list comprehensions but use parentheses instead of brackets.

Example

squares = (x*x for x in range(5))
for square in squares:
    print(square)

Use Cases of Generators

Processing Large Data Sets

def read_large_file(filepath):
    with open(filepath, 'r') as file:
        for line in file:
            yield line

for line in read_large_file("bigfile.txt"):
    print(line.strip())

Streaming APIs or Logs

Generators are ideal for consuming streaming data such as logs or sensor feeds.

Generator vs Iterator

Comparison Table

Using itertools

Introduction to itertools

The itertools module provides fast, memory-efficient tools for working with iterators.

Example: count(), cycle(), repeat()

import itertools

for num in itertools.count(5):
    print(num)
    if num > 10:
        break

for item in itertools.cycle(['A', 'B']):
    print(item)
    break

for item in itertools.repeat('Hello', 3):
    print(item)

Exception Handling in Generators

Handling Generator Completion

gen = (x for x in range(3))

try:
    while True:
        print(next(gen))
except StopIteration:
    print("Generator exhausted")

Chaining Generators

Example

def gen1():
    yield from range(3)

def gen2():
    yield from 'abc'

for item in gen1():
    print(item)

for char in gen2():
    print(char)

Real World Use Cases

CSV File Parser

import csv

def csv_reader(filepath):
    with open(filepath) as file:
        reader = csv.reader(file)
        for row in reader:
            yield row

for row in csv_reader("data.csv"):
    print(row)

Web Scraping with Generators

import requests
from bs4 import BeautifulSoup

def fetch_pages(urls):
    for url in urls:
        response = requests.get(url)
        yield BeautifulSoup(response.text, 'html.parser')

urls = ['http://example.com', 'http://example.org']
for page in fetch_pages(urls):
    print(page.title.text)

Best Practices

Use Generators for Large or Infinite Sequences

Avoid creating lists when a generator can be used to save memory.

Combine with Functions Like map(), filter()

even_numbers = filter(lambda x: x % 2 == 0, (x for x in range(10)))
print(list(even_numbers))

Avoid Side Effects Inside Generators

Generators should be pure and predictable to keep them maintainable.

Advanced Topics

Generator Delegation: yield from

def generator_a():
    yield 1
    yield 2

def generator_b():
    yield from generator_a()
    yield 3

for value in generator_b():
    print(value)

Closing a Generator

def my_generator():
    try:
        yield 1
        yield 2
    finally:
        print("Generator closed")

gen = my_generator()
print(next(gen))
gen.close()

Iterators and generators are indispensable tools in Python for managing data flow, especially when working with large datasets, infinite sequences, or stream-based data. Generators, in particular, offer concise and elegant solutions for building iterators with lazy evaluation. By using them effectively, Python developers can write efficient, scalable, and clean code. Familiarity with both concepts enhances your ability to leverage Python's full potential, making your programs both performant and memory efficient.