Python Iterator Patterns: Advanced Iteration Techniques

Python's iterator protocol and patterns provide powerful tools for efficient data processing. This guide explores advanced iterator techniques that can make your code more elegant and performant.

Table of Contents #

Understanding Iterators
Custom Iterator Classes
Generator Functions
Itertools Module
Advanced Iterator Patterns
Performance Considerations
Best Practices

Understanding Iterators #

Iterator Protocol Basics #

Understanding how Python's iterator protocol works:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Built-in Iterators #

Explore Python's built-in iterator types:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Custom Iterator Classes #

Basic Custom Iterator #

Create your own iterator classes:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Fibonacci Iterator #

A more complex iterator example:

🐍 Try it yourself

class FibonacciIterator:
    def __init__(self, max_count=None):
        self.max_count = max_count
        self.count = 0
        self.current = 0
        self.next_val = 1
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.max_count and self.count >= self.max_count:
            raise StopIteration
        
        if self.count == 0:
            self.count += 1
            return self.current
        elif self.count == 1:
            self.count += 1
            return self.next_val
        else:
            new_val = self.current + self.next_val
            self.current, self.next_val = self.next_val, new_val
            self.count += 1
            return new_val

# Using Fibonacci iterator
print("First 10 Fibonacci numbers:")
fib = FibonacciIterator(10)
for num in fib:
    print(num, end=" ")
print()

# Infinite Fibonacci (with manual break)
print("\nInfinite Fibonacci (first 8):")
infinite_fib = FibonacciIterator()
count = 0
for num in infinite_fib:
    print(num, end=" ")
    count += 1
    if count >= 8:
        break
print()

Output:

Click "Run Code" to see the output

Iterable Class #

Create classes that can be iterated multiple times:

🐍 Try it yourself

class NumberRange:
    def __init__(self, start, end, step=1):
        self.start = start
        self.end = end
        self.step = step
    
    def __iter__(self):
        return NumberRangeIterator(self.start, self.end, self.step)

class NumberRangeIterator:
    def __init__(self, start, end, step):
        self.current = start
        self.end = end
        self.step = step
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if (self.step > 0 and self.current >= self.end) or \
           (self.step < 0 and self.current <= self.end):
            raise StopIteration
        
        value = self.current
        self.current += self.step
        return value

# Using iterable class
numbers = NumberRange(1, 10, 2)

print("First iteration:")
for num in numbers:
    print(num, end=" ")
print()

print("Second iteration (works because class is iterable):")
for num in numbers:
    print(num, end=" ")
print()

# Different ranges
print("\nCountdown range:")
countdown = NumberRange(10, 0, -2)
print(list(countdown))

Output:

Click "Run Code" to see the output

Generator Functions #

Basic Generator Functions #

Use yield to create generator functions:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Advanced Generator Patterns #

More sophisticated generator techniques:

🐍 Try it yourself

# Generator with state
def sliding_window(data, window_size):
    """Generate sliding windows of data"""
    if len(data) < window_size:
        return
    
    for i in range(len(data) - window_size + 1):
        yield data[i:i + window_size]

# Using sliding window
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print("Sliding windows of size 3:")
for window in sliding_window(data, 3):
    print(window)

# Generator with send() method
def accumulator():
    total = 0
    while True:
        value = yield total
        if value is not None:
            total += value

print("\nAccumulator generator:")
acc = accumulator()
next(acc)  # Prime the generator

print(f"Send 5: {acc.send(5)}")
print(f"Send 10: {acc.send(10)}")
print(f"Send 3: {acc.send(3)}")

# Generator pipeline
def numbers():
    for i in range(10):
        yield i

def squares(nums):
    for num in nums:
        yield num ** 2

def evens(nums):
    for num in nums:
        if num % 2 == 0:
            yield num

# Pipeline usage
print("\nGenerator pipeline:")
pipeline = evens(squares(numbers()))
result = list(pipeline)
print(f"Even squares from 0-9: {result}")

Output:

Click "Run Code" to see the output

Itertools Module #

Basic Itertools Functions #

Explore the powerful itertools module:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Combinatorial Itertools #

Advanced combinatorial functions:

🐍 Try it yourself

# permutations() - all permutations
print("itertools.permutations():")
letters = ['A', 'B', 'C']
perms = list(itertools.permutations(letters, 2))
print(f"Permutations of 2 from {letters}: {perms}")

# combinations() - combinations without replacement
print("\nitertools.combinations():")
numbers = [1, 2, 3, 4]
combos = list(itertools.combinations(numbers, 2))
print(f"Combinations of 2 from {numbers}: {combos}")

# combinations_with_replacement()
print("\nitertools.combinations_with_replacement():")
combos_repl = list(itertools.combinations_with_replacement([1, 2, 3], 2))
print(f"Combinations with replacement: {combos_repl}")

# product() - Cartesian product
print("\nitertools.product():")
colors = ['red', 'blue']
sizes = ['S', 'M', 'L']
products = list(itertools.product(colors, sizes))
print(f"Product combinations: {products}")

# groupby() - group consecutive elements
print("\nitertools.groupby():")
data = [1, 1, 2, 2, 2, 3, 1, 1]
grouped = [(key, list(group)) for key, group in itertools.groupby(data)]
print(f"Grouped consecutive: {grouped}")

Output:

Click "Run Code" to see the output

Filtering and Mapping Itertools #

Advanced filtering and mapping:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Advanced Iterator Patterns #

Iterator Chaining and Composition #

Combine multiple iterators:

🐍 Try it yourself

# Custom iterator composition
def data_pipeline(*iterators):
    """Chain multiple iterators into a pipeline"""
    for iterator in iterators:
        yield from iterator

# Create different data sources
def source1():
    yield from range(1, 4)

def source2():
    yield from ['a', 'b', 'c']

def source3():
    yield from [10, 20, 30]

# Combine sources
print("Pipeline composition:")
pipeline = data_pipeline(source1(), source2(), source3())
result = list(pipeline)
print(f"Combined data: {result}")

# Conditional iteration
def conditional_iterator(data, condition):
    """Yield items based on dynamic condition"""
    for item in data:
        if condition(item):
            yield item

# Dynamic filtering
numbers = range(20)

# First: only evens
even_filter = conditional_iterator(numbers, lambda x: x % 2 == 0)
evens = list(even_filter)
print(f"\nEven numbers: {evens}")

# Then: only numbers > 10
large_filter = conditional_iterator(numbers, lambda x: x > 10)
large_numbers = list(large_filter)
print(f"Numbers > 10: {large_numbers}")

Output:

Click "Run Code" to see the output

Stateful Iterators #

Iterators that maintain state:

🐍 Try it yourself

class MovingAverage:
    def __init__(self, window_size):
        self.window_size = window_size
        self.values = []
    
    def __iter__(self):
        return self
    
    def __next__(self):
        # This would typically get values from an external source
        # For demo, we'll simulate with a simple sequence
        if len(self.values) >= 20:  # Stop after 20 values
            raise StopIteration
        
        # Simulate new data point
        import random
        new_value = random.randint(1, 100)
        self.values.append(new_value)
        
        # Calculate moving average
        if len(self.values) >= self.window_size:
            window = self.values[-self.window_size:]
            avg = sum(window) / len(window)
            return {'value': new_value, 'moving_avg': round(avg, 2)}
        else:
            return {'value': new_value, 'moving_avg': new_value}

# Using stateful iterator
print("Moving average iterator:")
ma_iter = MovingAverage(3)

for i, data in enumerate(ma_iter):
    if i >= 10:  # Show first 10 items
        break
    print(f"Value: {data['value']}, Moving Avg: {data['moving_avg']}")

# Lazy evaluation with caching
class CachedIterator:
    def __init__(self, source):
        self.source = source
        self.cache = []
        self.index = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.index < len(self.cache):
            value = self.cache[self.index]
        else:
            try:
                value = next(self.source)
                self.cache.append(value)
            except StopIteration:
                raise
        
        self.index += 1
        return value
    
    def reset(self):
        self.index = 0

# Using cached iterator
print("\nCached iterator:")
source = iter([1, 2, 3, 4, 5])
cached = CachedIterator(source)

print("First pass:", list(cached))

cached.reset()
print("Second pass (from cache):", list(cached))

Output:

Click "Run Code" to see the output

Performance Considerations #

Iterator vs List Performance #

Compare memory and time performance:

🐍 Try it yourself

Output:

Click "Run Code" to see the output

Efficient Iterator Patterns #

Best practices for iterator performance:

🐍 Try it yourself

# Efficient batch processing
def batch_iterator(iterable, batch_size):
    """Process items in batches for efficiency"""
    iterator = iter(iterable)
    while True:
        batch = list(itertools.islice(iterator, batch_size))
        if not batch:
            break
        yield batch

# Using batch iterator
large_dataset = range(25)
print("Batch processing:")

for batch_num, batch in enumerate(batch_iterator(large_dataset, 5)):
    batch_sum = sum(batch)
    print(f"Batch {batch_num + 1}: {batch} -> sum: {batch_sum}")

# Efficient infinite iteration with break conditions
def infinite_primes():
    """Generate prime numbers infinitely"""
    def is_prime(n):
        if n < 2:
            return False
        for i in range(2, int(n ** 0.5) + 1):
            if n % i == 0:
                return False
        return True
    
    n = 2
    while True:
        if is_prime(n):
            yield n
        n += 1

print("\nFirst 10 prime numbers:")
prime_gen = infinite_primes()
primes = []
for prime in prime_gen:
    primes.append(prime)
    if len(primes) >= 10:
        break

print(primes)

Output:

Click "Run Code" to see the output

Best Practices #

Iterator Design Guidelines #

Always implement both __iter__ and __next__ for iterator classes
Use generators for simple iteration logic
Make iterators lazy - compute values on demand
Handle StopIteration properly
Consider memory usage for large datasets
Use itertools for complex iteration patterns

Common Patterns Summary #

🐍 Try it yourself

# Summary of common iterator patterns

# Pattern 1: Generator function
def simple_generator(n):
    for i in range(n):
        yield i ** 2

# Pattern 2: Generator expression
squares = (x ** 2 for x in range(10))

# Pattern 3: Iterator class
class SimpleIterator:
    def __init__(self, data):
        self.data = data
        self.index = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.index >= len(self.data):
            raise StopIteration
        value = self.data[self.index]
        self.index += 1
        return value

# Pattern 4: Using itertools
import itertools
combined = itertools.chain([1, 2, 3], [4, 5, 6])

# Pattern 5: Custom iteration protocol
class Iterable:
    def __init__(self, data):
        self.data = data
    
    def __iter__(self):
        return iter(self.data)

# Test all patterns
print("Pattern examples:")
print("Generator function:", list(simple_generator(5)))
print("Generator expression:", list(squares))
print("Iterator class:", list(SimpleIterator([1, 2, 3])))
print("Itertools chain:", list(combined))
print("Custom iterable:", list(Iterable([7, 8, 9])))

Output:

Click "Run Code" to see the output

Conclusion #

Python iterator patterns provide powerful tools for:

Memory-efficient data processing
Lazy evaluation of large datasets
Clean, readable code for complex iterations
Composable data pipelines
Custom iteration logic

Key takeaways:

Use generators for simple iteration needs
Implement iterator classes for complex state management
Leverage itertools for advanced iteration patterns
Always consider memory usage and performance
Follow the iterator protocol correctly

Master these patterns to write more efficient and elegant Python code!