Python Iterators
Introduction
Iterators are one of Python's most powerful features that might seem abstract at first but serve as the foundation for many Pythonic operations. At their core, iterators are objects that allow you to traverse through all the elements of a collection (like a list or tuple), one element at a time.
When you use a for loop in Python to iterate over a list, dictionary, or any other collection, you're actually using iterators behind the scenes. Understanding iterators will help you write more efficient code and unlock advanced programming techniques in Python.
What is an Iterator?
An iterator in Python is an object that implements two special methods:
__iter__()- Returns the iterator object itself__next__()- Returns the next value in the sequence
When an iterator has returned all its items, it raises a StopIteration exception to signal that the iteration is complete.
Iterator vs Iterable
Before diving deeper, let's clarify two related concepts:
- Iterable: An object capable of returning its elements one at a time. Examples include lists, tuples, dictionaries, and strings.
- Iterator: The object that produces the next value in a sequence when you call
next()on it.
Every iterator is an iterable, but not every iterable is an iterator.
Using Iterators in Python
Basic Iterator Usage
Let's start with a simple example using Python's built-in iter() and next() functions:
my_list = [1, 2, 3, 4, 5]
# Get an iterator from the list
my_iterator = iter(my_list)
# Use next() to get elements from the iterator
print(next(my_iterator)) # Output: 1
print(next(my_iterator)) # Output: 2
print(next(my_iterator)) # Output: 3
When you call iter() on a list, it returns an iterator object. Then, each time you call next() on that iterator, it gives you the next item in the sequence.
StopIteration Exception
What happens when we reach the end of the collection? Let's see:
my_list = [1, 2, 3]
my_iterator = iter(my_list)
print(next(my_iterator)) # Output: 1
print(next(my_iterator)) # Output: 2
print(next(my_iterator)) # Output: 3
# print(next(my_iterator)) # This will raise StopIteration exception
If you uncomment the last line, you'll see a StopIteration exception because we've exhausted all items in the iterator.
For Loops and Iterators
When you use a for loop to iterate over an iterable, Python handles the StopIteration exception behind the scenes:
my_list = [1, 2, 3, 4, 5]
# This is what Python does behind the scenes with a for loop:
iterator = iter(my_list)
while True:
try:
item = next(iterator)
print(item)
except StopIteration:
break
The output would be:
1
2
3
4
5
Creating Your Own Iterators
Let's create our own iterator class to understand how iterators work internally:
class CountUp:
def __init__(self, start, end):
self.current = start
self.end = end
def __iter__(self):
return self
def __next__(self):
if self.current > self.end:
raise StopIteration
else:
self.current += 1
return self.current - 1
# Using our custom iterator
counter = CountUp(1, 5)
for num in counter:
print(num)
The output would be:
1
2
3
4
5
In this example, we created a CountUp iterator class that counts from a start number to an end number. The __iter__ method returns the object itself since it's already an iterator. The __next__ method returns the next number in the sequence or raises StopIteration if we've reached the end.
Infinite Iterators
Iterators don't need to have a fixed end. They can be infinite, generating values indefinitely:
class InfiniteCounter:
def __init__(self, start=0):
self.current = start
def __iter__(self):
return self
def __next__(self):
self.current += 1
return self.current - 1
# Using our infinite iterator (be careful!)
counter = InfiniteCounter()
for i in counter:
print(i)
if i >= 9: # We need a condition to break out of the loop
break
The output would be:
0
1
2
3
4
5
6
7
8
9
Real-World Applications
1. Memory Efficiency with Large Datasets
Iterators are particularly useful when working with large datasets that might not fit into memory:
def read_large_file(file_path):
with open(file_path, 'r') as file:
for line in file: # file is an iterator that reads one line at a time
yield line.strip() # We'll learn about yield in the "Generators" lesson
# Usage
for line in read_large_file('very_large_file.txt'):
# Process one line at a time without loading the entire file
print(line[:50] + '...' if len(line) > 50 else line)
2. Custom Data Stream Processing
Imagine you're building a system to process streaming data:
class DataStream:
def __init__(self, source):
self.source = source
self.is_connected = False
def connect(self):
print(f"Connecting to {self.source}...")
self.is_connected = True
def disconnect(self):
print(f"Disconnecting from {self.source}...")
self.is_connected = False
def __iter__(self):
if not self.is_connected:
self.connect()
return self
def __next__(self):
# In a real application, this would fetch actual data
import random
import time
if random.random() < 0.1: # 10% chance to end the stream
self.disconnect()
raise StopIteration
time.sleep(0.5) # Simulate waiting for data
return random.randint(1, 100) # Return random data
# Using our data stream
stream = DataStream("sensor-123")
for data in stream:
print(f"Received data: {data}")
3. Custom Range Function
Let's implement our own version of Python's range function using iterators:
class MyRange:
def __init__(self, start, stop=None, step=1):
if stop is None:
self.start = 0
self.stop = start
else:
self.start = start
self.stop = stop
self.step = step
def __iter__(self):
self.current = self.start
return self
def __next__(self):
if (self.step > 0 and self.current >= self.stop) or \
(self.step < 0 and self.current <= self.stop):
raise StopIteration
result = self.current
self.current += self.step
return result
# Using our custom range
for i in MyRange(1, 10, 2):
print(i)
The output would be:
1
3
5
7
9
Iterator Tools in Python
Python provides the itertools module, filled with functions that create and manipulate iterators in efficient ways:
import itertools
# Creating an infinite iterator that counts up from 1
counter = itertools.count(1)
for i in counter:
print(i)
if i >= 5:
break
print("---")
# Cycling through a finite sequence infinitely
colors = itertools.cycle(['red', 'green', 'blue'])
for i, color in enumerate(colors):
print(color)
if i >= 5:
break
print("---")
# Repeat an element a specified number of times
repeater = itertools.repeat("Hello", 3)
for message in repeater:
print(message)
The output would be:
1
2
3
4
5
---
red
green
blue
red
green
blue
---
Hello
Hello
Hello
Summary
Iterators are powerful tools in Python that allow you to work with sequences efficiently. They:
- Enable traversal through collections one element at a time
- Help conserve memory when working with large datasets
- Form the foundation for advanced features like generators and comprehensions
- Allow you to create custom sequences with specific behaviors
By understanding iterators, you've taken a significant step toward advanced Python programming. In the next lessons, we'll explore related concepts like generators, which build upon iterators to provide even more powerful functionality.
Exercises
- Create an iterator that returns the Fibonacci sequence up to a specified limit.
- Implement an iterator that skips every nth element in a list.
- Write a program that uses iterators to merge two sorted lists into a single sorted sequence.
- Create an iterator that returns all possible pairs of elements from two different iterables (similar to
itertools.product()). - Implement a "chunking" iterator that breaks a large iterable into chunks of a specified size.
Additional Resources
If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)