Python Coroutines
Introduction
Coroutines are a powerful feature in Python that enable you to write non-blocking, asynchronous code in a more readable and maintainable way. They're particularly useful for I/O-bound tasks, such as making network requests, reading files, or interacting with databases, where your program might spend a lot of time waiting for external resources.
In this tutorial, we'll explore what coroutines are, how they work in Python, and when you should use them. By the end, you'll have a good understanding of how to implement coroutines in your Python programs to make them more efficient and responsive.
What Are Coroutines?
Coroutines are specialized functions that can pause execution and yield control back to the caller without losing their state. Unlike regular functions that run from start to finish when called, coroutines can be suspended and resumed, allowing multiple tasks to cooperatively share a single thread of execution.
In Python, coroutines evolved from generators and were formalized with the async/await syntax introduced in Python 3.5. They form the foundation of Python's asynchronous programming model.
Evolution of Coroutines in Python
Stage 1: Generators as Simple Coroutines
Initially, Python developers used generators as simple coroutines by leveraging the yield statement:
def simple_coroutine():
print("-> coroutine started")
x = yield
print("-> received:", x)
# Create the coroutine object
co = simple_coroutine()
# Initialize the coroutine
next(co) # Advance to the first yield
# Send a value to the coroutine
co.send("Hello!")
Output:
-> coroutine started
-> received: Hello!
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
These generator-based coroutines had limitations and weren't true asynchronous constructs.
Stage 2: Enhanced Generators (PEP 342)
Python 3.3 introduced yield from, allowing generators to delegate part of their operations to other generators:
def generator_1():
yield "Hello"
yield "World"
def generator_2():
yield from generator_1()
yield "!"
for item in generator_2():
print(item)
Output:
Hello
World
!
Stage 3: Native Coroutines with async/await (PEP 492)
Python 3.5 introduced native coroutines with the async and await keywords:
import asyncio
async def greet(name):
print(f"Hello, {name}!")
await asyncio.sleep(1)
print(f"Goodbye, {name}!")
async def main():
await greet("World")
asyncio.run(main())
Output:
Hello, World!
Goodbye, World!
This is the modern way of working with coroutines in Python, and what we'll focus on in the rest of this tutorial.
Understanding async/await
The async/await syntax is the cornerstone of modern Python coroutines:
async defdefines a coroutine functionawaitis used within a coroutine to wait for another coroutine to complete
Basic Coroutine Example
import asyncio
async def say_hello():
print("Hello")
await asyncio.sleep(1) # Non-blocking sleep
print("World")
# Run the coroutine
asyncio.run(say_hello())
Output:
Hello
World
Running Multiple Coroutines Concurrently
One of the key benefits of coroutines is the ability to run multiple tasks concurrently:
import asyncio
import time
async def count(name, delay):
print(f"{name} starting")
for i in range(3):
await asyncio.sleep(delay) # Non-blocking sleep
print(f"{name}: {i}")
print(f"{name} finished")
async def main():
start_time = time.time()
# Run tasks concurrently
await asyncio.gather(
count("Task A", 0.5),
count("Task B", 1)
)
end_time = time.time()
print(f"Completed in {end_time - start_time:.2f} seconds")
asyncio.run(main())
Output:
Task A starting
Task B starting
Task A: 0
Task B: 0
Task A: 1
Task A: 2
Task B: 1
Task A finished
Task B: 2
Task B finished
Completed in 3.01 seconds
Note that we completed both tasks in about 3 seconds, even though running them sequentially would take 4.5 seconds!
Coroutines vs Threads vs Processes
It's worth understanding how coroutines differ from other concurrency mechanisms in Python:
- Coroutines: Cooperative multitasking on a single thread. Tasks yield control voluntarily.
- Threads: Preemptive multitasking managed by the operating system, with potential for race conditions.
- Processes: Separate memory spaces with true parallelism, but higher overhead.
Coroutines are ideal for I/O-bound workloads with many waiting periods, while processes are better for CPU-bound tasks that need to leverage multiple cores.
Practical Applications of Coroutines
Web Scraping with aiohttp
Here's an example of using coroutines for concurrent web scraping:
import asyncio
import aiohttp
import time
async def fetch_page(session, url):
async with session.get(url) as response:
return await response.text()
async def get_multiple_pages(urls):
async with aiohttp.ClientSession() as session:
tasks = [fetch_page(session, url) for url in urls]
return await asyncio.gather(*tasks)
async def main():
sites = [
"https://python.org",
"https://stackoverflow.com",
"https://github.com"
]
start_time = time.time()
pages = await get_multiple_pages(sites)
end_time = time.time()
print(f"Downloaded {len(pages)} pages in {end_time - start_time:.2f} seconds")
print(f"Page sizes: {[len(page) for page in pages]}")
# Run the main coroutine
asyncio.run(main())
Output:
Downloaded 3 pages in 0.87 seconds
Page sizes: [51293, 257842, 187621]
File I/O with aiofiles
Reading/writing files asynchronously:
import asyncio
import aiofiles
async def read_file(file_path):
async with aiofiles.open(file_path, 'r') as file:
return await file.read()
async def write_file(file_path, content):
async with aiofiles.open(file_path, 'w') as file:
await file.write(content)
async def main():
# Write to a file
await write_file('example.txt', 'Hello, asynchronous world!')
# Read from the file
content = await read_file('example.txt')
print(f"File content: {content}")
asyncio.run(main())
Output:
File content: Hello, asynchronous world!
Advanced Coroutine Features
Timeouts
Setting timeouts for coroutines:
import asyncio
async def long_operation():
await asyncio.sleep(5)
return "Operation complete"
async def main():
try:
result = await asyncio.wait_for(long_operation(), timeout=2)
print(result)
except asyncio.TimeoutError:
print("Operation timed out")
asyncio.run(main())
Output:
Operation timed out
Cancellation
Cancelling running coroutines:
import asyncio
async def task():
try:
while True:
print("Task running...")
await asyncio.sleep(0.5)
except asyncio.CancelledError:
print("Task was cancelled")
raise
async def main():
t = asyncio.create_task(task())
await asyncio.sleep(2) # Let the task run for 2 seconds
t.cancel() # Cancel the task
try:
await t
except asyncio.CancelledError:
print("Main: task has been cancelled")
asyncio.run(main())
Output:
Task running...
Task running...
Task running...
Task running...
Task was cancelled
Main: task has been cancelled
Common Pitfalls and Best Practices
1. Blocking the Event Loop
Avoid CPU-intensive operations in coroutines:
import asyncio
import time
async def blocking_operation():
print("Starting blocking operation...")
time.sleep(2) # This blocks the event loop! Use asyncio.sleep instead
print("Blocking operation complete")
async def non_blocking_operation():
print("Starting non-blocking operation...")
await asyncio.sleep(2) # This yields control back to the event loop
print("Non-blocking operation complete")
async def main():
# This will block the event loop for 2 seconds
await blocking_operation()
# This is the correct way
await non_blocking_operation()
asyncio.run(main())
2. Not Awaiting Coroutines
Always await coroutines or you'll get a warning:
import asyncio
async def hello():
print("Hello")
await asyncio.sleep(1)
return "World"
async def main():
# Incorrect - coroutine not awaited
hello() # This will produce a warning
# Correct way
result = await hello()
print(result)
asyncio.run(main())
3. Using asyncio.create_task() for Concurrent Execution
Tasks allow for more control over concurrent execution:
import asyncio
async def task1():
await asyncio.sleep(1)
return "Result from task1"
async def task2():
await asyncio.sleep(2)
return "Result from task2"
async def main():
# Create tasks to run concurrently
t1 = asyncio.create_task(task1())
t2 = asyncio.create_task(task2())
# Wait for both tasks to complete
result1 = await t1
result2 = await t2
print(result1)
print(result2)
asyncio.run(main())
Output:
Result from task1
Result from task2
Summary
Coroutines in Python provide an efficient way to handle concurrent operations, especially when dealing with I/O-bound tasks. They allow you to write asynchronous code that's almost as readable as synchronous code, while gaining significant performance benefits.
Key takeaways:
- Coroutines are declared with
async defand can useawaitto call other coroutines - They're based on cooperative multitasking, where tasks voluntarily yield control
- They're ideal for I/O-bound operations like network requests or file operations
- They avoid the complexities of thread synchronization while providing efficient concurrency
- The
asynciolibrary provides the infrastructure to run and manage coroutines
Exercises
- Write a coroutine that fetches data from multiple APIs concurrently
- Create a script that downloads multiple files simultaneously using coroutines
- Implement a simple asynchronous web server using
aiohttp - Write a program that performs multiple database operations concurrently
- Implement a timeout mechanism for potentially slow operations
Additional Resources
If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)