Python Concurrency: Threads, Processes, and asyncio

Python offers several concurrency tools that let a program handle other work while waiting on I/O, or use multiple CPU cores through multiple processes. This article covers the basic usage and typical use cases of threading, multiprocessing, asyncio, and concurrent.futures.

1. Threading

Python’s threading module lets you create multiple threads within the same process. Threads share memory, which suits I/O-bound work like network requests or file reads and writes, but shared state also brings the risk of race conditions and deadlocks. In the common CPython implementation, the GIL prevents multiple threads from executing Python CPU-bound code at the same time, so threads are not the same thing as CPU parallelism.

1.1 Basic use of threads

import threading

def print_numbers():
    for i in range(5):
        print(i)

# create the thread
thread = threading.Thread(target=print_numbers)
thread.start()
thread.join()

2. Multiprocessing

To get around Python’s Global Interpreter Lock (GIL), the multiprocessing module lets you create multiple processes. Each process has its own memory space and its own Python interpreter, so it achieves true parallel computation.

2.1 Basic use of multiprocessing

from multiprocessing import Process

def print_numbers():
    for i in range(5):
        print(i)

# create the process
process = Process(target=print_numbers)
process.start()
process.join()

3. Asynchronous Programming (asyncio)

Python’s asyncio library provides asynchronous programming capability, letting you write single-threaded concurrent code. Using the async and await keywords, you can carry out multiple operations without blocking the main thread.

3.1 Basic use of asynchronous programming

import asyncio

async def print_numbers():
    for i in range(5):
        print(i)
        await asyncio.sleep(1)

# run the event loop
asyncio.run(print_numbers())

4. Higher-Level Concurrency Tools

Python also provides higher-level tools to simplify concurrent programming, such as the concurrent.futures module, which offers a high-level interface for thread pools and process pools.

4.1 Using a thread pool

from concurrent.futures import ThreadPoolExecutor

def perform_task(x):
    return x * x

with ThreadPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(perform_task, range(5)))

4.2 Using a process pool

from concurrent.futures import ProcessPoolExecutor

def perform_task(x):
    return x * x

with ProcessPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(perform_task, range(5)))

5. Summary

Python’s concurrency tools solve different problems: I/O-bound tasks generally suit threads or asyncio, while CPU-bound tasks generally suit multiprocessing. Before choosing, first determine whether the bottleneck is waiting or computing, then weigh engineering costs such as shared state, error handling, and task cancellation.

---

This article is based on the concurrent-execution section of the official Python documentation, covering the main modules and tools Python provides for concurrent execution along with usage examples. Hopefully it helps you better understand and apply concurrent programming in Python.