concurrent.futures and Multiprocessing

Difficulty: Intermediate

Question

How do ThreadPoolExecutor and ProcessPoolExecutor work? When do you use each?

Answer

concurrent.futures provides a high-level interface for parallelism.

ThreadPoolExecutor: - Runs in multiple threads but shares the GIL - Best for I/O-bound tasks (network requests, file I/O, database queries) - Threads are lightweight - thousands are feasible - GIL prevents true parallel execution of Python bytecode

ProcessPoolExecutor: - Runs in separate processes with separate GILs - true parallelism - Best for CPU-bound tasks (image processing, data analysis, encryption) - Processes are heavy - typically CPU core count - Data must be picklable to pass between processes

Code examples

ThreadPoolExecutor for I/O

from concurrent.futures import ThreadPoolExecutor, as_completed
import time
import urllib.request

def fetch_url(url):
    time.sleep(0.1)  # Simulate network delay
    return f'Data from {url}'

urls = [f'https://api.example.com/{i}' for i in range(10)]

# Sequential: 10 * 0.1s = 1.0s
start = time.time()
results = [fetch_url(url) for url in urls]
print(f'Sequential: {time.time() - start:.2f}s')

# Concurrent: ~0.1s (all wait in parallel)
start = time.time()
with ThreadPoolExecutor(max_workers=10) as executor:
    # Submit all at once
    futures = {executor.submit(fetch_url, url): url for url in urls}

    # Process as they complete
    for future in as_completed(futures):
        url = futures[future]
        try:
            result = future.result()
            print(f'Done: {url}')
        except Exception as e:
            print(f'Error {url}: {e}')

print(f'Concurrent: {time.time() - start:.2f}s')

as_completed() yields futures as they finish (not in submission order). ThreadPoolExecutor context manager auto-shuts down and waits for completion.

ProcessPoolExecutor for CPU

from concurrent.futures import ProcessPoolExecutor
import time
import math

def is_prime(n):
    if n < 2: return False
    for i in range(2, int(math.sqrt(n)) + 1):
        if n % i == 0: return False
    return True

numbers = list(range(106, 106 + 100))

# Sequential
start = time.time()
seq_result = list(map(is_prime, numbers))
print(f'Sequential: {time.time() - start:.2f}s')

# Multiprocessing
start = time.time()
with ProcessPoolExecutor() as executor:
    par_result = list(executor.map(is_prime, numbers))
print(f'Parallel: {time.time() - start:.2f}s')

print('Results match:', seq_result == par_result)

# executor.map vs executor.submit
# map: simple, ordered results, like built-in map
# submit: returns Future, flexible, as_completed, error handling

# Limitations of ProcessPoolExecutor:
# - Functions must be picklable (no lambdas!)
# - Arguments must be picklable
# - More overhead per task than threads

CPU-bound tasks get true speedup with processes. executor.map preserves order unlike as_completed. Functions and args must be picklable.

Key points

• GIL allows only one thread to execute Python bytecode at a time
• I/O releases the GIL - threads work well for network/file/DB operations
• ProcessPoolExecutor default workers = os.cpu_count()
• multiprocessing.Pool is the lower-level equivalent of ProcessPoolExecutor

Concepts covered

ThreadPoolExecutor, ProcessPoolExecutor, Future, GIL, I/O vs CPU