Python Performance Profiling

Difficulty: Intermediate

Question

How do you profile and optimize Python code performance?

Answer

Profiling workflow: measure first, optimize second. Never guess - premature optimization wastes time.

Tools: - timeit: measure small code snippets - cProfile: function-level call counts and time - line_profiler: line-by-line timing (requires @profile decorator) - memory_profiler: memory usage per line - py-spy: sampling profiler, production-safe

Optimization strategies: 1. Use built-in functions and C extensions (map, filter, sorted, NumPy) 2. List comprehensions > loops for simple transformations 3. Local variable lookup is faster than global/attribute lookup 4. Avoid creating unnecessary objects in tight loops 5. Use generators for large sequences (memory efficiency) 6. Use slots for classes with many instances

Code examples

timeit and cProfile

import timeit
import cProfile
import pstats

# timeit: benchmark small operations
print(timeit.timeit('[x2 for x in range(1000)]', number=1000))
print(timeit.timeit('list(map(lambda x: x2, range(1000)))', number=1000))

# Compare: list comprehension vs generator for sum
print(timeit.timeit('sum(x2 for x in range(10000))', number=100))
print(timeit.timeit('sum([x2 for x in range(10000)])', number=100))

# cProfile: identify hot spots
def fibonacci(n):
    if n <= 1: return n
    return fibonacci(n-1) + fibonacci(n-2)

# Profile to file
with cProfile.Profile() as pr:
    fibonacci(30)

stats = pstats.Stats(pr)
stats.sort_stats('cumulative')  # Sort by cumulative time
stats.print_stats(10)  # Show top 10 functions

# Quick profile
cProfile.run('fibonacci(30)', sort='cumulative')

cProfile shows which functions are called most and take most time. Sort by cumulative to find the most expensive call chains.

Common Optimizations

import timeit

# 1. Built-in vs loop
numbers = list(range(10000))

# SLOW: manual loop
def sum_loop(nums):
    total = 0
    for n in nums:
        total += n
    return total

# FAST: built-in (C implementation)
def sum_builtin(nums):
    return sum(nums)

# 2. Local variable lookup
import math

# SLOW: global lookup each iteration
def compute_slow(n):
    result = []
    for i in range(n):
        result.append(math.sqrt(i))
    return result

# FAST: cache locals
def compute_fast(n):
    sqrt = math.sqrt  # Cache global in local
    append = [].append  # Cache method
    result = []
    for i in range(n):
        result.append(sqrt(i))
    return result

# 3. String concatenation
# SLOW: creates new string each time O(n^2)
def join_slow(words):
    result = ''
    for word in words:
        result += word + ','
    return result

# FAST: join O(n)
def join_fast(words):
    return ','.join(words)

words = ['word'] * 1000
print(timeit.timeit(lambda: join_slow(words), number=100))
print(timeit.timeit(lambda: join_fast(words), number=100))

sum() is a C function - far faster than a Python loop. String + is O(n^2) because strings are immutable. join() is O(n).

Key points

• Profile in your actual runtime environment - not a different machine
• NumPy vectorized operations are 100x faster than Python loops for numerical work
• CPython: local variables use LOAD_FAST (array index), globals use LOAD_GLOBAL (dict lookup)
• Memory profiling: tracemalloc is built-in; memory_profiler for line-level analysis

Concepts covered

cProfile, timeit, line_profiler, memory_profiler, Big O