subprocess Module

Difficulty: Intermediate

Question

How do you run external commands in Python with the subprocess module?

Answer

subprocess (Python 3.5+) is the modern way to run external programs, replacing os.system() and os.popen().

subprocess.run(): high-level API for simple command execution. Waits for completion. subprocess.Popen(): low-level API for streaming output, async execution.

Security: never use shell=True with user input - it enables shell injection. Pass arguments as a list instead.

Key parameters: - check=True: raises CalledProcessError on non-zero exit - capture_output=True: captures stdout and stderr - text=True: decode bytes to string (encoding=sys.stdout.encoding)

Code examples

subprocess.run() Patterns

import subprocess
import sys

# Basic run - wait for completion
result = subprocess.run(
    ['ls', '-la'],
    capture_output=True,
    text=True,
    check=True  # Raise on non-zero exit
)
print(result.stdout)
print('Return code:', result.returncode)

# Handle errors
try:
    subprocess.run(
        ['git', 'status'],
        check=True,
        capture_output=True,
        text=True
    )
except subprocess.CalledProcessError as e:
    print(f'Command failed: {e.returncode}')
    print(f'Stderr: {e.stderr}')
except FileNotFoundError:
    print('git not installed')

# Timeout
try:
    subprocess.run(['sleep', '10'], timeout=2, check=True)
except subprocess.TimeoutExpired:
    print('Command timed out')

# NEVER do this with user input:
user_input = 'safe.txt; rm -rf /'
# subprocess.run(f'cat {user_input}', shell=True)  # SHELL INJECTION!

# SAFE: pass as list
subprocess.run(['cat', 'safe.txt'], check=True)  # Arguments are not interpreted by shell

check=True turns non-zero exit codes into exceptions. capture_output=True = stdout=PIPE, stderr=PIPE. Lists prevent shell injection.

Popen for Streaming

import subprocess

# Popen for streaming output
with subprocess.Popen(
    ['ping', '-c', '4', 'google.com'],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True
) as proc:
    for line in proc.stdout:
        print(line, end='')  # Stream line by line

# Communicate: send stdin, get stdout/stderr
result = subprocess.run(
    ['python', '-c', 'import sys; print(sys.stdin.read().upper())'],
    input='hello world',
    capture_output=True,
    text=True
)
print(result.stdout)  # HELLO WORLD\n

# Run Python subprocess (same Python executable)
result = subprocess.run(
    [sys.executable, '-c', 'print(1 + 1)'],
    capture_output=True,
    text=True
)
print(result.stdout)  # 2

# Check if command exists
def is_installed(command):
    return subprocess.run(
        ['which', command],
        capture_output=True
    ).returncode == 0

print(is_installed('git'))   # True or False
print(is_installed('foo'))   # False

Popen enables line-by-line streaming without buffering all output in memory. sys.executable ensures the same Python version is used for subprocesses.

Key points

• Never use shell=True with unsanitized input - use shlex.quote() if shell=True is unavoidable
• capture_output=True is shorthand for stdout=PIPE, stderr=PIPE
• Popen is async - use proc.wait() or communicate() to avoid zombies
• shlex.split() parses shell-style strings into lists: shlex.split('ls -la /tmp')

Concepts covered

subprocess, run(), Popen, check=True, Shell Injection