Working with Files

Difficulty: Intermediate

Beyond basic read and write operations, Python provides powerful tools for working with structured file formats. Real-world data is rarely stored as plain text; it comes in formats like CSV (Comma-Separated Values), JSON (JavaScript Object Notation), XML, and more. Python's standard library includes dedicated modules for the most common formats, making it straightforward to parse and generate structured data.

The readlines() method reads all lines from a file and returns them as a list of strings, where each string includes the trailing newline character. This is convenient when you need random access to specific lines, but be cautious with very large files as the entire content is loaded into memory. The writelines() method is the counterpart: it takes an iterable of strings and writes each one to the file. Importantly, writelines() does not add newline characters between items, so you must include them yourself.

The csv module provides reader and writer objects for CSV data. csv.reader() parses each row into a list of strings, handling quoting, escaping, and delimiters automatically. csv.writer() produces properly formatted CSV output. For working with named columns, csv.DictReader reads each row as a dictionary with column headers as keys, and csv.DictWriter writes dictionaries as rows. The csv module handles edge cases like commas within quoted fields that would break naive string splitting.

The json module handles JSON data through four main functions: json.dumps() serializes a Python object to a JSON string, json.loads() parses a JSON string into a Python object, json.dump() writes JSON directly to a file object, and json.load() reads JSON from a file object. JSON maps naturally to Python types: objects become dicts, arrays become lists, strings remain strings, numbers become int or float, true/false become True/False, and null becomes None.

The os.path module (or its modern replacement, pathlib) provides utilities for working with file paths in a cross-platform way. Functions like os.path.join() combines path components using the correct separator for the OS, os.path.exists() checks if a path exists, os.path.basename() extracts the file name, and os.path.splitext() separates the name from the extension. These functions abstract away platform differences between Windows backslashes and Unix forward slashes.

Code examples

readlines() and writelines()

import io

# readlines() returns a list of lines
f = io.StringIO("apple\nbanana\ncherry\n")
lines = f.readlines()
print(lines)
print(len(lines))

# writelines() writes a list of strings (no auto-newlines)
out = io.StringIO()
out.writelines(["one\n", "two\n", "three\n"])
print(out.getvalue())

readlines() preserves newlines in each element. writelines() writes each string exactly as given without adding separators. You must include '\n' in each string if you want separate lines.

Parsing CSV Data

import csv
import io

csv_data = """name,age,city
Alice,30,Mumbai
Bob,25,Delhi
Charlie,35,Bangalore"""

f = io.StringIO(csv_data)
reader = csv.DictReader(f)
for row in reader:
    print(f"{row['name']} is {row['age']} from {row['city']}")

csv.DictReader uses the first row as column headers. Each subsequent row becomes a dictionary mapping header names to values. This is much safer than splitting on commas because the csv module handles quoting and escaping correctly.

Working with JSON Data

import json
import io

# Python dict to JSON string
data = {
    "name": "Alice",
    "scores": [95, 87, 92],
    "active": True
}

json_string = json.dumps(data, indent=2)
print(json_string)
print("---")

# JSON string back to Python dict
parsed = json.loads(json_string)
print(type(parsed))
print(parsed["name"])
print(sum(parsed["scores"]))

json.dumps() converts Python objects to a JSON string. The indent parameter makes the output human-readable. json.loads() reverses the process. Note that Python True becomes JSON true (lowercase) and Python None becomes JSON null.

os.path Utilities

import os.path

# Join path components
path = os.path.join("home", "user", "documents", "report.txt")
print(path)

# Extract file name and extension
print(os.path.basename(path))
print(os.path.dirname(path))

name, ext = os.path.splitext("report.txt")
print(f"Name: {name}, Extension: {ext}")

os.path.join() uses the correct path separator for your OS. basename() extracts the filename, dirname() extracts the directory, and splitext() splits the filename from its extension. These work purely on strings and do not require the file to exist.

Key points

Use csv.DictReader instead of manually splitting CSV lines by commas. Manual splitting breaks when field values contain commas inside quotes.
json.dumps() with indent=2 produces readable output for debugging. Without indent, you get a compact single-line string suitable for APIs and storage.
readlines() loads the entire file into memory. For very large files, iterate over the file object directly instead: for line in f.

Concepts covered

readlines(), writelines(), csv Module, json Module, os.path Basics