Python for Data

Difficulty: Beginner

Python's built-in features make it an exceptionally powerful language for data manipulation even without external libraries. Before learning NumPy and Pandas, it is essential to master the core Python tools for data processing: list comprehensions, dictionary operations, and built-in functions like map, filter, sorted, zip, and enumerate. These tools form the foundation of data manipulation and will make you more effective even when you move to specialized libraries.

List comprehensions are one of Python's most powerful features for data transformation. They provide a concise way to create new lists by applying an expression to each element of an iterable, optionally filtering elements with a condition. The syntax [expression for item in iterable if condition] replaces what would otherwise be a multi-line loop. List comprehensions are not just syntactic sugar; they are also faster than equivalent for loops because Python optimizes them internally. For data work, they are indispensable for tasks like extracting fields, computing derived values, and filtering records.

Dictionary operations in Python go far beyond simple key-value lookups. Dictionary comprehensions ({key_expr: value_expr for item in iterable}) let you build dictionaries from any iterable. The get() method provides safe access with default values. The items(), keys(), and values() methods enable iteration over dictionaries in different ways. For data analytics, dictionaries are the natural choice for grouping, counting, and creating lookup tables. The pattern of accumulating values in a dictionary (group-by) is one of the most common operations in data processing.

Python's built-in functions provide functional programming capabilities that are perfect for data pipelines. The map() function applies a transformation to every element of an iterable. The filter() function selects elements that satisfy a predicate. The sorted() function returns a new sorted list with a customizable key function. The zip() function pairs elements from multiple iterables together, which is invaluable for working with parallel data sequences. The enumerate() function adds an index counter to any iterable, useful for numbered output and positional logic.

Combining these tools lets you build powerful data processing pipelines without any imports. You can chain operations together: filter a dataset, transform the remaining records, group them by a key, and compute aggregates, all in a few readable lines of Python. This functional approach to data manipulation not only produces cleaner code but also maps directly to the operations you will use in Pandas later. Understanding these patterns in pure Python will make the transition to data analysis libraries much smoother.

Code examples

List Comprehensions for Data

# Raw temperature data in Celsius
temps_celsius = [22, 35, 18, 40, 28, 15, 33]

# Transform: Convert all to Fahrenheit
temps_fahrenheit = [round(c * 9/5 + 32, 1) for c in temps_celsius]
print("Fahrenheit: " + str(temps_fahrenheit))

# Filter: Only hot days (above 30C)
hot_days = [c for c in temps_celsius if c > 30]
print("Hot days (>30C): " + str(hot_days))

# Combined: Fahrenheit of hot days only
hot_fahrenheit = [round(c * 9/5 + 32, 1) for c in temps_celsius if c > 30]
print("Hot days (F): " + str(hot_fahrenheit))

# Nested: Flatten a matrix
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [x for row in matrix for x in row]
print("Flat: " + str(flat))

List comprehensions can transform, filter, or do both simultaneously. The syntax reads naturally: 'give me (expression) for each (item) in (iterable) if (condition)'. Nested comprehensions can flatten multi-dimensional data structures.

Dictionary Comprehensions and Grouping

# Sales data
sales = [
    {"rep": "Alice", "amount": 500},
    {"rep": "Bob", "amount": 300},
    {"rep": "Alice", "amount": 700},
    {"rep": "Charlie", "amount": 450},
    {"rep": "Bob", "amount": 600},
    {"rep": "Alice", "amount": 200},
]

# Group sales by rep
grouped = {}
for sale in sales:
    rep = sale["rep"]
    if rep not in grouped:
        grouped[rep] = []
    grouped[rep].append(sale["amount"])

# Dict comprehension: total per rep
totals = {rep: sum(amounts) for rep, amounts in grouped.items()}
print("Totals: " + str(totals))

# Dict comprehension: average per rep
averages = {rep: round(sum(amounts) / len(amounts), 1) for rep, amounts in grouped.items()}
print("Averages: " + str(averages))

# Dict comprehension: count per rep
counts = {rep: len(amounts) for rep, amounts in grouped.items()}
print("Counts: " + str(counts))

The group-by pattern (accumulate lists in a dict, then aggregate) is fundamental in data analytics. Dictionary comprehensions then let you compute any aggregate (sum, average, count, max, min) in a single expressive line.

map, filter, and sorted

# Employee data
employees = [
    {"name": "Alice", "salary": 75000, "dept": "Engineering"},
    {"name": "Bob", "salary": 82000, "dept": "Marketing"},
    {"name": "Charlie", "salary": 95000, "dept": "Engineering"},
    {"name": "Diana", "salary": 68000, "dept": "Sales"},
    {"name": "Eve", "salary": 71000, "dept": "Marketing"},
]

# map: Extract just the names
names = list(map(lambda e: e["name"], employees))
print("Names: " + str(names))

# filter: Only engineers
engineers = list(filter(lambda e: e["dept"] == "Engineering", employees))
for eng in engineers:
    print("Engineer: " + eng["name"] + " (quot; + str(eng["salary"]) + ")")

# sorted: By salary descending
by_salary = sorted(employees, key=lambda e: e["salary"], reverse=True)
print("\nBy Salary (highest first):")
for emp in by_salary:
    print("  " + emp["name"] + ": 
quot; + str(emp["salary"]))

map() transforms each element, filter() selects elements matching a condition, and sorted() orders elements by a key. These three functions cover the most common data operations: transformation, selection, and ordering.

zip and enumerate for Parallel Data

# Parallel lists (common in data work)
months = ["Jan", "Feb", "Mar", "Apr", "May"]
revenue = [45000, 52000, 48000, 61000, 58000]
expenses = [30000, 35000, 32000, 38000, 36000]

# zip: Combine parallel sequences
print("Monthly P&L Report:")
for month, rev, exp in zip(months, revenue, expenses):
    profit = rev - exp
    print("  " + month + ": Revenue=
quot; + str(rev) + " Expenses=
quot; + str(exp) + " Profit=quot; + str(profit))

# enumerate: Add ranking with index
print("\nRevenue Ranking:")
sorted_data = sorted(zip(months, revenue), key=lambda x: x[1], reverse=True)
for rank, (month, rev) in enumerate(sorted_data, 1):
    print("  #" + str(rank) + " " + month + ": 
quot; + str(rev))

zip() is essential when data is stored in parallel lists (one list per column). enumerate() adds positional numbering, which is useful for rankings. Combined with sorted(), these tools create powerful data analysis pipelines.

Key points

List comprehensions are generally preferred over map() and filter() in modern Python for readability. However, knowing both styles helps when reading code from different projects and when working with more complex functional pipelines.
The sorted() function always returns a new list and never modifies the original. This immutability is important in data analytics where you want to preserve the original data while creating derived views.
zip() stops at the shortest iterable. If your parallel lists have different lengths, zip() will silently truncate. Use itertools.zip_longest() if you need to handle uneven lengths.

Concepts covered

List Comprehensions, Dictionary Operations, Built-in Functions for Data, Functional Data Processing