Handling Missing Data

Difficulty: Intermediate

Missing data is one of the most common problems you will encounter in real-world datasets. Values can be absent for many reasons: a sensor malfunction, a user skipping a form field, a failed API call, or data corruption during transfer. In pandas, missing values are represented by NaN (Not a Number) from NumPy, or by Python's None. Understanding how to detect, analyze, and handle missing data is a critical skill for any data analyst or scientist, because the strategy you choose directly affects the quality and reliability of your analysis.

Detecting missing values is the first step. Pandas provides isnull() (and its alias isna()) to create a Boolean mask that is True wherever a value is missing. You can chain this with sum() to count missing values per column, or with any() to check whether any column contains nulls. The info() method also gives a quick overview showing the count of non-null entries per column. Before deciding how to handle missing data, you should always understand the pattern: are values missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR)? This classification guides your imputation strategy.

The simplest approach is to remove rows or columns containing missing values using dropna(). By default, dropna() removes any row that has at least one NaN. You can pass axis=1 to drop columns instead, how='all' to drop only rows where every value is NaN, or thresh=N to keep rows that have at least N non-null values. Dropping data is appropriate when the percentage of missing values is very small (typically less than 5%) and the missingness is random, but it can introduce bias if the missing data is systematic.

A more nuanced approach is imputation with fillna(). You can fill missing values with a constant, the column mean, median, mode, or a forward/backward fill strategy (method='ffill' or method='bfill'). Forward fill propagates the last valid observation forward, which is especially useful for time series data. For numeric columns with a natural ordering, the interpolate() method can estimate missing values by fitting between known data points using linear or polynomial interpolation. The choice between mean, median, and mode depends on the distribution: use median for skewed data, mean for normally distributed data, and mode for categorical columns.

In practice, you will often use different strategies for different columns within the same DataFrame. Categorical columns might be filled with the mode or a placeholder like 'Unknown', while numeric columns might use the median. For time series, forward fill or interpolation preserves temporal patterns better than a global statistic. Always document your imputation strategy and consider its impact on downstream analysis, especially variance and correlations, which can be artificially reduced by naive imputation.

Code examples

Detecting Missing Values

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Name': ['Alice', 'Bob', None, 'Diana'],
    'Age': [25, np.nan, 30, np.nan],
    'Salary': [50000, 60000, np.nan, 70000]
})

print("DataFrame:")
print(df)
print("\nMissing values per column:")
print(df.isnull().sum())
print("\nTotal missing values:", df.isnull().sum().sum())

isnull() returns a DataFrame of booleans. Chaining with sum() counts True values per column. Calling sum() again on the result gives the grand total of missing values across the entire DataFrame.

Dropping Missing Values with dropna()

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A': [1, np.nan, 3, 4],
    'B': [np.nan, 2, 3, 4],
    'C': [1, 2, np.nan, 4]
})

print("Original:")
print(df)
print("\nDrop rows with any NaN:")
print(df.dropna())
print("\nDrop rows where all values are NaN:")
print(df.dropna(how='all'))
print("\nKeep rows with at least 2 non-null values:")
print(df.dropna(thresh=2))

dropna() with default parameters removes any row containing at least one NaN. The how='all' parameter only drops rows where every single value is missing. The thresh parameter provides fine-grained control by requiring a minimum number of non-null values to keep a row.

Filling Missing Values with fillna()

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Temperature': [22.0, np.nan, 25.0, np.nan, 28.0],
    'City': ['Delhi', None, 'Mumbai', 'Delhi', None]
})

print("Original:")
print(df)

# Fill numeric column with median
median_temp = df['Temperature'].median()
df['Temperature'] = df['Temperature'].fillna(median_temp)

# Fill categorical column with mode
mode_city = df['City'].mode()[0]
df['City'] = df['City'].fillna(mode_city)

print("\nAfter fillna:")
print(df)

Numeric columns are filled with the median (25.0), which is robust to outliers. Categorical columns are filled with the mode (most frequent value), which is 'Delhi'. Using different strategies per column type is standard practice in data cleaning.

Forward Fill and Interpolation

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Day': [1, 2, 3, 4, 5, 6],
    'Value': [10.0, np.nan, np.nan, 40.0, np.nan, 60.0]
})

print("Original:")
print(df['Value'].tolist())

print("\nForward fill:")
print(df['Value'].ffill().tolist())

print("\nLinear interpolation:")
print(df['Value'].interpolate().tolist())

Forward fill (ffill) carries the last known value forward, producing flat segments. Linear interpolation estimates values by drawing a straight line between known points, giving a smoother and often more accurate result for time series and sequential data.

Key points

Concepts covered

NaN, isnull, dropna, fillna, interpolate