Outlier Detection & Handling

Difficulty: Intermediate

Outliers are data points that significantly deviate from the overall pattern of a dataset. They can arise from measurement errors, data entry mistakes, or represent genuine extreme values in the population. Identifying and appropriately handling outliers is one of the most important steps in data analysis because they can dramatically skew summary statistics, distort model training, and lead to misleading conclusions.

The Interquartile Range (IQR) method is one of the most widely used approaches for outlier detection. It defines outliers as values that fall below Q1 - 1.5*IQR or above Q3 + 1.5*IQR, where Q1 and Q3 are the 25th and 75th percentiles respectively. This method is robust because it relies on the median-based spread rather than the mean, making it resistant to the very outliers it is trying to detect.

The z-score method measures how many standard deviations a data point is from the mean. Values with an absolute z-score greater than 3 (or sometimes 2) are typically considered outliers. While simpler to compute, this method is less robust than IQR because the mean and standard deviation themselves are affected by outliers, potentially masking extreme values in heavily skewed data.

Handling outliers requires domain knowledge and careful judgment. Common strategies include removing them (when they are clearly erroneous), capping or winsorizing them (replacing with boundary values), transforming the data (log or square root transforms reduce the effect of extreme values), or using robust statistical methods that are inherently less sensitive to outliers. The choice depends on whether the outliers represent errors or genuine extreme observations.

Code examples

IQR Method for Outlier Detection

import numpy as np

data = [12, 15, 14, 10, 13, 100, 11, 14, 13, 12, 15, 14, -20, 13, 11]

arr = np.array(data)
q1 = np.percentile(arr, 25)
q3 = np.percentile(arr, 75)
iqr = q3 - q1
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr

print(f"Q1: {q1}, Q3: {q3}, IQR: {iqr}")
print(f"Lower bound: {lower_bound}")
print(f"Upper bound: {upper_bound}")
print()

outliers = arr[(arr < lower_bound) | (arr > upper_bound)]
print(f"Outliers found: {outliers.tolist()}")
print(f"Clean data points: {len(arr) - len(outliers)}")

The IQR method identifies 100 and -20 as outliers since they fall outside the [8.25, 18.25] bounds. This approach is robust because Q1, Q3, and IQR are based on percentiles rather than the mean, so existing outliers do not distort the detection thresholds.

Z-Score Method for Outlier Detection

import numpy as np

data = [45, 47, 50, 52, 48, 200, 46, 51, 49, 47, 50, 48, 53, 46, 51]

arr = np.array(data, dtype=float)
mean = np.mean(arr)
std = np.std(arr)

print(f"Mean: {mean:.2f}")
print(f"Std Dev: {std:.2f}")
print()

z_scores = (arr - mean) / std
threshold = 3.0

print("Z-scores for each value:")
for val, z in zip(data, z_scores):
    flag = " <-- OUTLIER" if abs(z) > threshold else ""
    if abs(z) > 2.0:
        print(f"  {val}: z={z:.2f}{flag}")

outlier_count = np.sum(np.abs(z_scores) > threshold)
print(f"\nOutliers (|z| > {threshold}): {outlier_count}")

The z-score method standardizes each value by subtracting the mean and dividing by the standard deviation. Values beyond 3 standard deviations from the mean are flagged as outliers. Only values with |z| > 2.0 are printed to keep the output focused on potentially problematic data points.

Impact of Outliers on Statistics

import numpy as np

# Dataset with an outlier
with_outlier = [25, 28, 30, 27, 26, 29, 31, 28, 200, 27]
without_outlier = [25, 28, 30, 27, 26, 29, 31, 28, 27]

print("With outlier (200):")
print(f"  Mean: {np.mean(with_outlier):.1f}")
print(f"  Median: {np.median(with_outlier):.1f}")
print(f"  Std Dev: {np.std(with_outlier):.1f}")
print()

print("Without outlier:")
print(f"  Mean: {np.mean(without_outlier):.1f}")
print(f"  Median: {np.median(without_outlier):.1f}")
print(f"  Std Dev: {np.std(without_outlier):.1f}")
print()

mean_diff = np.mean(with_outlier) - np.mean(without_outlier)
print(f"Mean shifted by: {mean_diff:.1f}")
print(f"Median shifted by: {np.median(with_outlier) - np.median(without_outlier):.1f}")

A single outlier (200) inflated the mean from 27.9 to 45.1 and the standard deviation from 1.7 to 51.3. The median remained at 28.0, demonstrating why the median is considered a robust measure of central tendency. This is why EDA should always report both mean and median.

Capping (Winsorizing) Outliers

import numpy as np

data = [10, 12, 11, 13, 150, 14, 10, 12, -30, 11, 13, 12]
arr = np.array(data, dtype=float)

q1 = np.percentile(arr, 25)
q3 = np.percentile(arr, 75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr

print(f"Bounds: [{lower:.1f}, {upper:.1f}]")
print(f"Before capping: mean={np.mean(arr):.1f}, std={np.std(arr):.1f}")

capped = np.clip(arr, lower, upper)
print(f"After capping:  mean={np.mean(capped):.1f}, std={np.std(capped):.1f}")
print()

changed = np.sum(arr != capped)
print(f"Values capped: {changed}")

Winsorizing replaces outliers with the nearest boundary value instead of removing them. The value 150 is capped to 17.0 and -30 is capped to 6.75. This preserves the dataset size while eliminating the distorting effect of extreme values, making it a good choice when you cannot afford to lose data points.

Key points

• The 1.5*IQR rule is a convention, not a law. Some analyses use 2.0*IQR or 3.0*IQR for stricter outlier definitions.
• Z-score outlier detection assumes approximately normal data. For heavily skewed distributions, use the IQR method or modified z-scores instead.
• Always investigate outliers before removing them. They may represent the most interesting or valuable data points in your analysis.
• Robust statistics (median, MAD, trimmed mean) are less affected by outliers and should be reported alongside traditional measures.

Concepts covered

IQR method, z-score, outlier impact, handling outliers