Descriptive Statistics

Difficulty: Intermediate

Descriptive statistics are numerical measures that summarize and describe the main features of a dataset. They provide a way to condense large amounts of data into a few meaningful numbers, making it easier to understand patterns, central tendencies, and the spread of data. Descriptive statistics form the bedrock of any data analysis workflow -- before building models or running experiments, you must first understand what your data looks like.

The three most common measures of central tendency are the mean, median, and mode. The mean (arithmetic average) is calculated by summing all values and dividing by the count. The median is the middle value when data is sorted in order -- if there is an even number of observations, it is the average of the two middle values. The mode is the value that appears most frequently. Each measure has strengths: the mean uses all data points but is sensitive to outliers, the median is robust to outliers, and the mode captures the most common observation.

Measures of spread (or dispersion) tell you how spread out the data is. The range is the simplest measure: max minus min. Variance measures the average squared deviation from the mean, and standard deviation is its square root. A low standard deviation means data points cluster tightly around the mean, while a high standard deviation indicates wide spread. In Python, the statistics module uses sample variance (dividing by n-1) by default, which is the correct choice for estimating population variance from a sample.

Python's built-in statistics module provides functions like mean(), median(), mode(), variance(), stdev(), and more. For larger datasets or array-based workflows, numpy offers np.mean(), np.median(), np.var(), and np.std(). One critical difference: numpy computes population variance/std by default (ddof=0), while the statistics module computes sample variance/std (dividing by n-1). You can control this in numpy using the ddof parameter.

Understanding when to use each statistic is crucial. For symmetric distributions without outliers, the mean is ideal. For skewed data or data with outliers (like income distributions), the median is a better measure of center. The mode is most useful for categorical data or when you need to identify the most common category. In practice, reporting both the mean and median together can reveal skewness in your data.

Code examples

Computing Mean, Median, and Mode

import statistics

data = [4, 8, 6, 5, 3, 8, 9, 2, 8]

mean_val = statistics.mean(data)
median_val = statistics.median(data)
mode_val = statistics.mode(data)

print(f"Mean: {mean_val}")
print(f"Median: {median_val}")
print(f"Mode: {mode_val}")

The mean is 53/9 = 5.889. When sorted [2,3,4,5,6,8,8,8,9], the middle value (5th of 9) is 6. The mode is 8 because it appears three times, more than any other value.

Variance and Standard Deviation

import statistics

data = [10, 12, 23, 23, 16, 23, 21, 16]

variance = statistics.variance(data)
stdev = statistics.stdev(data)
pvariance = statistics.pvariance(data)
pstdev = statistics.pstdev(data)

print(f"Sample variance: {variance}")
print(f"Sample std dev: {round(stdev, 4)}")
print(f"Population variance: {pvariance}")
print(f"Population std dev: {round(pstdev, 4)}")

Sample variance divides by n-1 (7), population variance divides by n (8). The mean is 144/8 = 18. Sum of squared deviations = 192. Population variance = 192/8 = 24.0, sample variance = 192/7 = 27.429. Standard deviation is the square root of variance.

Range and Quantiles

import statistics

data = [15, 20, 35, 40, 50, 55, 60, 70, 80, 95]

data_range = max(data) - min(data)
q1 = statistics.quantiles(data, n=4)[0]
q2 = statistics.quantiles(data, n=4)[1]
q3 = statistics.quantiles(data, n=4)[2]
iqr = q3 - q1

print(f"Range: {data_range}")
print(f"Q1: {q1}")
print(f"Q2 (Median): {q2}")
print(f"Q3: {q3}")
print(f"IQR: {iqr}")

The range is 95 - 15 = 80. The quantiles function splits the data into four equal parts. Q1 is the 25th percentile, Q2 is the median, Q3 is the 75th percentile. The interquartile range (IQR) measures the spread of the middle 50% of the data.

Using NumPy for Descriptive Statistics

import numpy as np

data = np.array([4, 8, 6, 5, 3, 8, 9, 2, 8])

print(f"Mean: {np.mean(data)}")
print(f"Median: {np.median(data)}")
print(f"Std (population): {round(np.std(data), 4)}")
print(f"Std (sample): {round(np.std(data, ddof=1), 4)}")
print(f"Min: {np.min(data)}, Max: {np.max(data)}")

NumPy uses population standard deviation by default (ddof=0). Pass ddof=1 to get sample standard deviation, which matches the statistics module's stdev(). For data analysis, sample std (ddof=1) is typically used when your data is a sample from a larger population.

Key points

Concepts covered

Mean, Median, Mode, Range, Variance, Standard deviation