Difficulty: Advanced
Hypothesis testing is a systematic framework for making decisions about populations based on sample data. It is one of the most widely used statistical methods in data science, powering everything from A/B tests at tech companies to clinical trials in medicine. The fundamental idea is simple: you start with an assumption (the null hypothesis), collect data, and determine whether the data provides enough evidence to reject that assumption.
The process begins by formulating two hypotheses. The null hypothesis (H0) represents the status quo or 'no effect' -- for example, 'there is no difference between group A and group B.' The alternative hypothesis (H1 or Ha) is what you want to prove -- for example, 'group A performs better than group B.' The null hypothesis is assumed true until the data provides strong evidence against it. This is analogous to 'innocent until proven guilty' in legal systems.
The p-value is the probability of observing data as extreme as (or more extreme than) your sample data, assuming the null hypothesis is true. A small p-value means the observed data would be very unlikely under the null hypothesis, suggesting the null hypothesis may be wrong. The significance level (alpha, typically 0.05) is the threshold: if p-value < alpha, you reject the null hypothesis. A p-value of 0.03 means there is only a 3% chance of seeing data this extreme if H0 were true -- which is considered sufficient evidence to reject H0 at the 5% significance level.
The t-test is used to compare means. A one-sample t-test checks if a sample mean differs from a hypothesized value. A two-sample t-test compares means of two groups. The test computes a t-statistic that measures how many standard errors the sample mean is from the hypothesized mean. The t-statistic is then converted to a p-value using the t-distribution. For large samples (n > 30), the t-distribution approximates the normal distribution.
The chi-square test is used for categorical data. The chi-square goodness-of-fit test checks if observed frequencies match expected frequencies. The chi-square test of independence checks if two categorical variables are associated. It computes a chi-square statistic by comparing observed and expected counts in a contingency table. Like the t-test, this statistic is converted to a p-value. Type I error (false positive) occurs when you reject a true null hypothesis, and Type II error (false negative) occurs when you fail to reject a false null hypothesis. Balancing these errors is a key consideration in experimental design.
import math
import statistics
# Sample data: daily website visitors
data = [105, 112, 98, 110, 108, 115, 103, 109, 111, 107]
# H0: mean = 100 (old average)
# H1: mean != 100 (two-tailed)
mu_0 = 100
n = len(data)
sample_mean = statistics.mean(data)
sample_std = statistics.stdev(data)
se = sample_std / math.sqrt(n)
t_stat = (sample_mean - mu_0) / se
print(f"Sample mean: {sample_mean}")
print(f"Sample std: {round(sample_std, 4)}")
print(f"Standard error: {round(se, 4)}")
print(f"T-statistic: {round(t_stat, 4)}")
print(f"Degrees of freedom: {n - 1}")The t-statistic of 5.06 is very large, meaning the sample mean of 107.8 is about 5 standard errors away from the hypothesized mean of 100. With 9 degrees of freedom, this corresponds to a very small p-value (well below 0.05), so we would reject H0 and conclude the mean has changed.
import math
import statistics
# Group A: control, Group B: treatment
group_a = [78, 82, 85, 80, 76, 79, 83, 81]
group_b = [88, 92, 85, 90, 87, 91, 89, 93]
mean_a = statistics.mean(group_a)
mean_b = statistics.mean(group_b)
std_a = statistics.stdev(group_a)
std_b = statistics.stdev(group_b)
n_a = len(group_a)
n_b = len(group_b)
# Welch's t-test (unequal variances)
se = math.sqrt((std_a**2 / n_a) + (std_b**2 / n_b))
t_stat = (mean_a - mean_b) / se
print(f"Group A: mean={mean_a}, std={round(std_a, 4)}")
print(f"Group B: mean={mean_b}, std={round(std_b, 4)}")
print(f"Difference: {round(mean_b - mean_a, 4)}")
print(f"T-statistic: {round(t_stat, 4)}")The large negative t-statistic (-6.39) indicates a significant difference between the groups. Group B (mean 89.375) performed much better than Group A (mean 80.5). In an A/B test, this would be strong evidence that the treatment had a positive effect.
# Test if a die is fair
# Observed frequencies from 120 rolls
observed = [25, 17, 22, 18, 19, 19]
expected = [20, 20, 20, 20, 20, 20] # Fair die: 120/6 = 20 each
# Chi-square statistic: sum of (O - E)^2 / E
chi2 = sum((o - e) ** 2 / e for o, e in zip(observed, expected))
df = len(observed) - 1 # degrees of freedom
print("Face | Observed | Expected | (O-E)^2/E")
for i in range(6):
contrib = (observed[i] - expected[i]) ** 2 / expected[i]
print(f" {i+1} | {observed[i]:2d} | {expected[i]} | {contrib:.2f}")
print(f"\nChi-square statistic: {chi2}")
print(f"Degrees of freedom: {df}")
print(f"Critical value (alpha=0.05, df=5): 11.07")
print(f"Reject H0: {chi2 > 11.07}")The chi-square statistic (2.2) is well below the critical value (11.07 for df=5, alpha=0.05). We fail to reject H0: there is not enough evidence to conclude the die is unfair. The observed frequencies are within the range of normal random variation.
# Framework for interpreting p-values
def interpret_p_value(p_value, alpha=0.05):
if p_value < 0.001:
strength = "Very strong evidence against H0"
elif p_value < 0.01:
strength = "Strong evidence against H0"
elif p_value < 0.05:
strength = "Moderate evidence against H0"
elif p_value < 0.1:
strength = "Weak evidence against H0"
else:
strength = "No evidence against H0"
decision = "Reject H0" if p_value < alpha else "Fail to reject H0"
return strength, decision
test_cases = [0.001, 0.008, 0.03, 0.07, 0.45]
for p in test_cases:
strength, decision = interpret_p_value(p)
print(f"p={p:.3f} -> {decision} | {strength}")This framework shows how p-values map to evidence strength. Note that 'fail to reject H0' is NOT the same as 'accept H0' -- we simply lack sufficient evidence to reject it. Also, a p-value just above 0.05 (like 0.07) still suggests some evidence against H0 -- the 0.05 cutoff is a convention, not a magic boundary.
Null hypothesis, P-value, Significance level, T-test, Chi-square test