Correlation and Covariance

Difficulty: Intermediate

Correlation and covariance are measures that quantify the relationship between two variables. They are fundamental tools in exploratory data analysis, helping you understand how variables move together -- whether they tend to increase and decrease in tandem, move in opposite directions, or have no discernible linear relationship. In data science, understanding correlations is crucial for feature selection, multicollinearity detection, and generating hypotheses about causal relationships.

Covariance measures the direction of the linear relationship between two variables. A positive covariance means they tend to move in the same direction; a negative covariance means they move in opposite directions. However, covariance is difficult to interpret because its magnitude depends on the units of the variables. A covariance of 1000 between height (cm) and weight (kg) tells you the direction but not the strength of the relationship.

The Pearson correlation coefficient solves the interpretability problem by normalizing covariance. It divides the covariance by the product of the standard deviations of both variables, producing a value between -1 and +1. A value of +1 means a perfect positive linear relationship, -1 means a perfect negative linear relationship, and 0 means no linear relationship. Pearson correlation is the most commonly used correlation measure, but it only captures linear relationships -- two variables can have a strong nonlinear relationship with a Pearson correlation near 0.

Spearman rank correlation is a non-parametric alternative that measures the monotonic relationship between variables. Instead of using the raw values, it ranks them and computes the Pearson correlation of the ranks. This makes it robust to outliers and able to capture any monotonic relationship, not just linear ones. For example, if y = x^2 for positive x, the Spearman correlation would be 1 (perfectly monotonic) while the Pearson correlation would be less than 1 (not perfectly linear).

A correlation matrix shows the pairwise correlation coefficients between all variables in a dataset. It is a symmetric matrix with 1s on the diagonal (every variable is perfectly correlated with itself). Correlation matrices are invaluable for identifying redundant features (highly correlated variables), spotting potential predictors, and detecting multicollinearity in regression models. In practice, heatmap visualizations of correlation matrices are one of the first things data scientists create during EDA.

Code examples

Computing Covariance

import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])

# Manual covariance computation
n = len(x)
mean_x = np.mean(x)
mean_y = np.mean(y)
cov_manual = np.sum((x - mean_x) * (y - mean_y)) / (n - 1)

# NumPy covariance matrix
cov_matrix = np.cov(x, y)

print(f"Mean of x: {mean_x}")
print(f"Mean of y: {mean_y}")
print(f"Covariance (manual): {cov_manual}")
print(f"Covariance (numpy): {cov_matrix[0][1]}")

Covariance measures how x and y move together. Deviations from means: x-mean_x = [-2,-1,0,1,2], y-mean_y = [-2,0,1,0,1]. Products sum to 4+0+0+0+2 = 6. Dividing by n-1 = 4 gives 1.5. Positive covariance means they tend to increase together.

Pearson Correlation Coefficient

import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])

# Manual Pearson correlation
cov_xy = np.cov(x, y)[0][1]
std_x = np.std(x, ddof=1)
std_y = np.std(y, ddof=1)
pearson_manual = cov_xy / (std_x * std_y)

# NumPy corrcoef
pearson_np = np.corrcoef(x, y)[0][1]

print(f"Std of x: {round(std_x, 4)}")
print(f"Std of y: {round(std_y, 4)}")
print(f"Pearson (manual): {round(pearson_manual, 4)}")
print(f"Pearson (numpy): {round(pearson_np, 4)}")

Pearson correlation = covariance / (std_x * std_y) = 1.5 / (1.5811 * 1.2247) = 0.7746. This indicates a strong positive linear relationship. The value is between -1 and 1, making it much easier to interpret than raw covariance.

Spearman Rank Correlation

import numpy as np

def spearman_correlation(x, y):
    """Compute Spearman rank correlation manually"""
    n = len(x)
    # Rank the data (average ranks for ties)
    def rank_data(arr):
        sorted_indices = np.argsort(arr)
        ranks = np.empty(n)
        for i, idx in enumerate(sorted_indices):
            ranks[idx] = i + 1
        return ranks
    
    rank_x = rank_data(x)
    rank_y = rank_data(y)
    
    # Pearson correlation of ranks
    return np.corrcoef(rank_x, rank_y)[0][1]

x = np.array([1, 2, 3, 4, 5])
y = np.array([1, 8, 27, 64, 125])  # y = x^3 (monotonic but nonlinear)

pearson = np.corrcoef(x, y)[0][1]
spearman = spearman_correlation(x, y)

print(f"Pearson correlation: {round(pearson, 4)}")
print(f"Spearman correlation: {round(spearman, 4)}")

The relationship y = x^3 is perfectly monotonic but not perfectly linear. Spearman captures this perfectly (1.0) because the ranks are perfectly aligned. Pearson gives 0.9431 because the raw values do not fall on a straight line. This demonstrates when Spearman is more appropriate.

Correlation Matrix

import numpy as np

np.random.seed(42)

# Generate correlated data
n = 1000
x1 = np.random.normal(0, 1, n)
x2 = 0.8 * x1 + 0.2 * np.random.normal(0, 1, n)
x3 = -0.5 * x1 + 0.5 * np.random.normal(0, 1, n)

data = np.column_stack([x1, x2, x3])
corr_matrix = np.corrcoef(data, rowvar=False)

labels = ['x1', 'x2', 'x3']
print("Correlation Matrix:")
print(f"{'':>4}", end="")
for label in labels:
    print(f"{label:>8}", end="")
print()
for i, label in enumerate(labels):
    print(f"{label:>4}", end="")
    for j in range(len(labels)):
        print(f"{corr_matrix[i][j]:>8.4f}", end="")
    print()

The correlation matrix reveals that x1 and x2 are strongly positively correlated (0.97) as expected since x2 is derived from x1. x1 and x3 are negatively correlated (-0.70) because x3 has a negative dependency on x1. The diagonal is always 1 (self-correlation).

Key points

Concepts covered

Pearson correlation, Spearman rank correlation, Covariance, Correlation matrix