Difficulty: Advanced
Evaluating a machine learning model goes far beyond simple accuracy. While accuracy (correct predictions / total predictions) is intuitive, it can be deeply misleading in real-world scenarios. Consider a fraud detection system where only 1% of transactions are fraudulent: a model that always predicts 'not fraud' would achieve 99% accuracy while being completely useless. This is why we need a richer set of evaluation metrics.
The confusion matrix is the foundation for understanding classification performance. For binary classification, it is a 2x2 matrix with four values: True Positives (TP), correctly predicted positive, False Positives (FP), incorrectly predicted positive, True Negatives (TN), correctly predicted negative, and False Negatives (FN), incorrectly predicted negative. Every classification metric can be derived from these four numbers.
Precision answers the question: 'Of all the instances the model predicted as positive, how many were actually positive?' It is computed as TP / (TP + FP). Precision is important when the cost of false positives is high, for example, in spam filtering, marking a legitimate email as spam (false positive) is costly. Recall (also called sensitivity or true positive rate) answers: 'Of all actual positive instances, how many did the model correctly identify?' It is computed as TP / (TP + FN). Recall is critical when missing positive cases is dangerous, such as in cancer screening.
The F1 score is the harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). It provides a single metric that balances both concerns. The harmonic mean is used instead of the arithmetic mean because it penalizes extreme imbalances, if either precision or recall is very low, the F1 score will also be low. F1 is particularly useful when you need a single number to compare models and the class distribution is imbalanced.
The ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (recall) against the False Positive Rate (FP / (FP + TN)) at various classification thresholds. The AUC (Area Under the ROC Curve) summarizes this curve as a single number between 0 and 1, where 1.0 represents a perfect classifier and 0.5 represents a random classifier. AUC is threshold-independent, making it useful for comparing models regardless of the specific decision threshold chosen.
def confusion_matrix(y_true, y_pred, positive=1):
tp = sum(1 for t, p in zip(y_true, y_pred) if t == positive and p == positive)
fp = sum(1 for t, p in zip(y_true, y_pred) if t != positive and p == positive)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == positive and p != positive)
tn = sum(1 for t, p in zip(y_true, y_pred) if t != positive and p != positive)
return tp, fp, fn, tn
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 0, 0]
tp, fp, fn, tn = confusion_matrix(y_true, y_pred)
print("Confusion Matrix:")
print(f" Predicted Pos Predicted Neg")
print(f"Actual Pos TP={tp} FN={fn}")
print(f"Actual Neg FP={fp} TN={tn}")
print(f"\nTotal: {tp+fp+fn+tn} samples")The confusion matrix breaks down predictions into four categories. TP=3 (correctly identified positives), FP=1 (negative sample incorrectly predicted as positive), FN=2 (positive samples missed), TN=4 (correctly identified negatives).
def compute_metrics(y_true, y_pred, positive=1):
tp = sum(1 for t, p in zip(y_true, y_pred) if t == positive and p == positive)
fp = sum(1 for t, p in zip(y_true, y_pred) if t != positive and p == positive)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == positive and p != positive)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
accuracy = sum(1 for t, p in zip(y_true, y_pred) if t == p) / len(y_true)
return accuracy, precision, recall, f1
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 0, 0]
acc, prec, rec, f1 = compute_metrics(y_true, y_pred)
print(f"Accuracy: {acc:.4f}")
print(f"Precision: {prec:.4f}")
print(f"Recall: {rec:.4f}")
print(f"F1 Score: {f1:.4f}")With TP=3, FP=1, FN=2, TN=4: Accuracy = (3+4)/10 = 0.70. Precision = 3/(3+1) = 0.75. Recall = 3/(3+2) = 0.60. F1 = 2*0.75*0.6/(0.75+0.6) = 0.9/1.35 = 0.6667.
# Simulating different thresholds on model confidence scores
scores = [0.9, 0.8, 0.7, 0.6, 0.4, 0.35, 0.3, 0.2, 0.15, 0.1]
y_true = [1, 1, 1, 1, 0, 1, 0, 0, 0, 0]
print(f"{'Threshold':>10} {'Precision':>10} {'Recall':>8} {'F1':>8}")
print("-" * 40)
for threshold in [0.8, 0.5, 0.3]:
y_pred = [1 if s >= threshold else 0 for s in scores]
tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
print(f"{threshold:>10.1f} {prec:>10.4f} {rec:>8.4f} {f1:>8.4f}")At threshold 0.8, only the top 2 scores (0.9, 0.8) are predicted positive (both truly positive), so precision is perfect but recall is low (2 of 5 positives caught). At 0.5, five predictions are positive (scores 0.9,0.8,0.7,0.6,0.55, all are true positives), precision=4/4=1.0, recall=4/5=0.8. At 0.3, seven predictions are positive (5 TP + 2 FP), precision=5/7=0.714, recall=5/5=1.0. This shows the classic tradeoff: lower threshold catches more positives but also more false positives.
from collections import defaultdict
def multi_class_report(y_true, y_pred):
classes = sorted(set(y_true))
# Build confusion matrix
matrix = defaultdict(lambda: defaultdict(int))
for t, p in zip(y_true, y_pred):
matrix[t][p] += 1
print("Confusion Matrix:")
print(f"{'':>10}", end="")
for c in classes:
print(f"{'Pred_'+c:>10}", end="")
print()
for actual in classes:
print(f"{'Act_'+actual:>10}", end="")
for pred in classes:
print(f"{matrix[actual][pred]:>10}", end="")
print()
# Per-class accuracy
overall_correct = sum(1 for t, p in zip(y_true, y_pred) if t == p)
print(f"\nOverall Accuracy: {overall_correct}/{len(y_true)} = {overall_correct/len(y_true):.1%}")
y_true = ['cat', 'dog', 'cat', 'bird', 'dog', 'bird', 'cat', 'dog', 'bird']
y_pred = ['cat', 'dog', 'dog', 'bird', 'dog', 'cat', 'cat', 'dog', 'bird']
multi_class_report(y_true, y_pred)The multi-class confusion matrix shows how each actual class was predicted. The diagonal (2, 2, 3) represents correct predictions. Off-diagonal elements show misclassifications: one bird was predicted as cat, and one cat was predicted as dog.
Accuracy, Precision, Recall, F1 score, Confusion matrix, ROC and AUC