Difficulty: Advanced
Machine Learning (ML) is a subset of artificial intelligence that enables systems to learn patterns from data and make predictions or decisions without being explicitly programmed for each case. Instead of writing rules by hand, you provide data to an algorithm and let it discover the underlying structure. The core idea is that a model improves its performance on a task as it is exposed to more data.
There are three main categories of machine learning. Supervised learning uses labeled data where each input has a known output, and the goal is to learn a mapping from inputs to outputs. Examples include predicting house prices (regression) or classifying emails as spam or not (classification). Unsupervised learning works with unlabeled data, seeking to discover hidden patterns such as customer segments (clustering) or reducing the dimensionality of data. Reinforcement learning involves an agent that learns by interacting with an environment and receiving rewards or penalties.
A fundamental concept in ML is the train-test split. You divide your dataset into a training set (typically 70-80%) used to fit the model, and a test set (20-30%) used to evaluate how well the model generalizes to unseen data. This separation is critical because evaluating a model on the same data it was trained on gives an overly optimistic estimate of its true performance.
Overfitting occurs when a model learns the training data too well, including its noise and random fluctuations, resulting in excellent training performance but poor generalization to new data. Underfitting happens when a model is too simple to capture the underlying patterns in the data. The goal is to find the sweet spot where the model captures the true signal without memorizing noise. Techniques like cross-validation, regularization, and early stopping help manage this bias-variance tradeoff.
The typical ML workflow consists of: (1) data collection and preprocessing, (2) exploratory data analysis, (3) feature engineering, (4) model selection and training, (5) evaluation, and (6) deployment. Each step is crucial, and data scientists often iterate between these steps multiple times before arriving at a satisfactory model.
import random
# Sample dataset: (feature, label)
data = [(1, 2), (2, 4), (3, 6), (4, 8), (5, 10),
(6, 12), (7, 14), (8, 16), (9, 18), (10, 20)]
# Shuffle with a fixed seed for reproducibility
random.seed(42)
shuffled = data.copy()
random.shuffle(shuffled)
# 80/20 split
split_idx = int(len(shuffled) * 0.8)
train = shuffled[:split_idx]
test = shuffled[split_idx:]
print(f"Total samples: {len(data)}")
print(f"Training samples: {len(train)}")
print(f"Test samples: {len(test)}")
print(f"Train data: {sorted(train)}")
print(f"Test data: {sorted(test)}")We manually implement a train-test split by shuffling the data with a fixed random seed and then slicing it into training (80%) and test (20%) portions. The fixed seed ensures reproducibility.
# Supervised learning: we have input-output pairs
supervised_data = [
{"size": 1400, "bedrooms": 3, "price": 250000},
{"size": 1600, "bedrooms": 3, "price": 300000},
{"size": 1800, "bedrooms": 4, "price": 350000},
]
print("Supervised Learning (Regression):")
print(" Input features: size, bedrooms")
print(" Target label: price")
print(f" Example: size=1400, bedrooms=3 -> price={supervised_data[0]['price']}")
# Unsupervised learning: no labels, find structure
unsupervised_data = [
{"age": 25, "spending": 200},
{"age": 30, "spending": 180},
{"age": 55, "spending": 400},
{"age": 60, "spending": 450},
]
print("\nUnsupervised Learning (Clustering):")
print(" Features: age, spending")
print(" No target label, goal is to discover groups")
print(f" Data points: {len(unsupervised_data)}")This example contrasts supervised learning (where each data point has a known target value) with unsupervised learning (where we only have features and seek to discover hidden structure like clusters).
# Simple demonstration: memorizing vs generalizing
train_data = {1: 3, 2: 5, 3: 7, 4: 9}
test_data = {5: 11, 6: 13}
# "Overfit" model: memorizes training data exactly (lookup table)
def overfit_model(x):
return train_data.get(x, 0) # Returns 0 for unseen data
# "Good" model: learns the pattern y = 2x + 1
def good_model(x):
return 2 * x + 1
print("=== Training Performance ===")
for x, y in train_data.items():
print(f" x={x}: overfit={overfit_model(x)}, good={good_model(x)}, actual={y}")
print("\n=== Test Performance ===")
for x, y in test_data.items():
print(f" x={x}: overfit={overfit_model(x)}, good={good_model(x)}, actual={y}")The overfit model memorizes training data perfectly but fails on unseen test data (returns 0). The good model learns the true pattern (y = 2x + 1) and generalizes correctly. This illustrates why we need a separate test set to evaluate models.
# Mean Absolute Error calculation
def mean_absolute_error(predictions, actuals):
total = sum(abs(p - a) for p, a in zip(predictions, actuals))
return total / len(predictions)
# Training data
train_x = [1, 2, 3, 4, 5]
train_y = [2.1, 3.9, 6.2, 7.8, 10.1]
# Test data
test_x = [6, 7, 8]
test_y = [12.0, 14.1, 15.9]
# Simple model: y = 2x
train_pred = [2 * x for x in train_x]
test_pred = [2 * x for x in test_x]
train_error = mean_absolute_error(train_pred, train_y)
test_error = mean_absolute_error(test_pred, test_y)
print(f"Train predictions: {train_pred}")
print(f"Train actuals: {train_y}")
print(f"Train MAE: {train_error:.2f}")
print(f"\nTest predictions: {test_pred}")
print(f"Test actuals: {test_y}")
print(f"Test MAE: {test_error:.2f}")We calculate Mean Absolute Error (MAE) on both training and test sets. Train MAE: (|2-2.1|+|4-3.9|+|6-6.2|+|8-7.8|+|10-10.1|)/5 = (0.1+0.1+0.2+0.2+0.1)/5 = 0.7/5 = 0.14. Test MAE: (|12-12.0|+|14-14.1|+|16-15.9|)/3 = (0.0+0.1+0.1)/3 = 0.2/3 = 0.067, rounded to 0.07. A large gap between train and test error often signals overfitting.
ML overview, Supervised vs unsupervised learning, Training and testing, Overfitting and underfitting