Difficulty: Advanced
Feature engineering is arguably the most important step in the machine learning pipeline. It is the process of transforming raw data into features that better represent the underlying problem, enabling algorithms to learn more effectively. As the saying goes in data science: 'garbage in, garbage out.' Even the most sophisticated algorithm will fail if the features fed to it are poorly constructed or improperly preprocessed.
Feature scaling is essential for many ML algorithms, particularly those based on distance calculations (KNN, SVM) or gradient descent (linear regression, neural networks). There are two main approaches: min-max normalization scales features to a fixed range (typically 0 to 1) using the formula (x - min) / (max - min). Standardization (z-score normalization) transforms features to have zero mean and unit variance using (x - mean) / std. Normalization is preferred when the data does not follow a Gaussian distribution, while standardization is better when the data is approximately normal or when algorithms assume standardized input.
Categorical variables (non-numeric data like colors, cities, or product types) must be converted to numeric form before most ML algorithms can use them. Label encoding assigns a unique integer to each category (e.g., red=0, green=1, blue=2), but this can introduce a false ordinal relationship. One-hot encoding creates a binary column for each category, avoiding the ordinal problem. For example, a 'color' column with values [red, green, blue] becomes three columns: is_red, is_green, is_blue, each containing 0 or 1.
Beyond scaling and encoding, feature engineering includes creating new features from existing ones (e.g., extracting day-of-week from a date, computing ratios between columns), handling missing values (imputation with mean, median, or mode), removing or transforming outliers, and polynomial features (creating x^2, x*y terms for non-linear relationships). Domain knowledge is invaluable here, knowing which features are likely to be predictive can dramatically improve model performance.
A critical rule in feature engineering is to fit preprocessing transformations only on the training data and then apply the same transformation to the test data. For example, when standardizing features, you compute the mean and standard deviation from the training set and use those exact values to transform both training and test sets. Computing statistics on the test set would leak information and give an overly optimistic estimate of model performance.
def min_max_normalize(data):
min_val = min(data)
max_val = max(data)
return [(x - min_val) / (max_val - min_val) for x in data]
# Example: normalizing age and salary to [0, 1]
ages = [22, 25, 30, 35, 60]
salaries = [30000, 45000, 55000, 70000, 120000]
norm_ages = min_max_normalize(ages)
norm_salaries = min_max_normalize(salaries)
print("Ages:")
for original, normalized in zip(ages, norm_ages):
print(f" {original:>3} -> {normalized:.4f}")
print("\nSalaries:")
for original, normalized in zip(salaries, norm_salaries):
print(f" {original:>6} -> {normalized:.4f}")Min-max normalization maps all values to [0, 1]. For ages: range is 60-22=38. (25-22)/38 = 3/38 = 0.0789. For salaries: range is 120000-30000=90000. (45000-30000)/90000 = 15000/90000 = 0.1667. Now both features are on the same scale.
import math
def standardize(data):
mean = sum(data) / len(data)
variance = sum((x - mean) ** 2 for x in data) / len(data)
std = math.sqrt(variance)
return [(x - mean) / std for x in data], mean, std
values = [10, 20, 30, 40, 50]
standardized, mean, std = standardize(values)
print(f"Original: {values}")
print(f"Mean: {mean:.2f}")
print(f"Std Dev: {std:.2f}")
print(f"Standardized: [{', '.join(f'{v:.2f}' for v in standardized)}]")
# Verify: mean of standardized should be ~0, std should be ~1
new_mean = sum(standardized) / len(standardized)
new_std = math.sqrt(sum((x - new_mean) ** 2 for x in standardized) / len(standardized))
print(f"\nVerification:")
print(f" New mean: {new_mean:.2f}")
print(f" New std: {new_std:.2f}")Standardization centers data at mean=0 with std=1. Mean of [10,20,30,40,50] = 30. Variance = (400+100+0+100+400)/5 = 200. Std = sqrt(200) = 14.14. Each value: (x-30)/14.14. So 10 becomes (10-30)/14.14 = -1.41.
def one_hot_encode(data, column_name):
categories = sorted(set(data))
encoded = {}
for cat in categories:
col_name = f"{column_name}_{cat}"
encoded[col_name] = [1 if val == cat else 0 for val in data]
return encoded
colors = ['red', 'blue', 'green', 'red', 'blue']
encoded = one_hot_encode(colors, 'color')
print("Original:", colors)
print("\nOne-Hot Encoded:")
for col, values in encoded.items():
print(f" {col}: {values}")One-hot encoding creates a binary column for each unique category. Each row has exactly one 1 (the active category) and 0s elsewhere. This avoids imposing a false ordinal relationship between categories.
import math
def preprocess_dataset(train_data, test_data):
"""Standardize numeric features using train statistics."""
n_features = len(train_data[0])
# Compute mean and std from training data only
stats = []
for j in range(n_features):
col = [row[j] for row in train_data]
mean = sum(col) / len(col)
std = math.sqrt(sum((x - mean) ** 2 for x in col) / len(col))
stats.append((mean, std))
def transform(data, stats):
result = []
for row in data:
new_row = []
for j, val in enumerate(row):
mean, std = stats[j]
new_row.append(round((val - mean) / std, 4) if std > 0 else 0)
result.append(new_row)
return result
return transform(train_data, stats), transform(test_data, stats)
train = [[100, 1], [200, 2], [300, 3], [400, 4]]
test = [[150, 1.5], [350, 3.5]]
train_scaled, test_scaled = preprocess_dataset(train, test)
print("Train (original -> scaled):")
for orig, scaled in zip(train, train_scaled):
print(f" {orig} -> {scaled}")
print("\nTest (original -> scaled):")
for orig, scaled in zip(test, test_scaled):
print(f" {orig} -> {scaled}")The preprocessing pipeline computes mean and std from training data only (mean=250, std=sqrt(12500)=111.80 for feature 1; mean=2.5, std=sqrt(1.25)=1.118 for feature 2), then applies the same transformation to both train and test sets. This prevents data leakage.
Feature scaling, Encoding categorical variables, One-hot encoding, Normalization, Standardization