EDA Workflow & Data Profiling

Difficulty: Intermediate

Exploratory Data Analysis (EDA) is the critical first step in any data science project. It involves systematically examining datasets to understand their structure, identify patterns, detect anomalies, and form hypotheses before applying formal modeling techniques. A well-executed EDA can save hours of debugging and prevent flawed conclusions later in the analysis pipeline.

The standard EDA workflow follows a structured sequence: first, load and inspect the data shape and types; second, compute summary statistics to understand distributions; third, check for missing values and data quality issues; fourth, examine relationships between variables; and finally, document findings and form hypotheses. This systematic approach ensures nothing is overlooked and provides a solid foundation for downstream analysis.

Pandas provides a powerful suite of profiling tools that make the initial exploration phase efficient. The `info()` method reveals column types and non-null counts at a glance, while `describe()` computes key statistical measures across all numeric columns. The `value_counts()` method is indispensable for understanding categorical distributions, and `shape` gives you the dataset dimensions immediately.

Beyond individual column analysis, a thorough EDA examines relationships between variables through correlation matrices, cross-tabulations, and grouped aggregations. The goal is to build an intuition about the data that guides feature engineering, model selection, and interpretation of results. Seasoned data scientists treat EDA as an iterative process, revisiting earlier steps as new patterns emerge.

Code examples

Dataset Shape and Structure Inspection

import pandas as pd
import numpy as np

# Create a sample dataset
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'age': [25, 30, 35, 28, 32],
    'salary': [50000, 65000, 80000, 55000, 72000],
    'department': ['Engineering', 'Marketing', 'Engineering', 'HR', 'Marketing'],
    'years_exp': [2, 5, 10, 3, 7]
}
df = pd.DataFrame(data)

# Step 1: Check shape
print(f"Shape: {df.shape}")
print(f"Rows: {df.shape[0]}, Columns: {df.shape[1]}")
print()

# Step 2: Column types
print("Column Types:")
for col in df.columns:
    print(f"  {col}: {df[col].dtype}")

The first step in any EDA is understanding the dataset dimensions and column types. The shape attribute gives you (rows, columns) immediately. Knowing column dtypes helps you determine which columns are numeric vs categorical, guiding your analysis strategy.

Summary Statistics with describe()

import pandas as pd
import numpy as np

data = {
    'age': [25, 30, 35, 28, 32, 45, 22, 38],
    'salary': [50000, 65000, 80000, 55000, 72000, 95000, 42000, 78000],
    'rating': [4.2, 3.8, 4.5, 3.9, 4.1, 4.8, 3.5, 4.3]
}
df = pd.DataFrame(data)

stats = df.describe()
print("Summary Statistics:")
print(f"  Mean age: {stats.loc['mean', 'age']:.1f}")
print(f"  Median salary: {stats.loc['50%', 'salary']:.0f}")
print(f"  Salary std dev: {stats.loc['std', 'salary']:.0f}")
print(f"  Min rating: {stats.loc['min', 'rating']:.1f}")
print(f"  Max rating: {stats.loc['max', 'rating']:.1f}")
print()

# Quick spread check
for col in df.columns:
    iqr = df[col].quantile(0.75) - df[col].quantile(0.25)
    print(f"{col} IQR: {iqr:.1f}")

The describe() method computes count, mean, std, min, 25th/50th/75th percentiles, and max for all numeric columns. The IQR (interquartile range) is the difference between Q3 and Q1 and measures the spread of the middle 50% of data, which is useful for identifying outliers.

Categorical Analysis with value_counts()

import pandas as pd

data = {
    'department': ['Engineering', 'Marketing', 'Engineering', 'HR', 'Marketing',
                   'Engineering', 'HR', 'Engineering', 'Marketing', 'Finance'],
    'level': ['Junior', 'Senior', 'Mid', 'Junior', 'Mid',
              'Senior', 'Mid', 'Junior', 'Senior', 'Junior']
}
df = pd.DataFrame(data)

# Value counts for department
print("Department Distribution:")
counts = df['department'].value_counts()
for dept, count in counts.items():
    pct = count / len(df) * 100
    print(f"  {dept}: {count} ({pct:.0f}%)")

print()

# Cross-tabulation
print("Department x Level:")
ct = pd.crosstab(df['department'], df['level'])
for dept in ct.index:
    parts = [f"{lvl}={ct.loc[dept, lvl]}" for lvl in ct.columns]
    print(f"  {dept}: {', '.join(parts)}")

value_counts() sorts categories by frequency, making it easy to spot dominant and rare categories. Cross-tabulation reveals how two categorical variables interact, which is essential for understanding relationships in categorical data.

Missing Data Assessment

import pandas as pd
import numpy as np

data = {
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank'],
    'age': [25, np.nan, 35, 28, np.nan, 40],
    'salary': [50000, 65000, np.nan, 55000, 72000, np.nan],
    'department': ['Eng', 'Mkt', 'Eng', None, 'Mkt', 'HR']
}
df = pd.DataFrame(data)

# Missing value summary
print("Missing Values Report:")
print(f"  Total cells: {df.size}")
print(f"  Missing cells: {df.isnull().sum().sum()}")
print(f"  Missing percentage: {df.isnull().sum().sum() / df.size * 100:.1f}%")
print()

# Per-column missing
print("Per Column:")
for col in df.columns:
    missing = df[col].isnull().sum()
    pct = missing / len(df) * 100
    print(f"  {col}: {missing} missing ({pct:.1f}%)")
print()

# Rows with any missing value
rows_with_missing = df.isnull().any(axis=1).sum()
print(f"Rows with missing data: {rows_with_missing}/{len(df)}")

A thorough missing data assessment checks both column-level and row-level missingness. High missing percentages in a column may warrant dropping it entirely, while sporadic missing values might be imputed. Understanding the pattern of missingness (random vs systematic) is crucial for choosing the right handling strategy.

Key points

Concepts covered

EDA process, data profiling, summary statistics, initial exploration