Difficulty: Intermediate
Data quality assessment is the systematic process of evaluating datasets across multiple dimensions to determine their fitness for a specific purpose. Poor data quality is one of the most common reasons analytics projects fail, and studies consistently show that data scientists spend 60-80% of their time cleaning and preparing data. A structured quality assessment framework helps prioritize cleaning efforts and quantify the overall reliability of your dataset.
Completeness measures the proportion of non-missing values in the dataset. While some missingness is inevitable, the pattern and extent of missing data matter enormously. A column missing 5% of values randomly is very different from one where all missing values come from a specific demographic group. Completeness scores should be computed at both the column level and the row level to understand the full picture.
Consistency checks verify that data values follow expected rules and do not contradict each other. Examples include ensuring that end dates come after start dates, that age values are positive and reasonable, that email addresses follow a valid format, and that related fields agree (e.g., a city matches its stated country). Inconsistencies often indicate data entry errors, system bugs, or integration problems when merging data from multiple sources.
Accuracy refers to how closely data values correspond to reality. While accuracy is harder to assess programmatically than completeness or consistency (since you often do not have ground truth), you can check for values within valid ranges, look for suspicious patterns, and compare against known reference data. Computing a composite data quality score that weights these dimensions helps communicate overall data readiness to stakeholders and track quality improvements over time.
data = {
'name': ['Alice', 'Bob', None, 'Diana', 'Eve'],
'email': ['alice@co.com', None, 'charlie@co.com', None, 'eve@co.com'],
'age': [25, 30, None, 28, None],
'salary': [50000, 65000, 80000, 55000, 72000]
}
print("Completeness Report:")
print(f"{'Column':<10} {'Present':>8} {'Missing':>8} {'Score':>8}")
print("-" * 36)
total_cells = 0
missing_cells = 0
for col, values in data.items():
present = sum(1 for v in values if v is not None)
missing = sum(1 for v in values if v is None)
score = present / len(values) * 100
total_cells += len(values)
missing_cells += missing
print(f"{col:<10} {present:>8} {missing:>8} {score:>7.1f}%")
overall = (total_cells - missing_cells) / total_cells * 100
print("-" * 36)
print(f"{'Overall':<10} {total_cells - missing_cells:>8} {missing_cells:>8} {overall:>7.1f}%")This report shows completeness at both column and dataset levels. The salary column is fully complete while email and age are only 60% complete. The overall dataset completeness of 75% means one in four cells is missing data, which would need to be addressed before analysis.
records = [
{'name': 'Alice', 'age': 25, 'start_year': 2020, 'end_year': 2023, 'email': 'alice@company.com'},
{'name': 'Bob', 'age': -5, 'start_year': 2019, 'end_year': 2018, 'email': 'bob@co.com'},
{'name': 'Charlie', 'age': 200, 'start_year': 2021, 'end_year': 2024, 'email': 'not-an-email'},
{'name': 'Diana', 'age': 30, 'start_year': 2018, 'end_year': 2022, 'email': 'diana@corp.com'},
]
def validate_record(r):
issues = []
if r['age'] < 0 or r['age'] > 150:
issues.append(f"invalid age ({r['age']})")
if r['end_year'] < r['start_year']:
issues.append(f"end_year before start_year")
if '@' not in r['email'] or '.' not in r['email'].split('@')[-1]:
issues.append(f"invalid email format")
return issues
print("Consistency Check:")
total_checks = 0
passed_checks = 0
for rec in records:
issues = validate_record(rec)
checks = 3
total_checks += checks
passed = checks - len(issues)
passed_checks += passed
status = "PASS" if not issues else "FAIL"
print(f" {rec['name']}: {status}")
for issue in issues:
print(f" - {issue}")
print(f"\nConsistency Score: {passed_checks}/{total_checks} ({passed_checks/total_checks*100:.1f}%)")Consistency checks validate business rules across fields. Bob has a negative age and an end date before the start date. Charlie has an unrealistic age (200) and an improperly formatted email. These checks are domain-specific and should be designed based on business requirements.
data = {
'temperature_c': [22, 25, -300, 30, 28, 999, 24, 27],
'percentage': [85, 92, 105, 78, -10, 88, 95, 82],
'rating_1_5': [3, 5, 4, 7, 2, 1, 0, 4]
}
rules = {
'temperature_c': (-50, 60),
'percentage': (0, 100),
'rating_1_5': (1, 5)
}
print("Value Range Validation:")
for col, values in data.items():
low, high = rules[col]
valid = sum(1 for v in values if low <= v <= high)
invalid = len(values) - valid
bad_vals = [v for v in values if v < low or v > high]
score = valid / len(values) * 100
print(f"\n {col} (valid range: [{low}, {high}]):")
print(f" Valid: {valid}/{len(values)} ({score:.0f}%)")
if bad_vals:
print(f" Invalid values: {bad_vals}")Range validation checks whether numeric values fall within physically or logically possible bounds. A temperature of -300C or 999C is clearly erroneous. A percentage above 100 or below 0 violates the definition. These checks catch data entry errors and sensor malfunctions that would corrupt any analysis.
data = {
'name': ['Alice', 'Bob', None, 'Diana', 'Eve', 'Frank', None, 'Grace'],
'age': [25, 30, None, -5, 150, 28, 35, None],
'score': [85, 92, 105, 78, 88, None, 95, 82]
}
# 1. Completeness
total = sum(len(v) for v in data.values())
missing = sum(1 for v in data.values() for x in v if x is None)
completeness = (total - missing) / total * 100
# 2. Validity (range checks for numeric fields)
valid_checks = 0
valid_pass = 0
rules = {'age': (0, 120), 'score': (0, 100)}
for col, (low, high) in rules.items():
for v in data[col]:
if v is not None:
valid_checks += 1
if low <= v <= high:
valid_pass += 1
validity = valid_pass / valid_checks * 100 if valid_checks > 0 else 100
# 3. Uniqueness (check name field for duplicates)
names = [n for n in data['name'] if n is not None]
uniqueness = len(set(names)) / len(names) * 100 if names else 100
# Composite score (weighted average)
composite = completeness * 0.4 + validity * 0.4 + uniqueness * 0.2
print("Data Quality Scorecard:")
print(f" Completeness: {completeness:.1f}%")
print(f" Validity: {validity:.1f}%")
print(f" Uniqueness: {uniqueness:.1f}%")
print(f" Composite: {composite:.1f}%")
print()
if composite >= 90:
print("Quality: EXCELLENT")
elif composite >= 75:
print("Quality: GOOD")
elif composite >= 60:
print("Quality: FAIR")
else:
print("Quality: POOR")A composite data quality score combines multiple dimensions with business-appropriate weights. Here, completeness and validity are weighted 40% each (most critical for analysis), while uniqueness gets 20%. The overall GOOD rating means the data is usable but needs cleaning in specific areas before production analysis.
completeness, consistency, accuracy, data quality metrics