Difficulty: Intermediate
Probability is the mathematical framework for quantifying uncertainty. It assigns a number between 0 and 1 to events, where 0 means impossible and 1 means certain. In data science, probability underpins everything from statistical inference and hypothesis testing to machine learning algorithms like Naive Bayes and logistic regression. A solid grasp of probability is essential for interpreting model outputs, understanding sampling, and making data-driven decisions.
The basic probability of an event A is calculated as P(A) = number of favorable outcomes / total number of outcomes. For example, the probability of rolling a 3 on a fair six-sided die is 1/6. The complement rule states P(not A) = 1 - P(A). The addition rule for two events is P(A or B) = P(A) + P(B) - P(A and B), which accounts for the overlap between events. If events are mutually exclusive (they cannot both occur), then P(A and B) = 0 and the formula simplifies to P(A or B) = P(A) + P(B).
Conditional probability measures the probability of an event given that another event has already occurred. It is written as P(A|B) and calculated as P(A and B) / P(B). For example, the probability that a card is a king given that it is a face card is P(King|Face card) = P(King and Face card) / P(Face card) = (4/52) / (12/52) = 4/12 = 1/3. Conditional probability is fundamental to understanding how information updates our beliefs.
Bayes' theorem provides a way to reverse conditional probabilities. It states: P(A|B) = P(B|A) * P(A) / P(B). This is enormously powerful in practice -- it lets you update the probability of a hypothesis given new evidence. For example, in medical testing, Bayes' theorem helps calculate the probability that a patient actually has a disease given a positive test result, accounting for the test's sensitivity and the disease's prevalence.
Two events A and B are independent if the occurrence of one does not affect the probability of the other: P(A and B) = P(A) * P(B), or equivalently P(A|B) = P(A). Independence is a crucial concept in statistics -- many statistical tests assume independence of observations. In practice, verifying independence is important: for instance, consecutive stock prices are NOT independent, which is why simple probability models often fail in finance.
# Probability of events with a standard deck of 52 cards
total_cards = 52
hearts = 13
face_cards = 12 # J, Q, K in each of 4 suits
aces = 4
p_heart = hearts / total_cards
p_face = face_cards / total_cards
p_ace = aces / total_cards
print(f"P(Heart): {round(p_heart, 4)}")
print(f"P(Face card): {round(p_face, 4)}")
print(f"P(Ace): {round(p_ace, 4)}")
print(f"P(Not Ace): {round(1 - p_ace, 4)}")Basic probability is favorable outcomes divided by total outcomes. P(Heart) = 13/52 = 0.25. The complement rule gives P(Not Ace) = 1 - 4/52 = 48/52 = 0.9231.
# P(Heart OR Face card) in a deck of 52
total = 52
hearts = 13
face_cards = 12
heart_face_cards = 3 # J, Q, K of hearts
# P(A or B) = P(A) + P(B) - P(A and B)
p_heart_or_face = (hearts + face_cards - heart_face_cards) / total
print(f"Hearts: {hearts}")
print(f"Face cards: {face_cards}")
print(f"Heart face cards (overlap): {heart_face_cards}")
print(f"P(Heart OR Face): {round(p_heart_or_face, 4)}")There are 13 hearts and 12 face cards, but 3 cards are both (J/Q/K of hearts). Using the addition rule: (13 + 12 - 3)/52 = 22/52 = 0.4231. Without subtracting the overlap, we would double-count those 3 cards.
# A bag has 5 red and 3 blue balls
# Two balls drawn without replacement
# P(2nd is red | 1st was red)
red = 5
blue = 3
total = red + blue
# P(1st red)
p_first_red = red / total
# P(2nd red | 1st red) - one red removed
p_second_red_given_first_red = (red - 1) / (total - 1)
# P(both red) = P(1st red) * P(2nd red | 1st red)
p_both_red = p_first_red * p_second_red_given_first_red
print(f"P(1st red): {round(p_first_red, 4)}")
print(f"P(2nd red | 1st red): {round(p_second_red_given_first_red, 4)}")
print(f"P(both red): {round(p_both_red, 4)}")After drawing a red ball first (5/8), there are 4 red balls left out of 7 total. So P(2nd red | 1st red) = 4/7 = 0.5714. The joint probability is (5/8) * (4/7) = 20/56 = 0.3571.
# Medical test example
# Disease prevalence: 1%
# Test sensitivity (true positive rate): 95%
# Test specificity (true negative rate): 90%
prevalence = 0.01
sensitivity = 0.95 # P(positive | disease)
specificity = 0.90 # P(negative | no disease)
# P(positive) = P(pos|disease)*P(disease) + P(pos|no disease)*P(no disease)
p_positive = sensitivity * prevalence + (1 - specificity) * (1 - prevalence)
# P(disease | positive) = P(positive | disease) * P(disease) / P(positive)
p_disease_given_positive = (sensitivity * prevalence) / p_positive
print(f"P(positive test): {round(p_positive, 4)}")
print(f"P(disease | positive test): {round(p_disease_given_positive, 4)}")
print(f"Percentage: {round(p_disease_given_positive * 100, 2)}%")Despite the test being 95% sensitive, a positive result only means an 8.76% chance of actually having the disease. This is because the disease is rare (1%), so most positive results are false positives. This counterintuitive result is why Bayes' theorem is so important in data science.
Probability, Conditional probability, Bayes theorem, Independence