Statistics and probability

The mean is the arithmetic average (sum divided by count), while the median is the middle value when data is sorted. The median is more robust to outliers because extreme values heavily shift the mean but barely affect the position of the middle value. For example, in [1,2,3,4,100] the mean is 22 but the median is 3.

Variance measures how spread out the values of a feature are around its mean, computed as the average of squared deviations. A feature with near-zero variance carries almost no information and is often dropped, while high variance indicates the feature varies a lot across samples. Understanding variance also helps decide whether to scale or normalize features before training.

import numpy as np
x = np.array([2,4,4,4,5,5,7,9])
print(np.var(x))  # population variance = 4.0

The standard deviation is the square root of the variance, so it is expressed in the same units as the original data, making it easier to interpret. Variance is in squared units, which is mathematically convenient but less intuitive. For example, if heights have a variance of 25 cm², the standard deviation is 5 cm.

Probability equals favorable outcomes divided by total outcomes. There are 3 red balls out of 10 total, so P(red) = 3/10 = 0.3. This is the basic classical (equally likely) definition of probability.

The general addition rule is P(A ∪ B) = P(A) + P(B) − P(A ∩ B), where we subtract the intersection so overlapping outcomes are not counted twice. If A and B are mutually exclusive, P(A ∩ B) = 0, so it simplifies to P(A) + P(B).

Two events A and B are independent if the occurrence of one does not change the probability of the other. Mathematically, this means P(A ∩ B) = P(A) × P(B), or equivalently P(A | B) = P(A). Independence is a key assumption in models like Naive Bayes.

Bayes' theorem states P(A | B) = [P(B | A) × P(A)] / P(B). Here P(A) is the prior, P(B | A) is the likelihood, P(B) is the evidence (normalizing constant), and P(A | B) is the posterior probability of A after observing B. It lets us update beliefs given new data and underlies many ML classifiers.

Using Bayes' theorem: P(disease|+) = (0.99×0.01) / (0.99×0.01 + 0.01×0.99) = 0.0099/0.0198 = 0.5, about 50%. Despite the high accuracy, the low base rate means roughly half of positives are false positives. This illustrates the base-rate fallacy.

p_d=0.01; sens=0.99; spec=0.99
post=(sens*p_d)/(sens*p_d+(1-spec)*(1-p_d))
print(round(post,2))  # 0.5

Conditional probability is the probability of event A occurring given that B has already occurred. It is calculated as P(A | B) = P(A ∩ B) / P(B), provided P(B) > 0. It restricts the sample space to outcomes where B is true.

The expected value E[X] = Σ xᵢ·P(xᵢ) is the probability-weighted average of all possible values a random variable can take. It represents the long-run average outcome if the experiment were repeated many times. For a fair die, E[X] = (1+2+3+4+5+6)/6 = 3.5.

A binomial distribution models the number of successes in n independent trials, each with the same success probability p. Its two parameters are n (number of trials) and p (success probability), with mean np and variance np(1−p). An example is counting heads in 10 coin flips.

from scipy.stats import binom
print(binom.pmf(3, n=10, p=0.5))  # P(exactly 3 heads)

The normal distribution is symmetric and bell-shaped, fully described by its mean μ (center) and standard deviation σ (spread). About 68% of data lies within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean (the empirical rule). It appears frequently in nature and underlies many statistical methods.

Approximately 95% of the data falls within two standard deviations of the mean in a normal distribution. About 68% lies within one σ and about 99.7% within three σ. This rule is useful for quickly spotting outliers.

A z-score measures how many standard deviations a data point is from the mean, computed as z = (x − μ) / σ. It standardizes features to a common scale (mean 0, std 1), which helps algorithms that are sensitive to feature magnitude, like gradient descent or KNN. Z-scores also help identify outliers (e.g., |z| > 3).

z = (x - x.mean()) / x.std()

The Pearson correlation coefficient r measures the strength and direction of a linear relationship between two variables, ranging from −1 to +1. A value near +1 means strong positive linear correlation, near −1 means strong negative, and near 0 means no linear relationship. Importantly, correlation does not imply causation.

import numpy as np
print(np.corrcoef(x, y)[0,1])

A strong correlation between two variables does not mean one causes the other; a hidden third factor (confounder) may drive both. For example, ice cream sales and drowning deaths are correlated, but neither causes the other; hot weather (a confounder) increases both. This is critical to remember when interpreting data analysis results.

Covariance measures how two variables vary together: positive means they tend to increase together, negative means one rises as the other falls. Unlike correlation, covariance is not standardized, so its magnitude depends on the units of the variables and is hard to interpret directly. Correlation is covariance divided by the product of the two standard deviations, giving a unitless value in [−1, 1].

The Central Limit Theorem states that the distribution of sample means approaches a normal distribution as sample size grows, regardless of the population's original distribution (given finite variance). This is powerful because it justifies using normal-based methods like confidence intervals and hypothesis tests even when the underlying data is not normal. It typically holds well for sample sizes of about 30 or more.

A population is the entire group of items or individuals we want to study, while a sample is a subset of the population that we actually observe or measure. We use samples because measuring the whole population is often impractical, and we infer population parameters from sample statistics. Good sampling should be representative to avoid bias.

Dividing by (n−1), known as Bessel's correction, corrects for the bias that arises because the sample mean is used in place of the unknown true population mean. Using the sample mean makes deviations slightly too small, so dividing by the smaller (n−1) compensates and gives an unbiased estimate of the population variance. This matters most for small samples.

import numpy as np
np.var(x, ddof=1)  # sample variance, divides by n-1

The mode is the value (or values) that appears most frequently in a dataset. It is especially useful for categorical data where mean and median are not meaningful, such as finding the most common product category. A dataset can have no mode, one mode, or multiple modes (bimodal/multimodal).

Skewness measures the asymmetry of a distribution. In a positively (right) skewed distribution, the tail extends to the right and most data clusters on the left, so the mean is typically greater than the median. Income data is a classic example of right skew.

The IQR is the range of the middle 50% of data, calculated as Q3 − Q1 (third quartile minus first quartile). A common rule flags values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR as outliers. The IQR is robust because it ignores extreme values.

q1,q3 = np.percentile(x,[25,75]); iqr=q3-q1
low,high = q1-1.5*iqr, q3+1.5*iqr

A p-value is the probability of observing data as extreme as (or more extreme than) what was observed, assuming the null hypothesis is true. A small p-value (commonly < 0.05) suggests the observed result is unlikely under the null hypothesis, providing evidence against it. It is not the probability that the null hypothesis is true.

A Type I error (false positive) occurs when we reject a true null hypothesis, while a Type II error (false negative) occurs when we fail to reject a false null hypothesis. The significance level α controls the Type I error rate, and β relates to Type II errors. There is usually a trade-off between reducing one type and increasing the other.

A confidence interval is a range of values, computed from sample data, that likely contains the true population parameter. A 95% confidence interval means that if we repeated the sampling process many times, about 95% of such intervals would contain the true parameter. It does not mean there is a 95% probability the true value lies in this particular interval.

The Poisson distribution models the number of events occurring in a fixed interval of time or space when events happen independently at a constant average rate λ. Its mean and variance are both equal to λ. Examples include the number of emails received per hour or website visits per minute.

from scipy.stats import poisson
print(poisson.pmf(2, mu=3))  # P(2 events), lambda=3

The number of ways to get exactly 2 heads in 3 tosses is C(3,2) = 3, out of 2³ = 8 total equally likely outcomes. So the probability is 3/8 = 0.375. This is a binomial probability with n=3, p=0.5, k=2.

from math import comb
print(comb(3,2)/2**3)  # 0.375

The law of large numbers states that as the number of trials increases, the observed relative frequency of an event converges to its true probability. This justifies estimating probabilities empirically by running many trials or simulations. For example, the proportion of heads in coin flips approaches 0.5 as flips increase.

A probability distribution describes how probabilities are assigned to the possible values of a random variable. A discrete distribution assigns probabilities to countable outcomes (e.g., binomial, Poisson) and uses a probability mass function, while a continuous distribution (e.g., normal) spreads probability over a range and uses a probability density function where probabilities are areas under the curve. For continuous variables, the probability of any exact single value is zero.

Precision is the fraction of predicted positives that are actually positive: TP / (TP + FP). Recall is the fraction of actual positives that were correctly identified: TP / (TP + FN). Precision penalizes false positives while recall penalizes false negatives, so the right balance depends on the cost of each error type.

The complement rule states that the probability of an event not happening equals one minus the probability it does happen: P(not A) = 1 − P(A). This is useful when the complement is easier to compute, such as finding P(at least one) by subtracting P(none) from 1.

The expected value is the sum of each outcome times its probability: (1+2+3+4+5+6) × (1/6) = 21/6 = 3.5. Even though 3.5 is not a possible roll, it represents the long-run average over many rolls.

Sampling bias occurs when the collected sample is not representative of the target population, causing certain groups to be over- or under-represented. A model trained on biased data learns skewed patterns and generalizes poorly or unfairly to the real population. For example, a face-recognition model trained mostly on one demographic may perform badly on others.

There are 36 equally likely outcomes when rolling two dice. The combinations summing to 7 are (1,6),(2,5),(3,4),(4,3),(5,2),(6,1), which is 6 outcomes. So the probability is 6/36 = 1/6 ≈ 0.167.

For a continuous variable, the PDF gives the relative likelihood (density) at each value, where areas under the curve represent probabilities. The CDF, F(x), gives the probability that the variable is less than or equal to x, so it is the running integral of the PDF and increases from 0 to 1. The CDF is useful for finding probabilities like P(X ≤ a) directly.

from scipy.stats import norm
print(norm.cdf(0))  # 0.5

Entropy measures the average uncertainty or unpredictability in a probability distribution, defined as H = −Σ p(x) log p(x). High entropy means outcomes are spread out and unpredictable (e.g., a fair coin), while low entropy means outcomes are concentrated and predictable (e.g., a heavily biased coin). Entropy is used in decision trees to choose splits that reduce uncertainty.

import numpy as np
p=np.array([0.5,0.5])
print(-(p*np.log2(p)).sum())  # 1.0 bit

The mean only describes the center of the data and ignores how spread out the values are, so two very different datasets can share the same mean. A measure of dispersion such as the standard deviation or IQR should accompany the mean to convey variability. For example, [50,50,50] and [0,50,100] both have mean 50 but very different spreads.

A random variable is a function that assigns a numerical value to each outcome of a random experiment. A discrete random variable takes countable values (like the number of heads in coin flips), while a continuous random variable can take any value in a range (like a person's exact height). This distinction determines whether we use a PMF or a PDF to describe it.

Statistical significance helps determine whether an observed difference between version A and B is likely real or just due to random chance from sampling. Without it, you might act on noise, especially with small sample sizes, and falsely conclude a winner. A significance test (and adequate sample size) controls the risk of such false positives.