Statistics for data science

A Type I error is rejecting a true null hypothesis (false positive), occurring with probability α. A Type II error is failing to reject a false null hypothesis (false negative), with probability β; statistical power is 1−β. Lowering α reduces Type I errors but, holding sample size fixed, increases β and reduces power.

The p-value is the probability of observing data as or more extreme than what was seen, assuming the null hypothesis is true. It is NOT the probability that the null is true, nor the probability the result occurred by chance in any absolute sense. With α=0.05, p=0.03 leads to rejecting the null.

The CLT states that the sampling distribution of the sample mean approaches a normal distribution as sample size n grows, regardless of the population's distribution, provided observations are i.i.d. with finite variance. The mean equals the population mean μ and standard error equals σ/√n. It does not require the underlying population to be normal.

Standard deviation measures dispersion of individual observations around the mean within a sample or population. Standard error of the mean (SEM = σ/√n) measures the variability of the sample-mean estimator across repeated samples. SEM shrinks as n increases, while the population standard deviation does not.

A 95% CI is a procedure that, under repeated sampling, produces intervals containing the true parameter 95% of the time. For a single computed interval, the parameter is either in it or not; the 95% refers to the long-run coverage of the method, not the probability the parameter lies in this specific interval (in the frequentist framework).

Expected squared prediction error decomposes into bias² + variance + irreducible error. High-bias models underfit (too simple), high-variance models overfit (too sensitive to training data). The tradeoff is choosing model complexity that minimizes total error; reducing one often increases the other.

Use a t-test when the population standard deviation is unknown and estimated from the sample, especially with small n; the t-distribution has heavier tails accounting for this extra uncertainty. Use a z-test when σ is known or n is large enough that the t-distribution converges to the normal.

Testing many hypotheses inflates the family-wise error rate (probability of at least one false positive). Bonferroni controls FWER by testing each of m hypotheses at α/m, ensuring overall FWER ≤ α. It is conservative, reducing power, so methods like Benjamini-Hochberg (controlling FDR) are often preferred for large m.

FWER controls the probability of making even one false positive among all tests and is stringent (e.g., Bonferroni, Holm). FDR controls the expected proportion of false positives among rejected hypotheses (e.g., Benjamini-Hochberg), tolerating some false positives for greater power, which suits high-dimensional screening like genomics.

An estimator is unbiased if its expected value equals the true parameter. Unbiasedness is not always optimal: a biased estimator (e.g., ridge regression, James-Stein) can have lower mean squared error by trading a small bias for a large variance reduction. MSE = bias² + variance is the more complete criterion.

Dividing by n−1 yields an unbiased estimator of the population variance. Using the sample mean (rather than the true mean) systematically underestimates spread, costing one degree of freedom; n−1 corrects this so E[s²]=σ². The divisor n gives the maximum likelihood estimate but is biased downward.

Covariance measures the joint linear variability of two variables but its magnitude depends on their units/scales. Pearson correlation normalizes covariance by the product of standard deviations, yielding a unitless value in [−1, 1] that is scale-invariant and directly comparable across variable pairs.

Key assumptions are linearity in parameters, independent errors, homoscedasticity (constant error variance), and for exact small-sample inference, normally distributed errors. Predictors should not be perfectly collinear and should be uncorrelated with the error term (exogeneity). Violations affect coefficient validity, standard errors, or efficiency.

Heteroscedasticity is non-constant error variance across observations. OLS coefficient estimates remain unbiased and consistent, but the usual standard errors are biased, invalidating t-tests and confidence intervals. Remedies include heteroscedasticity-robust (White/HC) standard errors, weighted least squares, or transforming the response.

Multicollinearity is high correlation among predictors, inflating coefficient standard errors and making estimates unstable and hard to interpret. VIF for predictor j is 1/(1−Rj²) from regressing xj on the others; VIF > 5–10 signals problematic collinearity. It does not bias predictions, only individual coefficient inference.

L1 adds a penalty proportional to the sum of absolute coefficients, producing sparse solutions by driving some coefficients exactly to zero (feature selection). L2 penalizes squared coefficients, shrinking them smoothly toward zero without eliminating them, handling correlated predictors more gracefully. Elastic Net blends both.

A quantile-quantile plot compares the quantiles of observed data against the theoretical quantiles of a reference distribution (often normal). Points falling on the 45° line indicate the data match the distribution; systematic deviations reveal skew, heavy tails, or other departures. It is a standard normality and distributional-fit diagnostic.

MLE maximizes the likelihood of the data given parameters, P(D|θ). MAP maximizes the posterior P(θ|D) ∝ P(D|θ)P(θ), incorporating a prior. MAP equals MLE when the prior is uniform; a Gaussian prior on coefficients makes MAP equivalent to L2-regularized estimation.

Bayes' theorem: P(θ|D) = P(D|θ)P(θ) / P(D). The posterior P(θ|D) is updated belief about parameters after data; P(D|θ) is the likelihood; P(θ) is the prior belief; P(D) is the marginal/evidence normalizing constant. It formalizes updating beliefs with new evidence.

Using Bayes: P(D|+) = (0.99×0.01) / (0.99×0.01 + 0.05×0.99) = 0.0099 / (0.0099 + 0.0495) ≈ 0.167, about 17%. Despite high test accuracy, low prevalence makes most positives false positives—the base-rate fallacy.

p_d=0.01; sens=0.99; spec=0.95
post = sens*p_d / (sens*p_d + (1-spec)*(1-p_d))
print(round(post,3))  # 0.167

A PMF gives probabilities for discrete outcomes (each value has nonzero probability). A PDF describes continuous variables where probability is the integral over an interval (the density at a point is not a probability). A CDF, F(x)=P(X≤x), applies to both and is the integral/sum up to x, monotonically increasing from 0 to 1.

The Poisson models counts of independent events occurring at a constant average rate λ over a fixed interval (e.g., arrivals per minute). Its defining property is that the mean equals the variance (both λ). If variance exceeds the mean (overdispersion), a Negative Binomial model is preferred.

Parametric methods assume the data follow a specified distribution defined by a finite set of parameters (e.g., t-test assumes normality), giving more power when assumptions hold. Non-parametric methods (e.g., Mann-Whitney U, Wilcoxon, bootstrap) make minimal distributional assumptions, are more robust to outliers and non-normality, but are typically less powerful when parametric assumptions are valid.

Two variables can be correlated due to a common cause (confounder), reverse causation, selection bias, or coincidence, without one causing the other. Establishing causation requires randomized experiments or causal-inference techniques (instrumental variables, difference-in-differences, propensity adjustment) that control for confounders. Observational correlation alone is insufficient.

Simpson's paradox occurs when a trend present in aggregated data reverses or disappears when the data are split into subgroups, caused by a lurking confounding variable correlated with both the grouping and the outcome. It highlights why proper stratification and causal reasoning matter; naive aggregation can mislead.

Power (1−β) is the probability of correctly rejecting a false null hypothesis. It increases with larger sample size, larger true effect size, lower variance/noise, and a higher (less stringent) significance level α. Power analysis is used before an experiment to determine the sample size needed to detect a meaningful effect.

The bootstrap resamples the observed data with replacement many times (e.g., 10,000), computing the statistic on each resample to approximate its sampling distribution empirically. This yields standard errors and confidence intervals (e.g., percentile method) without assuming a parametric form, making it valuable for complex statistics lacking closed-form variance.

import numpy as np
rng=np.random.default_rng(0)
x=rng.normal(size=100)
means=[rng.choice(x,size=len(x),replace=True).mean() for _ in range(10000)]
ci=np.percentile(means,[2.5,97.5])

The goodness-of-fit test compares observed frequencies of a single categorical variable to expected frequencies under a hypothesized distribution. The test of independence assesses whether two categorical variables in a contingency table are associated, comparing observed counts to those expected if the variables were independent. Both use the chi-square statistic but answer different questions.

Repeatedly checking results and stopping when significance is reached inflates the Type I error rate far above α, because each look is an additional chance for a false positive. Mitigations include fixing the sample size in advance, alpha-spending/group-sequential designs (O'Brien-Fleming), or Bayesian/always-valid sequential testing methods that maintain valid error control under continuous monitoring.

CUPED (Controlled-experiment Using Pre-Experiment Data) reduces metric variance by regressing out a pre-experiment covariate correlated with the outcome, increasing sensitivity without bias. Stratification ensures balanced treatment assignment across subgroups and then combines within-stratum estimates. Both lower variance and increase power; CUPED leverages continuous pre-period signal, stratification uses discrete blocks.

ROC-AUC plots true positive rate vs false positive rate and is threshold-independent, but it can appear optimistic under heavy class imbalance because the FPR uses the large negative class. Precision-recall AUC focuses on the positive class (precision vs recall) and is more informative when positives are rare, as in fraud or disease detection.

Fixed-effects models estimate a separate parameter per group, controlling for all time-invariant group characteristics but not generalizing beyond observed groups. Random-effects (mixed/multilevel) models treat group effects as draws from a distribution, enabling partial pooling that shrinks group estimates toward the overall mean—more efficient when groups are exchangeable and some have little data.

R² is the proportion of variance in the response explained by the model and never decreases when predictors are added, even useless ones. Adjusted R² penalizes for the number of predictors relative to sample size, so it can decrease if added variables do not improve fit enough—making it better for comparing models of differing complexity.

Both penalize model complexity added to the negative log-likelihood (specifically, criterion = −2·log-likelihood + penalty), but AIC uses a penalty of 2k while BIC uses k·ln(n), where k is parameters and n is sample size. BIC penalizes complexity more heavily for large n and is consistent (selects the true model asymptotically if it exists), whereas AIC targets predictive accuracy and tends to favor larger models.

Classical standard errors assume constant error variance (homoscedasticity) and correct model specification. Robust 'sandwich' (Huber-White) standard errors remain valid under heteroscedasticity (and clustered versions under within-group correlation), giving asymptotically correct inference without assuming constant variance. Use them when residual variance is non-constant or observations are clustered.

The law of large numbers states the sample mean converges to the population mean as n grows (a statement about convergence of a single quantity to a constant). The CLT describes the shape and spread of the sampling distribution around that mean—that standardized sample means are approximately normal. LLN gives consistency; CLT gives the distributional form for inference.

A confounder is a variable associated with both the treatment and the outcome, creating spurious associations. Randomized assignment balances confounders (both measured and unmeasured) across groups in expectation, so any outcome difference can be attributed to the treatment. This is why randomized controlled trials are the gold standard for causal inference.

Independent events satisfy P(A∩B)=P(A)P(B); knowing one occurred does not change the other's probability. Mutually exclusive events cannot co-occur, so P(A∩B)=0. Two events with nonzero probability that are mutually exclusive are necessarily dependent, since one occurring rules out the other.

All three test parameter restrictions in likelihood-based models and are asymptotically equivalent. The Wald test uses the estimate and its standard error at the unrestricted MLE; the likelihood-ratio test compares maximized log-likelihoods of restricted vs unrestricted models; the score (Lagrange multiplier) test uses the gradient at the restricted estimate. LR is generally most reliable in small samples; Wald can be unstable near parameter boundaries.

With very large samples, even trivially small effects become statistically significant because standard errors shrink, so significance does not measure importance. Reports should include effect sizes (e.g., Cohen's d, odds ratios) and confidence intervals to convey magnitude and precision, enabling judgment of practical relevance rather than relying on p-values alone.