Cheat sheet · Statistics & Probability

Hypothesis Testing cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

A practitioner's reference covering the frequentist hypothesis-testing framework, the core statistical tests and their assumptions, and the end-to-end mechanics of designing, sizing, running, and analyzing controlled online experiments (A/B tests), including modern concerns like peeking, multiple testing, sample-ratio mismatch, CUPED variance reduction, and sequential/Bayesian alternatives.

Hypothesis Testing Fundamentals

Null hypothesis H0 = no effect / status quo; alternative H1 = the effect you suspect. You never 'accept' H0, you only fail to reject it.
p-value = P(a test statistic at least as extreme as observed | H0 true). It is NOT P(H0 true), NOT P(H1 true), and NOT the probability your specific result is 'due to chance'.
Significance level alpha (typically 0.05) is the Type I error rate (false positive) you tolerate; reject H0 when p < alpha. alpha must be fixed before seeing the data.
Type I error = reject a true H0 (rate alpha). Type II error = fail to reject a false H0 (rate beta). Power = 1 - beta, conventionally targeted at 0.80.
One-tailed vs two-tailed: use two-tailed by default; a one-tailed test puts all alpha on one side, so switching to it post hoc to gain significance is p-hacking.
Confidence interval and test are dual: a 95% CI for the difference that excludes 0 corresponds to p < 0.05 (two-sided). Report effect size + CI, not just a p-value.
Effect size (Cohen's d, relative lift, odds ratio) measures practical significance, which is distinct from statistical significance and does not shrink with larger n.

Choosing the Right Test

Two means, large n or known population variance: z-test; small n with unknown variance: two-sample t-test. Welch's t-test does NOT assume equal variances and is the safer default.
Paired/repeated measurements (same unit before/after): paired t-test, or its non-parametric counterpart, the Wilcoxon signed-rank test.
Two proportions / conversion rates: two-proportion z-test or chi-square test of independence on the 2x2 table; Fisher's exact test for small counts.
More than two groups: one-way ANOVA (means) or chi-square (categorical), followed by corrected post-hoc pairwise tests (Tukey HSD, Bonferroni).
Non-normal continuous or ordinal outcomes: Mann-Whitney U (a.k.a. Wilcoxon rank-sum) instead of the t-test; it tests stochastic dominance, not means.
Assumption checks: independence of observations, approximate normality of the SAMPLING distribution (CLT helps at large n, not raw-data normality), and homogeneity of variance where required.
For revenue / heavy-tailed metrics, prefer bootstrap CIs, rank-based or trimmed methods, or model the ratio with a delta-method variance estimator.

Designing an A/B Test

Define one primary OEC (Overall Evaluation Criterion) metric up front; track guardrail metrics (latency, crashes, revenue, unsubscribes) to catch harm.
Pick the randomization unit (user, session, device, cluster) to match the analysis unit and avoid contamination; user-level is most common to prevent leakage across a user's sessions.
Calculate sample size BEFORE launch from alpha, power (0.80), baseline rate, and minimum detectable effect (MDE). Halving the MDE roughly quadruples required n.
Set duration to cover full business cycles (typically >= 1-2 weeks) to absorb day-of-week and novelty/primacy effects; fix the horizon in advance.
Run an A/A test to validate splitting/instrumentation: it should yield roughly uniform p-values and no significant difference.
Check randomization health with a Sample Ratio Mismatch (SRM) test (chi-square on assignment counts vs the intended split); a significant SRM almost always signals a bug that invalidates results.
Pre-register the analysis plan (primary metric, segments, stopping rule) to avoid HARKing and garden-of-forking-paths bias.

Sample Size & Power

Per-group n for comparing two means: n ~= 2 * (z_{alpha/2} + z_{beta})^2 * sigma^2 / delta^2, where delta is the MDE and sigma the per-unit std dev.
Halving the MDE roughly quadruples the required sample size (n scales with 1/delta^2); this is the central tradeoff in experiment planning.
For proportions, variance is p(1-p) at baseline p; rarer events (small p) need much larger samples to detect a given relative lift.
Power increases with larger n, larger true effect, lower variance, and higher alpha. Underpowered tests yield noisy estimates and, when they do 'win', exaggerate effects (Type M magnitude and Type S sign errors).
Variance reduction reaches power faster: CUPED uses a pre-experiment covariate correlated with the metric to cut variance (often 30-50%); stratification and regression adjustment (CUPAC) also help.
Don't size only the primary metric, verify power for important guardrails too, or explicitly treat them as directional-only.

Analyzing & Interpreting Results

For the pre-declared primary metric first, report the point estimate (absolute and relative lift), its CI, and the p-value.
Statistical significance != business impact: weigh effect size, CI bounds, implementation cost, and guardrail movement before shipping.
Multiple comparisons inflate false positives: control family-wise error (FWER) with Bonferroni/Holm, or control the false discovery rate (FDR) with Benjamini-Hochberg when testing many metrics.
Peeking inflates Type I error: repeatedly checking and stopping at p < 0.05 with a fixed-horizon test can push the true false-positive rate well above 30%. Use a fixed horizon, or always-valid methods (group-sequential O'Brien-Fleming, mSPRT, or Bayesian) built for continuous monitoring.
Segment with care: subgroup analyses are exploratory and must be multiplicity-corrected; an overall null 'rescued' by one segment is a red flag.
Beware Simpson's paradox: aggregate and per-segment conclusions can reverse when group sizes or mix differ.
Bayesian alternative: report P(B > A) and expected loss / probability of being best, intuitive for stakeholders but requires sensible priors and a pre-agreed decision rule.

Advanced & Real-World Topics

Network/interference effects (social, marketplaces, two-sided platforms) violate SUTVA; use cluster-randomized or switchback designs to contain spillover.
Novelty and primacy effects: early metrics can over- or under-state long-run impact; inspect trends over time and consider a long-term holdback group.
Heterogeneous treatment effects: use CUPED + regression, or uplift modeling / causal ML (X-learner, causal forests) to find who benefits.
Multi-armed bandits trade exploration vs exploitation and adapt traffic toward winners, great for many short-lived variants but weaker for unbiased effect estimation and long-term metrics.
Ratio metrics (e.g., clicks/session) whose unit differs from the randomization unit need the delta method or a clustered bootstrap for a correct variance.
Quasi-experiments when randomization is impossible: difference-in-differences, regression discontinuity, instrumental variables, synthetic control.
Agree a practical-significance threshold and a ship/rollback decision framework with stakeholders BEFORE reading results.

Common pitfalls

Interpreting the p-value as P(H0 true) or the probability the result happened 'by chance', it is conditional on H0 being true.
Peeking and stopping the moment p < 0.05 under a fixed-horizon test; continuous monitoring can push the real false-positive rate above 30%.
Ignoring Sample Ratio Mismatch, a split that deviates from the intended ratio (significant chi-square) signals a logging/assignment bug that invalidates the test.
Testing many metrics or segments without multiple-comparison correction, then reporting only the one that 'won'.
Confusing statistical with practical significance; at huge n a trivial lift is 'significant' yet worthless.
Randomizing at a smaller unit than the analysis unit (e.g., randomize sessions but report per-user), which breaks the independence assumption and understates variance.
Running the test too short, capturing novelty effects or only part of a weekly cycle.
Switching to a one-tailed test or changing the primary metric after seeing data (HARKing) to get under 0.05.
Declaring 'no difference' from a non-significant, underpowered test, absence of evidence is not evidence of absence.
Applying a plain t/z-test to heavy-tailed revenue or ratio metrics without bootstrap/delta-method, producing wrong CIs.

Interview tips

Define p-value, alpha, beta, power, and Type I/II errors precisely in one sentence each, interviewers probe for the common misinterpretations.
Expect 'design an A/B test for feature X end to end': hypothesis, OEC + guardrails, randomization unit, sample size/MDE, duration, analysis, decision rule.
Know when to use which test (t vs z vs chi-square vs Mann-Whitney vs ANOVA) and the assumptions behind each; default to Welch's t-test for two means.
Walk through computing sample size from alpha, power, baseline, and MDE, and explain the quadratic n vs MDE relationship.
Explain the peeking problem and at least one valid fix (fixed horizon, sequential testing, or Bayesian), a frequent senior-level follow-up.
Discuss SRM and A/A tests and how you'd validate experiment integrity before trusting any result.
Be ready to handle multiple comparisons (Bonferroni/Holm vs Benjamini-Hochberg) and to explain Simpson's paradox with a concrete example.
Show awareness of variance reduction (CUPED) and interference/network effects with cluster or switchback designs, signals real production experience.
Frame conclusions as decisions: tie statistical results to practical significance, guardrails, and a ship/rollback recommendation.

Read the full lesson Test yourself — 19 question quiz