A/B testing and experimentation

A Type I error (false positive) is rejecting the null when there is truly no effect; its probability is alpha (the significance level). A Type II error (false negative) is failing to reject the null when a real effect exists; its probability is beta, and power = 1 - beta is the chance of detecting a true effect of a given size. You control Type I directly via alpha (commonly 0.05). You reduce Type II error (raise power) by increasing sample size, increasing the true effect size, or reducing metric variance, for a fixed alpha.

Repeatedly testing significance as data accumulates inflates the Type I error far above the nominal alpha because each look is another chance to cross the threshold by chance. A test designed for a single fixed-horizon analysis but monitored continuously can see its actual false-positive rate climb well above the intended 5% (toward ~30% or more with many looks). The fix is to use methods built for repeated looks: group-sequential boundaries with alpha-spending, or always-valid p-values / confidence sequences (e.g., mSPRT), which preserve the error rate under continuous monitoring.

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance-reduction technique that adjusts the metric using a pre-experiment covariate (typically the same metric measured before the test) that is correlated with the outcome but unaffected by treatment. By regressing out the covariate, it reduces metric variance by a factor of (1 - rho^2), where rho is the correlation between covariate and outcome; this tightens confidence intervals and effectively shrinks the required sample size. The adjusted treatment effect remains unbiased because the covariate is measured before randomization and is therefore independent of treatment assignment.

theta = cov(Y, X) / var(X)
Y_cuped = Y - theta * (X - X.mean())  # X = pre-period covariate; variance falls by ~(1 - corr(Y,X)**2)

Conversion is a binary (Bernoulli) outcome, so the sampling distribution of the difference in proportions is well-approximated by a normal distribution via the CLT when the expected counts np and n(1-p) are both reasonably large (commonly >= 5-10 per cell). The two-proportion z-test uses the analytic Bernoulli variance p(1-p)/n, typically with a pooled proportion under the null, rather than estimating a separate variance scale as a t-test does. A t-test assumes continuous, approximately normal data with unknown variance estimated from the sample; for large n both tests give nearly identical results on binary data.

from statsmodels.stats.proportion import proportions_ztest
z, p = proportions_ztest([conv_t, conv_c], [n_t, n_c])

SRM compares the observed allocation of units across variants to the intended split (e.g., 50/50) using a chi-square goodness-of-fit test; a very small p-value (a common alert threshold is < 0.001) signals a mismatch. An SRM indicates a bug in randomization, assignment, logging, redirection, or bot/duplicate filtering that very likely biases the treatment effect, so the result is untrustworthy regardless of how significant the metric looks. Detecting SRM should invalidate the analysis until the root cause is found and fixed.

from scipy.stats import chisquare
chisquare([n_control, n_treat], f_exp=[N*0.5, N*0.5])

SUTVA assumes one unit's treatment does not affect another unit's outcome, but network/marketplace interference (spillover) breaks this: a treated buyer can affect a control seller, or treated users influence their control friends. This contaminates the control group and biases the estimated treatment effect. Cluster-randomized designs (randomize whole clusters such as cities, network communities, or sessions), graph-cluster randomization, or switchback designs reduce interference, at the cost of higher variance and fewer effective independent units.

Simpson's paradox occurs when an aggregate effect reverses or disappears within subgroups, e.g., treatment wins overall but loses in every segment. Even with proper randomization it can appear when a ratio/aggregate metric is computed over heterogeneous traffic whose mix differs across variants over time, for example during a gradual ramp where treatment accumulated more of one user type or time period. Examining segment-level results and using a properly weighted analysis (or comparing only concurrently randomized traffic) prevents misleading conclusions.

When the unit of analysis (e.g., session or page-view) is finer than the unit of randomization (user), observations within a user are correlated, so the naive variance of a ratio metric is wrong and confidence intervals come out too narrow. The delta method approximates the variance of a ratio of means (E[X]/E[Y]) via a first-order Taylor expansion, accounting for the covariance between numerator and denominator. Combined with user-level aggregation, it yields correctly sized confidence intervals for metrics like CTR = total clicks / total sessions.

# Var(X/Y) ~ (1/Ybar^2)[Var(X) - 2(Xbar/Ybar)Cov(X,Y) + (Xbar/Ybar)^2 Var(Y)]

The MDE is the smallest true effect size the experiment is powered to detect at the chosen alpha and power. Required sample size scales roughly with 1/MDE^2, so halving the MDE quadruples the needed sample (and the duration at fixed traffic). Teams set the MDE to the smallest effect that is practically or economically meaningful, balancing the cost of a longer test against the value of being able to detect small wins.

You need the significance level alpha, the desired power (1-beta), the variance (or std) sigma of the metric, and the MDE delta (difference in means). For a two-sided, two-sample test with equal allocation and equal variance, the per-arm size is n = 2 * (z_{1-alpha/2} + z_{1-beta})^2 * sigma^2 / delta^2. Larger variance or a smaller delta increases n; higher power or a smaller alpha also increases n.

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=d, alpha=0.05, power=0.8)  # d = delta/sigma

A guardrail metric is a metric you do not intend to improve but must not harm, such as latency, crash rate, revenue, or unsubscribe rate. It protects against shipping a change that boosts the primary metric while degrading user experience or business health (e.g., increasing engagement by sending spammy notifications that raise churn). Guardrails are typically monitored for non-inferiority against a tolerance threshold rather than for improvement.

The OEC is a single (possibly composite) metric that captures the long-term business value the organization wants to optimize, balancing competing short-term signals. A good OEC aligns experiment decisions with strategic goals and resists gaming, whereas optimizing a narrow feature metric can produce local wins that hurt the overall product. Designing the OEC is hard because short-term proxies (clicks) may not correlate with long-term value (retention, lifetime revenue).

A novelty effect is a transient boost when users engage with a new feature simply because it is new, and the lift fades over time; a primacy effect is the opposite, where users initially resist or under-use a change before adopting it, so the effect grows over time. Detect both by plotting the treatment effect over time (segmenting by first-exposure cohort) and checking whether it converges to a stable level. Handle them by running the test long enough for the effect to stabilize and by analyzing new versus returning users separately.

Testing many hypotheses inflates the family-wise probability of at least one false positive; at alpha=0.05 with 20 independent true nulls you expect about one spurious 'significant' result. Corrections include the Bonferroni method (divide alpha by the number of tests to control the family-wise error rate, FWER) and the Benjamini-Hochberg procedure (control the false discovery rate, FDR, which is more powerful for many metrics). FDR is usually preferred when screening many secondary metrics, while FWER control suits a few critical confirmatory decisions.

from statsmodels.stats.multitest import multipletests
rej, p_adj, *_ = multipletests(pvals, alpha=0.05, method='fdr_bh')

A 95% CI means that under repeated sampling, 95% of such constructed intervals would contain the true parameter; it is a statement about the procedure's long-run coverage, not the probability for this one realized interval. The common misinterpretation is 'there is a 95% probability the true effect lies in this specific interval,' which is a Bayesian credible-interval statement, not a frequentist one. For a fixed realized interval, the true value is either in it or not.

Frequentist testing controls long-run error rates and reports p-values and confidence intervals under a fixed design, while Bayesian testing yields a posterior distribution over the effect, enabling direct statements such as P(treatment > control) or the expected loss of each decision. Bayesian methods naturally incorporate priors and support decision-theoretic stopping via expected-loss thresholds, and they avoid the common misreading of a p-value as 'probability the variant is better.' However, Bayesian results are sensitive to prior choice and do not automatically guarantee frequentist error control unless the design is built for it.

A switchback experiment alternates the entire system (often within a region) between treatment and control over consecutive time intervals, so the unit of randomization is a time window rather than the individual user. It is appropriate when there is strong marketplace or network interference or shared resources (e.g., ride-hailing pricing, delivery dispatch) that make user-level randomization invalid. The main challenges are temporal autocorrelation and carryover effects, which require well-chosen window lengths and appropriate (often cluster/time-block) variance estimation.

With very large samples, tiny effects become statistically significant (p < 0.05) even when the absolute change is too small to matter for users or revenue. Ship decisions should weigh the effect size and its confidence interval against implementation and maintenance cost and the pre-registered MDE, not just the p-value. A result can be significant yet not worth shipping, or non-significant but worth iterating on if its CI still includes meaningful gains.

Stratification splits randomization or analysis into homogeneous strata (e.g., by country or device) and combines within-stratum estimates, removing between-stratum variance to tighten the overall estimate. CUPED instead regresses out a continuous pre-experiment covariate from the metric. Both reduce variance without introducing bias; stratification exploits categorical structure and can be applied at design time, while CUPED leverages a correlated numeric pre-period variable and is applied at analysis time (and the two can be combined).

An A/A test runs two identical variants against each other, so any observed 'difference' is due to chance alone. It validates the experimentation pipeline: confirming that the false-positive rate matches alpha, that randomization/assignment and logging are unbiased (no SRM), and that variance and CI estimation are correctly calibrated. Over many repeated A/A tests, the p-values should be approximately uniform on [0,1] and significant results should appear at roughly the alpha rate.

During a ramp, the populations exposed at different allocation percentages can differ (early adopters, geos, or time-of-day mix), and pooling across stages mixes non-comparable cohorts, biasing the estimate. To avoid it, hold a stable, concurrently randomized control allocation throughout, and compare treatment versus control only within the same time periods. Avoid comparing post-ramp treatment to pre-ramp baselines; concurrent randomized control is the key safeguard.

ITT analyzes users according to their randomized assignment regardless of whether they actually received or used the treatment, preserving randomization and giving an unbiased estimate of the effect of offering the treatment. Per-protocol analysis (only compliers) breaks randomization because compliance is often correlated with outcomes, introducing selection bias. ITT estimates the real-world effect of the policy or launch, and instrumental-variable methods (CACE/LATE) can recover the effect on compliers when that is the question of interest.

Heavy tails inflate variance and slow CLT convergence, so the normal approximation for the difference in means may be poor at typical sample sizes and a few outliers can dominate the estimate, reducing power and distorting CIs. Mitigations include capping/winsorizing extreme values, transforming the metric, using bootstrap or rank-based inference, and applying variance reduction (CUPED). Reporting both the mean and a robust summary, and checking that conclusions are stable to the capping threshold, guards against outlier-driven results.

y_capped = np.clip(y, None, np.quantile(y, 0.99))

A holdout is a persistent group kept on the old experience (or excluded from a set of launches) for a long period to measure the cumulative, long-term impact of many shipped changes. It captures effects that short experiments miss, such as novelty decay, cannibalization across features, and long-term retention/revenue, and it detects 'win inflation' where summed short-term wins overstate true value. The cost is denying some users improvements, so holdouts are usually small and time-bounded.

Under a true null with a continuous test statistic, the p-value equals the CDF of the statistic evaluated at the observed value, and applying a variable's own CDF (the probability integral transform) yields a Uniform(0,1) variable. This is exactly why a fraction alpha of null tests cross the p < alpha threshold, giving the intended Type I rate. In practice, plotting p-values from many A/A tests and checking for uniformity validates that the analysis is correctly calibrated; deviations indicate variance miscalculation, unmodeled clustering, or SRM.

The absolute effect is the raw difference (e.g., +1.2 percentage points), while relative lift is the percentage change versus control (e.g., +6%). Relative lift is a ratio, so its variance must account for uncertainty in the control denominator (via the delta method); naively dividing the absolute CI endpoints by the control mean understates the true uncertainty. Stakeholders often prefer relative lift for comparability, but it becomes unstable when the control baseline is small.

The randomization unit should be stable and match how the treatment is experienced; if a single user is randomized inconsistently (per cookie or per session), they may see both variants, contaminating the comparison and diluting the measured effect. Cookie-based randomization also suffers from cookie churn (deletion) that re-randomizes users and biases retention metrics. Stable user-level IDs ensure consistent assignment, valid long-term metrics, and clean attribution.

A quantile treatment effect compares a distribution quantile (such as p95 latency) between variants, which matters because means hide tail behavior. The sampling variance of a sample quantile depends on the probability density at that quantile (asymptotically p(1-p)/(n*f(q)^2)), not the simple proportion-style variance, and clustering by user complicates it further. Correct inference uses quantile-specific (e.g., density-based) variance estimators or bootstrap/influence-function methods rather than a naive two-sample z-test.

If two experiments change overlapping surfaces, their effects can interact so that each one's measured effect depends on the other's assignment, biasing both. Platforms manage this with orthogonal randomization (independent hash seeds so assignments are uncorrelated), so on average each experiment's control and treatment contain the same mix of the other experiment's variants and the effects average out. When a true interaction is a real concern, conflicting tests are isolated in mutually exclusive layers/universes.

When randomizing by user but measuring per-event, events within a user are correlated, so the effective sample size is smaller than the raw event count. Ignoring clustering treats correlated events as independent, underestimating standard errors and producing false positives. Cluster-robust standard errors (clustered on the randomization unit) or the delta method account for intra-user correlation, yielding correctly sized CIs and valid p-values.

import statsmodels.formula.api as smf
m = smf.ols('y ~ treat', df).fit(cov_type='cluster', cov_kwds={'groups': df.user_id})

A non-inferiority test checks whether a new variant is not worse than control by more than a pre-specified margin (delta), rather than testing for improvement. It is used when you want to ship a change for reasons other than the primary metric (cost savings, simpler code, an infra migration) while ensuring the key metric does not regress meaningfully. The test is one-sided against the margin: you conclude non-inferiority when the confidence interval's lower bound stays above -delta.

Always-valid inference constructs p-values or confidence sequences that hold simultaneously at every sample size, so you can stop whenever the boundary is crossed without inflating the false-positive rate. The mSPRT mixes the likelihood ratio over a prior on the effect, producing a test statistic that controls Type I error under optional stopping. The tradeoff is somewhat lower power (wider boundaries) than a fixed-horizon test sized for a single look, in exchange for flexible, valid stopping at any time.

When you pick the maximum observed effect across many variants, you preferentially select variants whose random noise happened to be favorable, so the chosen winner's estimate is biased upward (regression to the mean). Mitigations include re-measuring the selected winner in a separate validation/replication run, applying shrinkage/empirical-Bayes estimators, or using holdout-based re-measurement before relying on the reported lift. Pre-registering a single primary variant also avoids selection-induced bias.

A pre-registered analysis plan specifies, before seeing outcome data, the primary metric and OEC, the MDE and required sample size/duration, the statistical test and any multiple-comparison corrections, the segments to examine, the guardrails, and the decision rule. This prevents p-hacking, HARKing (hypothesizing after results are known), and cherry-picking among many post-hoc metrics or segments. Anything decided after seeing results (exploratory analyses) should be labeled as such and treated as hypothesis-generating, not confirmatory.