Experimentation & Metrics

A valid A/B test requires: (1) Define the primary metric (OEC) and guardrail metrics before starting. (2) Compute required sample size from baseline rate, MDE, alpha, and power. (3) Randomize assignment at the correct unit (user, session, device). (4) Run for the full pre-determined duration — do not stop early. (5) Check for imbalance in covariate distributions between groups (SRM check). (6) Analyze using the pre-specified method. (7) Validate the result holds on a holdout or replication. Skipping any step introduces bias or validity threats that can produce misleading conclusions.

Sample Ratio Mismatch (SRM) occurs when the observed ratio of users in treatment vs control groups differs significantly from the intended allocation ratio (e.g., 50/50). It indicates a bug in the randomization or logging infrastructure — some users are being systematically excluded or double-counted in one variant. SRM invalidates the experiment because the groups are no longer comparable; the treated population is a biased subset. Detection: chi-squared goodness-of-fit test comparing observed vs expected counts per variant. Common causes: redirect bugs filtering mobile users, bot filtering applied asymmetrically, or cache issues with assignment. Always check for SRM before interpreting results.

from scipy import stats
observed = [48000, 52000]
expected = [50000, 50000]
_, p_srm = stats.chisquare(observed, expected)
if p_srm < 0.01:
    print('SRM detected — experiment is invalid')

CUPED is a variance reduction technique for A/B tests that uses pre-experiment data (covariate correlated with the outcome) to reduce noise and increase statistical power without increasing sample size. The adjusted outcome is Y_cuped = Y – theta * (X – E[X]), where X is the pre-experiment covariate and theta = Cov(Y,X)/Var(X) is estimated from the data. This subtracts the portion of outcome variance explained by the covariate, reducing variance proportionally to the squared correlation r^2. If r = 0.5, variance reduction is 25%; if r = 0.8, variance reduction is 64%. CUPED is equivalent to ANCOVA and is used widely at Netflix, Microsoft, and LinkedIn.

import numpy as np
def cuped_adjust(y, x_pre):
    theta = np.cov(y, x_pre)[0,1] / np.var(x_pre)
    return y - theta * (x_pre - x_pre.mean())

y_adj_control = cuped_adjust(y_control, x_pre_control)
y_adj_treat = cuped_adjust(y_treat, x_pre_treat)

Stratified randomization ensures that important covariates (e.g., country, device type, user segment) are balanced proportionally across treatment and control groups, rather than relying on simple random assignment to balance them by chance. It eliminates baseline imbalance in key confounders, reducing variance of treatment estimates and increasing power. In large-scale internet experiments, stratification is done implicitly through hashing (consistent hash of user_id + experiment_id), which distributes users uniformly but does not guarantee balance on observed covariates. Explicit stratification (block randomization) is used when covariate balance is critical — clinical trials and small-N experiments benefit most.

The novelty effect is the tendency for users to engage more with new features simply because they are new, not because the feature is intrinsically better. This inflates treatment metrics early in the experiment. As novelty wears off, the effect decays toward the true steady-state effect. Similarly, a 'change aversion' effect may suppress engagement initially if users dislike unfamiliar changes. Detection: plot treatment effect over time — if the effect decays toward zero, suspect novelty; if it grows, suspect change aversion. Mitigation: run experiments long enough to reach steady state (typically 2+ weeks), restrict analysis to new users who have no prior exposure, or analyze returning-user cohorts separately.

Network interference (also called spillover or SUTVA violation) occurs when the treatment status of one unit affects the outcome of another unit — violating the Stable Unit Treatment Value Assumption (SUTVA) required for valid causal inference. In social networks, treating users with a feature may change their behavior toward control users (sharing, messaging). In marketplaces, treating some sellers affects prices for all buyers. Standard A/B tests measure the Total Average Treatment Effect (TATE) minus spillover, potentially underestimating or overestimating the true effect. Mitigation: cluster randomization (randomize at cluster level), ego-network randomization, switchback experiments, or graph cluster approaches.

In a switchback experiment, the treatment assignment switches between treatment and control at regular time intervals (e.g., every hour or day) for all units simultaneously, rather than randomizing at the user level. This is used in marketplace, logistics, and supply/demand systems where individual randomization causes interference (e.g., pricing algorithms affect all users simultaneously). The causal estimate is the average over switches. Key considerations: the switchback period must be long enough for the system to reach equilibrium after each switch (carry-over effect), and the number of switches determines statistical power. Analysis uses a time-series approach with fixed effects for each period.

Interleaving is a method for comparing two ranking systems (e.g., recommendation or search algorithms) by presenting an interleaved list — combining results from both rankers — to the same user in a single session, then observing which ranker's results the user preferred (via clicks). Unlike A/B testing which splits users, interleaving controls for between-user variation since both rankers are evaluated on the same user behavior. This dramatically increases sensitivity, allowing 100x fewer users to detect ranking differences of the same magnitude. Two variants: Balanced Interleaving and Team Draft Interleaving. Used at Netflix, YouTube, and Spotify for search and recommendation evaluation.

The OEC is a single primary metric that represents the long-term success of the product and is used to make the go/no-go decision for a feature. A good OEC is: causally related to long-term business outcomes (revenue, retention), sensitive enough to detect meaningful changes within the experiment window, measurable for all users, not gameable by short-term hacks, and validated to correlate with long-term outcomes in holdback analyses. The OEC should be defined before the experiment. Common OECs: DAU (daily active users), session duration, purchase rate, Netflix: hours watched per member. Using multiple primary metrics inflates Type I error and requires pre-specified correction.

Guardrail metrics are metrics that must not be significantly degraded by a winning treatment, even if the primary OEC improves. They protect against tradeoffs that benefit one goal at the expense of critical others. Examples: latency (treatment must not slow the product), crash rate, customer service contacts, revenue (when OEC is engagement). Violations of guardrail metrics signal that the treatment's primary gain comes at unacceptable cost. Guardrail metrics are tested with a directional hypothesis (is this metric significantly worse?) using a one-tailed test. A treatment failing any guardrail is not shipped regardless of OEC improvement. They reflect constraints on acceptable experiment outcomes.

Regression to the mean is the phenomenon where extreme values in a measurement tend toward the average on re-measurement, not due to a real change but due to random variation. In experiments, selecting users based on an extreme metric (e.g., analyzing only users who crashed in the previous week) will show apparent improvement even without treatment, because their next period is expected to be less extreme by chance. This biases treatment effect estimates upward when targeting users selected on extreme metric values. Mitigation: use pre-experiment data for selection but analyze post-experiment outcomes; avoid selecting treatment populations based on the outcome metric; use randomization which does not suffer from this bias.

An A/B test is a fixed design: allocate traffic equally (or per a fixed ratio), run until the pre-specified sample size is reached, then make a decision. It optimizes statistical rigor but wastes traffic on inferior variants during the experiment. A multi-armed bandit (MAB) adaptively allocates more traffic to better-performing variants in real time (e.g., Thompson Sampling, UCB), balancing exploration and exploitation. MABs minimize regret (lost value from running inferior variants) and converge faster to the best variant but sacrifice statistical rigor: they do not provide valid frequentist p-values or confidence intervals for the treatment effect. Use A/B tests for inference; use MABs for maximizing cumulative reward during the decision period.

Simpson's paradox occurs when a trend that appears in aggregated data reverses or disappears when the data is disaggregated by subgroups. In experiments, this can occur when treatment and control groups have different distributions of a confounder (e.g., new users are more likely assigned to treatment, and new users have lower conversion rates). The aggregate treatment effect appears negative even though the treatment is positive within each user segment. Detection: compare group composition across variants; stratify analysis by key segments. Proper randomization prevents this, but implementation bugs causing SRM can reintroduce it. Always check segment-level results alongside aggregate results.

A holdout group is a permanently excluded user segment that never receives any experimental treatment for a sustained period (weeks to months). It enables measuring long-term effects that short-term experiments cannot capture: cumulative learning (users improve with a feature over time), network effects (the product's value grows as more users adopt a feature), and cannibalization effects. Comparing holdout users to fully treated users after ramp-up measures the Global Average Treatment Effect (GATE). Companies like Facebook maintain permanent holdout groups across all features. The trade-off is that holdout users receive an inferior product experience, creating an ethical and business cost.

Dilution occurs when users assigned to the treatment group never actually receive the treatment (e.g., they never triggered the feature), diluting the measured effect toward zero. Intent-to-Treat (ITT) analysis measures the average effect of assignment regardless of actual exposure — it underestimates the per-exposed effect. Complier Average Causal Effect (CACE) or Treatment-on-the-Treated (ToT) analysis restricts to users who actually received treatment, using instrumental variables (assignment as instrument for exposure). Dilution is most severe for features with low trigger rates. Always report both ITT and ToT for features with partial exposure, noting trigger rate as a context metric.

# ToT = ITT / trigger_rate
trigger_rate = treated_users_who_saw_feature / total_treated_users
tot_effect = itt_effect / trigger_rate

Metric sensitivity is a metric's ability to detect real effects in experiments — a sensitive metric shows statistically significant differences in experiments where a meaningful effect is known to exist. Low sensitivity means the metric rarely reaches significance, leading to inconclusive experiments. Sensitivity can be estimated via: (1) simulated A/A tests to measure variance, (2) historical experiments where the true effect is known, (3) computing minimum detectable effect (MDE) for a given budget. Strategies to increase sensitivity: variance reduction (CUPED, stratification), capping outliers, using per-user means instead of total counts, logging-transforming heavy-tailed metrics, or switching to a more granular metric.

An A/A test runs two identical variants (both control) to validate the experimental platform and check the null hypothesis that there is no difference. Expected results: roughly 5% of metrics should show p < 0.05 by chance (false positive rate = alpha). Systematic excess false positives indicates problems: incorrect variance calculation, non-independence of observations, or data processing bugs. A/A tests also validate randomization quality, check for SRM in a scenario where none is expected, and characterize the natural variance of metrics for power analysis. They should be run before launching a new experiment platform or after significant infrastructure changes.

Carryover (carry-over) effect occurs in within-subject designs (crossover trials, switchback experiments) when the treatment in one period affects outcomes in subsequent periods after treatment removal. This violates the assumption that periods are independent. For example, users who saw a recommendation algorithm improvement may form new habits that persist even after switching back to the old algorithm. Mitigation: include a washout period between treatment phases where no treatment is applied, allowing the system to return to baseline. The minimum washout duration depends on the expected persistence of behavioral change. Switchback experiments with short switching periods risk severe carryover bias.

Short-term metrics (clicks, session engagement, immediate conversions) are measurable within the experiment window but may not predict long-term business value — a clickbait feature boosts short-term engagement while destroying long-term retention. Long-term metrics (30-day retention, 6-month LTV, subscriber renewal) capture true value but require long holdout periods to measure, making them impractical for rapid iteration. The solution is to develop validated leading indicators: short-term metrics statistically validated to predict long-term outcomes through historical correlation analysis. This validation is ongoing as product and user behavior evolves. Companies maintain separate long-term measurement programs alongside rapid A/B testing.

A pre-post analysis compares a metric before and after a feature launch without a concurrent control group. It cannot distinguish the treatment effect from confounding trends: seasonality, concurrent product changes, external events (holidays, news), or natural user lifecycle trends would all appear as treatment effects. This is a before-after observational study, not a causal experiment. It may be used when randomization is impossible, but then requires strong assumptions (difference-in-differences, interrupted time series analysis with pre-period trend controls). In experimentation, always prefer concurrent randomized control groups over pre-post designs. Pre-post can complement A/B results as a sanity check but not replace them.

Revenue metrics (immediate conversions, purchase rate, ARPU) measure economic value directly but are often noisy, infrequent events with high variance that require large samples to detect. Engagement metrics (click-through rate, session duration, DAU) are lower variance, more frequent, and detect effects faster, but may not reflect actual revenue impact — features that increase engagement can decrease purchases or vice versa. The OEC must balance immediacy (measurement within experiment window) with relevance (predicts long-term revenue). Proxy metrics validated against long-term revenue through holdback analysis are the preferred solution. Never optimize engagement if the downstream revenue effect is unknown.

Bayesian A/B testing models the conversion rate (or other metric) as a random variable with a prior distribution, updates it with observed data to get a posterior, and computes the probability that variant B's conversion rate exceeds variant A's. Key outputs: P(B > A), expected loss from choosing B, and credible intervals. Unlike frequentist testing, it does not require pre-specified sample sizes, allows early stopping without Type I error inflation (using loss thresholds), and provides direct probability statements about hypotheses. The choice of prior matters for small samples. Bayesian methods are popular in online platforms for continuous monitoring. The expected loss stopping criterion (< 1% loss) is more decision-focused than p-value thresholds.

from scipy import stats
alpha_a, beta_a = 100+1, 1000-100+1  # Beta posterior
alpha_b, beta_b = 120+1, 1000-120+1
# P(B > A) via Monte Carlo
samples_a = stats.beta.rvs(alpha_a, beta_a, size=100000)
samples_b = stats.beta.rvs(alpha_b, beta_b, size=100000)
print(f'P(B>A) = {(samples_b > samples_a).mean():.3f}')

Geo-based experiments (geographic randomization) assign entire geographic regions (cities, DMAs, countries) to treatment or control rather than individual users. They are used when: user-level randomization causes interference (two-sided marketplace pricing affects all users in the same market), the feature is infrastructure-level (data center change), or measuring offline effects (store traffic from digital ads). The unit of randomization is geography, so statistical power is determined by the number of geographic units, not users — often only 10-100 geos are available, limiting power to large effects. Synthetic control and difference-in-differences methods are used for analysis.

Experiment velocity is the rate at which an organization can run, analyze, and act on experiments per unit time. Higher velocity enables faster learning and iteration. Bottlenecks: manual experiment setup, lack of self-serve tooling, long review processes, insufficient traffic for quick tests, overloaded analysis teams, and culture requiring perfect experiments. Solutions: build self-serve experiment platforms (Optimizely, Eppo, Statsig), automate SRM checks and standard analysis, enable feature flags for instant rollout/rollback, reduce OEC computation latency, train product managers in basic statistics, establish clear go/no-go decision criteria, and create a culture that treats experiments as learning opportunities rather than proof exercises.

Ratio metrics (click-through rate = clicks/impressions, revenue per user = revenue/sessions) have variance that is more complex than the variance of a simple mean. The delta method approximates variance of a ratio using: Var(Y/X) ≈ (μ_Y/μ_X)^2 * (Var(Y)/μ_Y^2 - 2*Cov(X,Y)/(μ_X*μ_Y) + Var(X)/μ_X^2). For A/B tests, using the ratio variance (delta method) rather than per-user means can substantially change confidence intervals and significance conclusions. The clustered standard error approach treats each user's sessions as a cluster, computing standard errors at the user level to account for intra-user correlation. Incorrect variance estimation is a common source of invalid A/B test results.

# Delta method for ratio variance
import numpy as np
def ratio_variance(num, den):
    mu_n, mu_d = num.mean(), den.mean()
    var_n, var_d = num.var(), den.var()
    cov = np.cov(num, den)[0,1]
    return (mu_n/mu_d)**2 * (var_n/mu_n**2 - 2*cov/(mu_n*mu_d) + var_d/mu_d**2) / len(num)

The Central Limit Theorem (Lindberg-Lévy form) guarantees that the sample mean of i.i.d. random variables is approximately normally distributed for sufficiently large N, regardless of the underlying distribution. This justifies using normal-theory tests (z-test, t-test) for A/B test metrics like revenue per user (which is log-normal or heavy-tailed) as long as N is large. The approximation quality depends on the skewness and kurtosis of the underlying distribution — heavier tails require larger N. Empirically, N > 1000 is usually sufficient for typical metrics. For very heavy-tailed metrics (e.g., ad revenue with whales), the CLT may require N > 10,000 for good approximation, or log-transformation before analysis.

P-values answer 'is the effect unlikely under the null?' but business decisions require 'is the effect large enough to justify rollout?' These are different questions. A p < 0.001 treatment effect of 0.01% conversion rate lift may not justify engineering cost. A p = 0.06 effect of 5% revenue lift may be worth shipping. Business decisions should integrate: point estimate and CI (practical significance), cost of the change, reversibility, risk of degrading other metrics, and organizational priorities. The p-value should be a filter for evidence, not the decision criterion. Decision frameworks combine statistical evidence with expected value calculation: E[lift] * traffic * revenue_per_conversion > deployment_cost.

Experiment population bias occurs when the experimental population (users in the experiment) differs systematically from the target deployment population. Sources: experiments run on a small percentage of traffic (1% ramp) may attract power users who opted in; new features may only trigger for users meeting eligibility criteria; bot filtering criteria may differ between experiment and production. The measured effect in the experiment may not generalize to full rollout. Mitigation: ramp experiments gradually to a representative population, compare experiment population attributes to the full user base, use holdback analysis at full deployment to measure the true production effect. External validity is as important as internal validity.

Testing many metrics in a single experiment inflates the family-wise Type I error rate. With 20 independent metrics at alpha = 0.05, the probability of at least one false positive is 1 - 0.95^20 ≈ 64%. Solutions: pre-specify a single primary OEC used for the go/no-go decision (other metrics are secondary/exploratory); apply Bonferroni or BH correction across all metrics; use composite metrics that aggregate multiple outcomes; distinguish between guardrail metrics (tested directionally) and exploratory metrics (labeled as such). Industry practice: accept higher false positive rates for secondary metrics but use them for directional guidance only, requiring confirmation in follow-up experiments.

Trigger analysis restricts experiment analysis to users who actually triggered the feature (were exposed to the change), rather than all assigned users. This increases statistical sensitivity by removing the noise from unexposed users who were assigned but never saw the treatment. The trigger condition must be defined independently of the treatment outcome to avoid selection bias — typically it is the action that would have exposed the user to the feature if they had been in treatment. Treatment-on-the-Triggered (ToT) or CACE estimates the effect among the population for whom the feature was relevant, which is the most actionable estimate for product decisions.

# Restrict to triggered users
triggered = df[df['triggered'] == 1]
control_triggered = triggered[triggered['variant'] == 'control']
treat_triggered = triggered[triggered['variant'] == 'treatment']
# Now analyze only this subset

Long-term retention (90-day, 180-day) requires waiting 90-180 days after experiment assignment before measuring, making it impractical for rapid iteration. During this period, users in both groups experience many other product changes, making it hard to attribute long-term retention changes to the original experiment. Solutions: validate short-term proxy metrics that predict long-term retention (e.g., second-week retention predicts 90-day retention), use surrogate endpoints, or maintain permanent holdbacks that measure long-term impacts of the overall product roadmap. Some teams estimate long-term effects via causal models using historical data showing the relationship between early and late behavior.

Experiment interaction (interaction effects between simultaneously running experiments) occurs when two experiments assign the same users to different treatments and the combined effect differs from the sum of individual effects. For example, simultaneously testing page layout and copy: a user assigned to both treatment_layout and treatment_copy may behave differently than expected from each experiment independently. Most platforms assume experiments are non-interacting (orthogonal) and use separate analyses. Detecting interactions requires factorial analysis of the joint assignment distribution. Mitigations: use disjoint user segments for high-risk experiments, sequence critical experiments, or explicitly model interactions via factorial experiment designs.

A feature flag (feature toggle) is a configuration mechanism that enables or disables code paths without deploying new code. In experimentation, feature flags enable: gradual rollout (ramp from 1% to 100% of users incrementally), instant rollback (disable a bad feature in seconds without a code deployment), experiment assignment (the flag determines variant), kill switches for emergencies, and dark launches (code deployed but not activated). Feature flag systems (LaunchDarkly, Statsig, GrowthBook) integrate with experiment platforms to provide A/B test assignment, metrics tracking, and automated alerting when guardrail metrics degrade. They decouple feature deployment from code deployment, reducing release risk.

Cook's distance measures how much a single observation influences a regression model's fitted values — it combines the observation's leverage (distance from the mean of X) and residual (distance from the fitted value). An observation with high Cook's distance has disproportionate influence on regression coefficients. In experiment analysis using regression (ANCOVA, CUPED), outlier users (revenue whales, bots with abnormal activity) can have extreme Cook's distances, biasing treatment effect estimates. Mitigation: cap metrics at a high percentile (Winsorization at 99th or 99.9th percentile) before analysis, or identify and remove confirmed bot traffic. Always inspect Cook's distance in regression-based experiment analyses.

import numpy as np
from statsmodels.stats.outliers_influence import OLSInfluence
influence = OLSInfluence(ols_result)
cooks_d = influence.cooks_distance[0]

Winsorization caps values above (or below) a specified percentile at the percentile boundary rather than removing them. For example, 99th percentile Winsorization replaces all values above the 99th percentile with the 99th percentile value. This reduces the influence of extreme outliers (revenue whales, bots) on mean-based statistics and variance, increasing statistical power. It must be applied identically to both treatment and control to avoid bias. The choice of threshold is arbitrary but 95th-99th percentile is conventional. Winsorization changes the estimand — you are now estimating the effect on the Winsorized distribution, not the original. Document the threshold and its rationale explicitly.

import numpy as np
def winsorize(x, pct=99):
    cap = np.percentile(x, pct)
    return np.minimum(x, cap)

y_control_w = winsorize(y_control)
y_treat_w = winsorize(y_treat, pct=99)  # use same cap

Modern experiment dashboards show dozens to hundreds of metrics simultaneously. With 100 metrics and alpha=0.05, expecting 5 false positives even under the global null. Practitioners must distinguish: (1) primary OEC — tested with full Type I error control; (2) guardrail metrics — tested directionally, false positive acceptable if conservative; (3) exploratory metrics — labeled as exploratory, high false positive expected. Applying BH FDR control across all exploratory metrics at FDR=0.10 limits expected false discoveries to 10% of rejections. Experiment platforms like Eppo and Statsig now integrate FDR control natively. Culture must support 'signal detection' rather than 'proof of effect' for exploratory metrics.

Expected loss (EL) is a Bayesian stopping criterion: it quantifies the expected cost of choosing one variant over another if you stopped today, under the posterior distribution. For each variant choice: EL(choose A) = E[max(0, theta_B - theta_A)], where theta is the metric (e.g., conversion rate). Choosing variant A when B is actually better costs the difference. You stop and ship when EL of the chosen variant falls below a threshold (e.g., EL < 0.0001 = 0.01% loss). This is more decision-aligned than p-values because it directly quantifies the cost of being wrong. It allows continuous monitoring without inflating Type I error, unlike repeated significance testing.

import numpy as np
def expected_loss(samples_a, samples_b):
    loss_a = np.mean(np.maximum(0, samples_b - samples_a))
    loss_b = np.mean(np.maximum(0, samples_a - samples_b))
    return loss_a, loss_b

SUTVA requires that: (1) a unit's potential outcome depends only on its own treatment assignment, not on others' assignments (no interference/spillover); and (2) there is only one version of each treatment (no hidden variations). Violations in internet experiments: social network effects (treating Alice affects Bob's behavior), marketplace equilibrium effects (changing prices for some buyers affects all sellers), and notification saturation (heavy notifications for treated users reduce attention for all). When SUTVA is violated, ITT estimates capture the net effect including spillover, not the pure direct effect. SUTVA violations require specialized designs: clustered randomization, geo experiments, or network exposure models.

Triggering defines the moment at which a user enters the experiment and begins contributing to the analysis. Proper triggering ensures only users who could be affected by the treatment are analyzed. If users are triggered too broadly (e.g., all users regardless of whether they visited the relevant page), dilution reduces sensitivity. If triggered too narrowly (e.g., only users who completed a checkout — endogenous to the treatment), selection bias inflates the effect. The trigger condition must be: (1) unaffected by treatment assignment, (2) clearly related to feature exposure, and (3) logged before the feature decision. Precise triggering dramatically improves MDE without additional traffic.

Absolute effect = treatment_metric - control_metric. Relative effect = (treatment - control) / control. Both are needed for complete interpretation. Absolute effect communicates the actual magnitude in the metric's units (e.g., +0.5 percentage points in CTR) and directly translates to business impact. Relative effect communicates the percentage change (e.g., +10% relative lift), which sounds larger and is often reported by stakeholders but can mislead when the baseline is extreme. A 50% relative lift on a 0.01% baseline is a 0.005% absolute lift — potentially negligible. Power analyses should use absolute MDE. Business cases should use expected absolute revenue impact = absolute effect * traffic * conversion value.

absolute_effect = treat_mean - control_mean
relative_effect = absolute_effect / control_mean
business_impact = absolute_effect * daily_traffic * revenue_per_conversion

Survivorship bias occurs when only users who survived (remained active) through a period are included in the analysis, excluding dropouts who may have left because of the treatment. In retention experiments, if you analyze only users who are still active at day 30, you exclude users who churned — but churned users are precisely those for whom the feature may have failed. This biases results toward positive effects. Always use cohort-based analysis with a fixed start date and include all users regardless of subsequent activity. Similarly, analyzing 'power users' at experiment end selects survivors, not a random sample. Define the analysis cohort at experiment start, not end.

Local Average Treatment Effect (LATE) measures the effect of a treatment on a specific subpopulation (compliers in IV designs; triggered users in online experiments). Global Average Treatment Effect (GATE) measures the effect across the entire target population, including users who would never be directly exposed. ITT approximates GATE when exposure is close to 100%; it underestimates it when there is dilution. In holdback analyses, GATE measures the effect of all deployed features collectively on the held-out population. For product decisions, ToT (local/triggered effect) guides feature quality; GATE from holdbacks guides portfolio-level value measurement. Both are important for complete impact understanding.

Regret in the bandit framework is the cumulative difference between the reward earned by the algorithm and the reward that would have been earned by always choosing the optimal action from the start. A fixed A/B test with 50/50 split for N periods incurs linear regret proportional to N * effect_size (half the traffic always goes to the inferior arm). A bandit algorithm (UCB, Thompson Sampling) achieves sub-linear regret — O(log N) — because it progressively shifts traffic toward the better arm. The trade-off: A/B tests achieve zero regret at decision time (they only choose the best arm after the test), while bandits minimize regret during the exploration period at the cost of statistical rigor in effect estimation.

The Fisher randomization test computes an exact p-value for an experiment by comparing the observed test statistic to the distribution of test statistics under all possible random assignments, without parametric assumptions. It treats the observed outcome values as fixed and asks: if treatment had been randomly assigned differently, how often would we see a statistic as extreme? The null hypothesis is the sharp null (no treatment effect for any individual). For large experiments, enumerate all assignments is impractical; permutation tests approximate it via Monte Carlo. It is exact, assumption-free, and particularly useful for small samples or non-standard test statistics. It forms the theoretical foundation for randomization inference in causal experiments.

import numpy as np
obs_stat = np.mean(y_treat) - np.mean(y_control)
combined = np.concatenate([y_treat, y_control])
null = [np.mean(np.random.permutation(combined)[:len(y_treat)]) -
        np.mean(np.random.permutation(combined)[len(y_treat):])
        for _ in range(10000)]
p_val = np.mean(np.abs(null) >= np.abs(obs_stat))

In two-sided marketplaces (rideshare, food delivery), individual-level randomization violates SUTVA because treating some supply-side units (drivers) affects all demand-side units (riders) through pricing and availability. Treating 50% of drivers with a new incentive changes market equilibrium for all riders, so control riders are exposed to a different market condition than the counterfactual. This makes the ITT estimate a biased approximation of the true direct effect. Solutions: geo-based or time-based (switchback) randomization that avoids market splitting, matched market experiments, or structural modeling of supply-demand equilibrium.

Between-subject design assigns each unit to only one condition (treatment or control), estimating the population-level ATE by comparing groups. It requires large samples because between-unit variability is part of the error. Within-subject design exposes the same unit to both conditions (in sequence or simultaneously via interleaving), directly measuring individual-level effects and canceling between-unit noise. Within-subject designs require far fewer participants but introduce carry-over effects, order effects, and learning. In online experiments, within-subject A/B tests are rare (except interleaving); crossover clinical trials are the classical within-subject design. The trade-off is variance reduction vs validity threats from repeated exposure.

When testing k metrics simultaneously, Bonferroni correction sets the per-metric significance threshold to alpha/k to control the family-wise error rate at alpha. For k=20 metrics at alpha=0.05, each metric requires p < 0.0025 to be considered significant. This is overly conservative for exploratory metrics (many false negatives) but appropriate for a small set of pre-specified primary metrics where any false positive would trigger an unnecessary product change. A practical approach: apply Bonferroni only to the set of pre-registered primary and guardrail metrics (typically 3-5), and label the remaining exploratory metrics with uncorrected p-values flagged as requiring replication.

MDE sensitivity analysis computes the minimum detectable effect at different traffic percentages and durations, helping teams decide how much traffic to allocate and how long to run an experiment. For a fixed alpha and power, MDE scales as 1/sqrt(N): doubling traffic halves the MDE in terms of effect size units. Presenting an MDE table (traffic% vs run-duration vs MDE) lets stakeholders make informed trade-offs: a 1% traffic allocation may only detect a 20% lift in 4 weeks, while 50% allocation detects a 3% lift in 1 week. This analysis is essential for high-traffic experiments where opportunity cost of traffic allocation must be weighed against detection sensitivity.

A full A/B test uses a fixed pre-specified split (e.g., 50/50) for the planned duration and analyzes results once. A ramp experiment gradually increases treatment exposure from a small initial percentage (1-5%) to 100%, with analysis checkpoints at each ramp stage. Ramping serves safety purposes — catching regressions early before they affect all users — rather than statistical inference goals. Results from ramped experiments should not be used for significance testing at each checkpoint (peeking problem). The correct approach: use ramp stages for safety monitoring via guardrail metrics, and run a full statistically-powered A/B test at the final stable allocation before making the go/no-go decision.

Deng, Shi, and Smyth (2013) derived the delta-method variance for ratio metrics (clicks/impressions) commonly used in A/B tests at Microsoft, addressing the underestimation of variance when treating ratio metrics as simple means. The estimator correctly accounts for the correlation between numerator and denominator within each user: Var(Y/X) ≈ (1/n) * (Var(Y) - 2*(Y/X)*Cov(X,Y) + (Y/X)^2 * Var(X)) / mu_X^2. Using this instead of the simpler per-user rate variance prevents inflated t-statistics and false positive rates, which were found to be significantly elevated in Microsoft's A/B testing platform before this correction was applied.

import numpy as np
# Correct variance for ratio metric Y/X
def ratio_var(y, x):
    n = len(y)
    mu_x = x.mean()
    var_y = np.var(y, ddof=1)
    var_x = np.var(x, ddof=1)
    cov_xy = np.cov(y, x)[0,1]
    ratio = y.sum() / x.sum()
    return (var_y - 2*ratio*cov_xy + ratio**2*var_x) / (n * mu_x**2)