A/B testing and experimentation quiz

Question 1

What is the difference between Type I and Type II error in the context of an A/B test, and how do significance level (alpha) and power relate to them?

Accepted Answer

Type I = false positive (prob alpha); Type II = false negative (prob beta), power = 1-beta

Answer

Type I = false negative (prob beta); Type II = false positive (prob alpha)

Answer

Type I and Type II both refer to false positives at different alpha levels

Answer

Type I error probability equals the p-value of the test

Question 2

Why is 'peeking' at A/B test results and stopping as soon as p < 0.05 a problem, and what is the principled fix?

Accepted Answer

It inflates Type I error; fix with sequential/always-valid methods

Answer

It inflates Type II error; fix by lowering power

Answer

It biases the point estimate; fix with CUPED

Answer

It has no effect if the sample is large enough

Question 3

What is CUPED and how does it increase the sensitivity of an experiment?

Accepted Answer

A variance-reduction method using a pre-period covariate to tighten CIs

Answer

A multiple-comparison correction for many metrics

Answer

A method to randomize at the cluster level

Answer

A Bayesian prior for conversion rates

Question 4

In a two-sample test of conversion rates, why might you prefer a two-proportion z-test over a t-test, and what assumption underlies the z-test?

Accepted Answer

Binary outcomes; CLT-based normal approx valid when np and n(1-p) are large

Answer

Because the t-test cannot handle sample sizes above 30

Answer

Because proportions are always exactly normally distributed

Answer

Because the z-test does not require randomization

Question 5

What is the Sample Ratio Mismatch (SRM) check, and why is it a critical guardrail before trusting an experiment's results?

Accepted Answer

A chi-square check that observed vs expected variant counts match; failure invalidates results

Answer

A check that conversion rates are equal across variants

Answer

A way to balance covariates after the experiment

Answer

A correction for multiple metrics

Question 6

When randomizing users who interact in groups (e.g., social networks, marketplaces), why can user-level randomization violate SUTVA, and what design addresses it?

Accepted Answer

Spillover violates SUTVA; use cluster-randomized or switchback designs

Answer

User-level randomization always satisfies SUTVA regardless of network

Answer

Interference only affects power, never bias; no design change needed

Answer

Use a larger alpha to correct for interference

Question 7

What is Simpson's paradox in the context of A/B tests, and how can it arise even with proper randomization?

Accepted Answer

Aggregate effect reverses in subgroups; often a mix-shift/ratio-aggregation artifact

Answer

A type of Type II error from low power

Answer

A guaranteed sign of broken randomization

Answer

Only occurs in observational studies, never randomized tests

Question 8

Explain the delta method and why it is needed for ratio metrics like click-through rate measured per session when randomization is per user.

Accepted Answer

Taylor-expansion estimate of a ratio's variance accounting for the numerator/denominator covariance and clustering

Answer

A method to correct for multiple comparisons

Answer

A technique to impute missing conversions

Answer

A Bayesian update for ratio metrics

Question 9

What is the Minimum Detectable Effect (MDE), and how does it trade off against sample size and test duration?

Accepted Answer

Smallest detectable effect; sample size scales ~1/MDE^2

Answer

Largest effect possible; sample size scales linearly with MDE

Answer

The observed effect after the test ends

Answer

The p-value threshold for stopping

Question 10

How do you correctly compute the sample size per arm for a two-sided test comparing two means, and what inputs are required?

Accepted Answer

n = 2*(z_{1-alpha/2}+z_{1-beta})^2 * sigma^2 / delta^2 per arm

Answer

n = (z_{1-alpha/2}) * sigma / delta per arm

Answer

n = sigma^2 / delta independent of alpha and power

Answer

n = (1-beta)/alpha * sigma^2

Question 11

What is a guardrail (or counter) metric, and why include it alongside the primary success metric?

Accepted Answer

A metric that must not regress (e.g., latency, churn), monitored for non-inferiority

Answer

The single primary metric used to declare a winner

Answer

A pre-period covariate used in CUPED

Answer

A metric used only for SRM checks

Question 12

Distinguish between an OEC (Overall Evaluation Criterion) and individual feature metrics. Why does a good OEC matter?

Accepted Answer

OEC is a single long-term-value criterion guiding decisions vs narrow feature metrics

Answer

OEC is the same as the guardrail metric

Answer

OEC is the p-value combined across metrics

Answer

OEC is the observed effect on clicks only

Question 13

What is the novelty (and primacy) effect in online experiments, and how do you detect and handle it?

Accepted Answer

Transient effect from newness; detect via effect-over-time, run longer to stabilize

Answer

A permanent bias fixed by increasing alpha

Answer

A randomization bug detected by SRM

Answer

A variance issue solved by CUPED

Question 14

Explain the multiple comparisons problem when evaluating many metrics or many variants, and name two correction approaches.

Accepted Answer

Inflated false positives; correct via Bonferroni (FWER) or Benjamini-Hochberg (FDR)

Answer

Inflated false negatives; correct by raising power only

Answer

Solved automatically by larger samples

Answer

Only a concern in Bayesian tests

Question 15

What does a 95% confidence interval for a treatment effect actually mean, and what is a common misinterpretation?

Accepted Answer

Long-run 95% of such intervals cover the truth; not P(truth in this interval)=0.95

Answer

There is exactly a 95% probability the true effect is in this interval

Answer

95% of the data falls within the interval

Answer

The interval contains 95% of future observations

Question 16

Compare frequentist and Bayesian A/B testing. What does a Bayesian approach give you that a p-value does not?

Accepted Answer

Bayesian gives P(B>A) and expected loss via a posterior; frequentist gives error-rate-controlled p-values

Answer

Bayesian eliminates the need for randomization

Answer

Frequentist gives the probability the variant is better directly

Answer

They always produce identical decisions

Question 17

What is a switchback experiment, and when is it the right design?

Accepted Answer

Alternates the system between treatment/control over time windows; used under marketplace interference

Answer

Randomizes users into two groups for the whole test

Answer

A sequential test that stops early

Answer

A method to reduce variance using pre-period data

Question 18

Why might statistically significant results not be practically significant, and how should this affect ship decisions?

Accepted Answer

Large n makes tiny effects significant; judge by effect size/CI and cost, not just p

Answer

Statistical significance always implies practical importance

Answer

Practical significance is just a lower alpha

Answer

Non-significant results can never be acted upon

Question 19

What is variance reduction via stratification (or post-stratification), and how does it differ from CUPED?

Accepted Answer

Stratification removes between-stratum variance via groups; CUPED regresses out a numeric pre-period covariate

Answer

They are identical techniques with different names

Answer

Stratification adds bias while CUPED does not

Answer

CUPED requires categorical strata

Question 20

What is an A/A test and what is it used for?

Accepted Answer

Two identical variants to validate the pipeline; p-values should be ~uniform, FPR ~ alpha

Answer

A test with two different treatments and no control

Answer

A test to measure the maximum possible effect

Answer

A Bayesian-only diagnostic

Question 21

In a ramped rollout, why can comparing treatment users to control users across different ramp stages introduce bias, and how do you avoid it?

Accepted Answer

Different ramp stages mix non-comparable cohorts; use concurrent randomized control throughout

Answer

Ramping never biases results if traffic is large

Answer

Compare post-ramp treatment to pre-ramp baseline to fix it

Answer

Bias is removed automatically by SRM checks

Question 22

What is the intention-to-treat (ITT) principle, and why is it preferred over per-protocol analysis in experiments with non-compliance?

Accepted Answer

ITT analyzes by assignment, preserving randomization; per-protocol introduces selection bias

Answer

Per-protocol is unbiased while ITT is biased

Answer

ITT only applies to Bayesian tests

Answer

ITT requires excluding non-compliers

Question 23

How does a heavy-tailed or skewed metric distribution (e.g., revenue per user) affect A/B test validity, and what mitigations exist?

Accepted Answer

Heavy tails harm CLT/power; mitigate via winsorizing, transforms, bootstrap, CUPED

Answer

Skew has no effect because of randomization

Answer

Always use the median, which is unbiased for the mean effect

Answer

Increase alpha to compensate for outliers

Question 24

What is a holdout (long-term) group, and why do teams maintain one despite shipping winning experiments?

Accepted Answer

A persistent old-experience group measuring cumulative long-term impact of launches

Answer

A second control used only for SRM

Answer

A group that always receives all treatments

Answer

A Bayesian prior group

Question 25

Explain why the p-value distribution under the null hypothesis is uniform, and how this property is used in practice.

Accepted Answer

Under H0 p-value = CDF(statistic) ~ Uniform(0,1); used to validate calibration via A/A tests

Answer

p-values are normally distributed under H0

Answer

Uniformity only holds under the alternative hypothesis

Answer

p-values cluster near 0 under the null

Question 26

What is the difference between absolute and relative (lift) treatment effects, and why does relative lift require care with confidence intervals?

Accepted Answer

Absolute = raw diff; relative = ratio needing delta-method CIs for the denominator

Answer

They are identical except for units

Answer

Relative lift CIs are always narrower than absolute

Answer

Absolute effect requires the delta method, relative does not

Question 27

Why is randomization at the right unit (e.g., user vs cookie vs session) important, and what problem arises if a user is split across variants?

Accepted Answer

Inconsistent unit lets a user see both variants, diluting the effect; use stable user IDs

Answer

Session-level randomization is always best for retention

Answer

Splitting a user across variants increases power

Answer

The unit choice does not affect bias

Question 28

What is a quantile (e.g., p95 latency) treatment effect, and why can't you just take the difference of per-group quantiles and use a standard z-test?

Accepted Answer

Quantile variance depends on the density at the quantile; use bootstrap/influence-function inference

Answer

Quantile differences follow the same z-test as proportions

Answer

Quantiles have zero variance so no test is needed

Answer

You must always use the mean instead

Question 29

How does an interaction effect between two simultaneously running experiments threaten validity, and how do large experimentation platforms manage it?

Accepted Answer

Use orthogonal randomization so effects average out; isolate conflicting tests in exclusive layers

Answer

Run only one experiment at a time, always

Answer

Interactions cannot occur with randomization

Answer

Increase alpha to absorb interactions

Question 30

What is the difference between cluster-robust standard errors and ignoring clustering when the metric is measured at a finer grain than the randomization unit?

Accepted Answer

Ignoring clustering underestimates SEs causing false positives; cluster-robust SEs fix it

Answer

Clustering inflates SEs and reduces power only

Answer

Clustering has no effect if events are many

Answer

Cluster-robust SEs are only for Bayesian tests

Test yourself