Experimentation & Metrics quiz

Question 1

What is a Sample Ratio Mismatch (SRM) and why is it dangerous?

Accepted Answer

SRM means the observed variant allocation differs from the intended allocation

Answer

SRM occurs when the treatment group outperforms the control

Answer

SRM indicates the experiment ran for too long

Answer

SRM is caused by insufficient statistical power

Question 2

What is a novelty effect and how do you account for it in experiments?

Accepted Answer

Novelty effects inflate treatment metrics early, decaying toward the true effect

Answer

Novelty effects cause treatment metrics to improve over time as users learn

Answer

Novelty effects only affect control group users

Answer

The only way to correct for novelty effects is to run experiments for longer

Question 3

What is a switchback experiment (time-based randomization)?

Accepted Answer

Switchback experiments switch all units between treatment and control at timed intervals

Answer

Switchback experiments randomize individual users across time periods

Answer

Switchback experiments are identical to standard A/B tests

Answer

Switchback experiments eliminate carry-over effects by design

Question 4

What is the Overall Evaluation Criterion (OEC) in experimentation?

Accepted Answer

The OEC is a single pre-defined primary metric for go/no-go decisions

Answer

The OEC should be defined after seeing preliminary experiment results

Answer

Using multiple primary metrics improves decision accuracy without any cost

Answer

The OEC must be a revenue metric in all experiments

Question 5

What is regression to the mean and how does it affect experiment analysis?

Accepted Answer

Extreme metric values tend toward the average on re-measurement due to random variation

Answer

Regression to the mean only affects the control group

Answer

Regression to the mean is eliminated by running experiments longer

Answer

Regression to the mean is the same as novelty effect

Question 6

What is Simpson's paradox and how can it affect experiment results?

Accepted Answer

Simpson's paradox occurs when aggregate trends reverse in disaggregated subgroups

Answer

Simpson's paradox can never occur in properly randomized experiments

Answer

Simpson's paradox means treatment effects are always larger in subgroups

Answer

Simpson's paradox is the same as the Berkson's paradox

Question 7

What is the dilution effect in experiment analysis?

Accepted Answer

Dilution occurs when treated users never actually receive the treatment, diluting the effect

Answer

Intent-to-treat analysis overestimates the true treatment effect

Answer

Dilution occurs when control users receive the treatment

Answer

CACE is always larger than ITT effect estimates

Question 8

What is an A/A test and what is it used for?

Accepted Answer

An A/A test validates the platform by expecting ~5% false positive rate

Answer

An A/A test expects 100% of metrics to show p > 0.05

Answer

An A/A test compares two different treatments

Answer

An A/A test with p < 0.05 always indicates a platform bug

Question 9

What is a pre-post analysis and why is it invalid for measuring treatment effects?

Accepted Answer

Pre-post analysis cannot distinguish treatment effects from concurrent trends

Answer

Pre-post analysis is a valid causal method when run for more than 30 days

Answer

Pre-post analysis requires randomization like a standard A/B test

Answer

Pre-post analysis is always more sensitive than a randomized experiment

Question 10

What is Bayesian A/B testing and how does it differ from frequentist?

Accepted Answer

Bayesian A/B testing computes the probability that a variant is better

Answer

Bayesian A/B testing requires pre-specifying sample size like frequentist tests

Answer

Bayesian A/B testing always requires an uninformative prior

Answer

Frequentist and Bayesian A/B tests always reach the same conclusions

Question 11

What is the Lindberg-Lévy CLT application to experiment analysis?

Accepted Answer

For A/B tests, the CLT justifies normal-theory tests for large N regardless of metric distribution

Answer

The CLT only applies when the underlying distribution is normal

Answer

The CLT requires equal sample sizes in treatment and control

Answer

The CLT does not apply to proportion metrics in A/B tests

Question 12

What is the multiple metrics problem in A/B testing?

Accepted Answer

Multiple metrics inflate Type I error; pre-specifying a single OEC controls it

Answer

Testing 20 metrics at alpha=0.05 maintains 5% family-wise error rate

Answer

The OEC is tested with Bonferroni correction in most experiments

Answer

Secondary metrics should always be tested at alpha=0.01

Question 13

What is experiment interaction and how does it arise?

Accepted Answer

Experiment interactions occur when two experiments share users and effects are non-additive

Answer

Experiment interactions are impossible when using a proper randomization engine

Answer

All experiments using the same users always produce interaction effects

Answer

Factorial experiment designs eliminate all interactions by definition

Question 14

What is Winsorization and when should you apply it in experiment analysis?

Accepted Answer

Winsorization must be applied identically to treatment and control groups

Answer

Winsorization removes all outlier observations from the analysis

Answer

Winsorization is only valid when applied to the control group

Answer

Winsorizing at the 50th percentile is the standard convention

Question 15

What is expected loss in Bayesian experimentation?

Accepted Answer

Expected loss quantifies the cost of choosing the wrong variant under the posterior

Answer

Expected loss is the probability that treatment B is better than A

Answer

Expected loss cannot be used as a stopping criterion

Answer

Expected loss requires pre-specifying sample size

Question 16

What is online controlled experiment triggering and why does it matter?

Accepted Answer

The trigger condition must be unaffected by treatment assignment

Answer

Triggering broadly to all users always maximizes statistical power

Answer

Triggering should always be based on the outcome metric

Answer

Only users who complete the target action should be triggered

Question 17

What is the problem of survivorship bias in cohort analysis?

Accepted Answer

Survivorship bias occurs when churned users are excluded from retention analysis

Answer

Survivorship bias overestimates harmful treatment effects

Answer

Analyzing only active users at day 30 eliminates noise from churned users

Answer

Cohort analysis is immune to survivorship bias by design

Question 18

What is a bandit algorithm's regret and how does it compare to A/B test regret?

Accepted Answer

Bandit algorithms achieve sub-linear regret by shifting traffic toward better arms

Answer

A/B tests achieve lower regret than bandits during the experiment period

Answer

Bandit algorithms achieve zero regret by always choosing the best arm

Answer

Regret is only relevant for A/B tests, not bandit algorithms

Question 19

What is the difference between a within-subject and between-subject experiment design?

Accepted Answer

Within-subject designs reduce variance by canceling between-unit noise

Answer

Within-subject designs require more participants than between-subject designs

Answer

Between-subject designs eliminate carry-over effects by using the same unit for all conditions

Answer

Between-subject and within-subject designs always estimate the same quantity

Question 20

What is the difference between a ramp experiment and a full A/B test?

Accepted Answer

Ramp experiments gradually increase treatment exposure for safety monitoring

Answer

Ramp experiments are designed primarily for statistical inference at each checkpoint

Answer

Full A/B tests always use a 50/50 traffic split

Answer

Analyzing results at each ramp checkpoint is statistically valid

Test yourself