Hypothesis Testing

A p-value is the probability of observing test results at least as extreme as those actually observed, assuming the null hypothesis is true. It is NOT the probability that the null hypothesis is true, nor the probability that the result was due to chance. A small p-value (e.g., < 0.05) means the data would be unlikely under the null, providing evidence against it. The threshold alpha is set a priori as the acceptable Type I error rate. p-values are widely misinterpreted; they say nothing about effect size, practical significance, or the probability the alternative hypothesis is true.

from scipy import stats
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f'p-value: {p_value:.4f}')

A Type I error (false positive) occurs when you reject a true null hypothesis — concluding an effect exists when it does not. Its probability is alpha (significance level), set by the researcher. A Type II error (false negative) occurs when you fail to reject a false null hypothesis — missing a real effect. Its probability is beta, and power = 1 - beta. There is a trade-off: reducing alpha (stricter significance threshold) increases beta (harder to detect real effects). Power analysis balances these by determining the sample size needed to achieve target power (typically 0.80) at a given alpha and minimum detectable effect size.

Statistical power is the probability of correctly rejecting a false null hypothesis (detecting a real effect) — it equals 1 - beta. Factors affecting power: sample size (larger N increases power), effect size (larger effects are easier to detect), significance level alpha (less strict = more power), and variance (less noise = more power). Sample size for a two-sample t-test is computed from Cohen's d (effect size), alpha, and target power using the formula or scipy's tt_ind_solve_power. Typically power = 0.80 and alpha = 0.05 are used, meaning 80% chance of detecting a real effect.

from statsmodels.stats.power import tt_ind_solve_power
n = tt_ind_solve_power(effect_size=0.2, alpha=0.05, power=0.80)
print(f'Required n per group: {n:.0f}')

Cohen's d measures the standardized difference between two group means: d = (mu1 - mu2) / pooled_std. It expresses effect size in standard deviation units, making it comparable across studies with different scales. Conventional benchmarks (Cohen 1988): small = 0.2, medium = 0.5, large = 0.8. A d of 0.5 means the means differ by half a standard deviation — approximately 69% of one distribution overlaps with the other. Cohen's d is independent of sample size, unlike p-values which conflate effect size and sample size. Always report effect sizes alongside p-values for complete results interpretation.

import numpy as np
def cohens_d(a, b):
    pooled_std = np.sqrt((a.std()**2 + b.std()**2) / 2)
    return (a.mean() - b.mean()) / pooled_std

Use a paired t-test when each observation in one group is matched or linked to a specific observation in the other group — before/after measurements on the same subjects, matched pairs by confounders, or repeated measures. The paired test analyzes the differences within pairs, eliminating between-subject variability, which increases power over an independent-samples test. The independent-samples t-test assumes the two groups are drawn from separate populations with no correspondence between observations. Using an independent-samples test on paired data wastes statistical power; using a paired test on independent data incorrectly reduces degrees of freedom and may increase Type I error.

from scipy import stats
# Paired: before/after on same subjects
t_stat, p_val = stats.ttest_rel(before, after)
# Independent: different subjects
t_stat, p_val = stats.ttest_ind(group_a, group_b)

The Bonferroni correction controls the family-wise error rate (FWER) by adjusting the significance threshold to alpha/m, where m is the number of tests. This ensures the probability of making even one Type I error across all tests stays at alpha. It is appropriate when any false positive is costly (clinical trials, safety-critical systems). However, it becomes overly conservative when tests are positively correlated (common in practice), when m is large (genomics with 100,000 tests), or when the cost of false negatives is high. Alternatives like Holm-Bonferroni (sequential, uniformly more powerful) or Benjamini-Hochberg FDR control are preferred for large-scale testing.

alpha = 0.05
m = 20  # number of tests
bonferroni_threshold = alpha / m  # 0.0025

The Benjamini-Hochberg (BH) procedure controls the False Discovery Rate (FDR) — the expected proportion of rejected nulls that are false positives — rather than the FWER. Algorithm: sort m p-values in ascending order; find the largest k where p(k) <= (k/m)*alpha; reject all hypotheses with p-values up to p(k). FDR = 0.05 means at most 5% of your significant results are expected to be false positives. BH is far less conservative than Bonferroni when m is large, making it standard in genomics, neuroimaging, and large-scale A/B testing with many metrics. statsmodels.stats.multitest.multipletests implements it.

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method='fdr_bh')
significant = p_adj[reject]

The Mann-Whitney U test (Wilcoxon rank-sum test) is a non-parametric alternative to the two-sample t-test that compares the rank-ordering of two groups rather than their means. It tests whether one group tends to have larger values than the other. Prefer it when: data is ordinal, data is not normally distributed and N is small, heavy outliers distort means, or the metric is revenue/latency (often log-normal or heavy-tailed). It requires no normality assumption. Its null hypothesis is that both groups are sampled from the same distribution. Power is ~95% of the t-test when normality holds, and higher when data is non-normal.

from scipy import stats
stat, p_val = stats.mannwhitneyu(group_a, group_b, alternative='two-sided')

A two-tailed test detects effects in either direction (treatment could increase or decrease the metric) and is the standard default. The rejection region is split across both tails: alpha/2 in each. A one-tailed test focuses the rejection region in one direction — only detecting increases (or only decreases), making it more powerful in that direction but blind to effects in the opposite direction. One-tailed tests are only appropriate when a reversal would have the same consequence as no effect and would not be acted upon. In practice, two-tailed tests are strongly preferred to avoid confirming expectations by design and to detect unexpected harmful effects.

# Two-tailed (default)
stats.ttest_ind(a, b, alternative='two-sided')
# One-tailed (greater)
stats.ttest_ind(a, b, alternative='greater')

A permutation test (randomization test) estimates the null distribution empirically by repeatedly shuffling the group labels and recomputing the test statistic, without relying on parametric distributional assumptions. The p-value is the fraction of permutations where the shuffled statistic is as extreme as the observed. It is more appropriate when: the data is small-N, the distribution is unknown or non-standard, the test statistic is a complex quantity (e.g., difference in medians, AUC), or you want an exact p-value without asymptotic approximations. The cost is computational: typically 10,000+ permutations needed for reliable p-values below 0.001.

import numpy as np
observed = np.mean(group_a) - np.mean(group_b)
combined = np.concatenate([group_a, group_b])
n_a = len(group_a)
null_dist = [np.mean(np.random.permutation(combined)[:n_a]) -
             np.mean(np.random.permutation(combined)[n_a:])
             for _ in range(10000)]
p_val = np.mean(np.abs(null_dist) >= np.abs(observed))

A bootstrap CI is computed by repeatedly resampling with replacement from the observed data, computing the statistic of interest on each resample, and taking quantiles of the resulting distribution. The percentile method CI uses the 2.5th and 97.5th percentiles for a 95% CI. The BCa (bias-corrected and accelerated) method adjusts for bias and skewness. Unlike parametric CIs (which assume a known sampling distribution, e.g., t-distribution for means), bootstrap CIs make no distributional assumptions and work for any statistic — median, correlation, AUC, quantile. They require sufficient data (N > 30) to work well and many resamples (1000-10000) for stability.

import numpy as np
data = np.array([...])
n_boot = 10000
boot_means = [np.mean(np.random.choice(data, len(data), replace=True))
              for _ in range(n_boot)]
ci = np.percentile(boot_means, [2.5, 97.5])

The Shapiro-Wilk test assesses whether a sample was drawn from a normal distribution. The null hypothesis is normality; a significant result (p < 0.05) indicates non-normality. It is one of the most powerful normality tests for small samples (N < 50). For larger samples, even tiny deviations from normality produce significant results due to high power — making it over-sensitive. Alternative normality tests: Kolmogorov-Smirnov (with Lilliefors correction), D'Agostino-Pearson (tests skewness and kurtosis). In practice, t-tests are robust to moderate non-normality for N > 30 (Central Limit Theorem), so normality testing is most critical for small samples.

from scipy import stats
stat, p_val = stats.shapiro(data)
if p_val < 0.05:
    print('Evidence of non-normality')

Levene's test assesses equality of variances (homoscedasticity) across groups, testing the null hypothesis that population variances are equal. It is more robust than Bartlett's test when data is non-normal. The standard two-sample t-test assumes equal variances; if this assumption is violated, Welch's t-test (which does not assume equal variances and adjusts degrees of freedom) should be used instead. Many practitioners recommend always using Welch's t-test regardless of Levene's result, as it performs nearly as well as Student's t-test when variances are equal and is substantially better when they differ. scipy.stats.ttest_ind uses Welch's by default with equal_var=False.

from scipy import stats
_, p_levene = stats.levene(group_a, group_b)
# Always use Welch's (equal_var=False is default in scipy)
t, p = stats.ttest_ind(group_a, group_b, equal_var=False)

One-way ANOVA tests whether the means of three or more independent groups are all equal, extending the t-test to multiple groups. It partitions total variance into between-group variance (signal) and within-group variance (noise), computing the F-statistic as their ratio. Assumptions: independence of observations, normality within groups (robust to violation with large N), and homoscedasticity (equal variances — use Welch's ANOVA or Kruskal-Wallis if violated). A significant F-test only indicates at least one group differs; post-hoc tests (Tukey HSD, Bonferroni, Games-Howell) identify which pairs differ. Never run multiple t-tests instead — it inflates Type I error.

from scipy import stats
f_stat, p_val = stats.f_oneway(group1, group2, group3)
# Post-hoc
from statsmodels.stats.multicomp import pairwise_tukeyhsd
result = pairwise_tukeyhsd(data, groups, alpha=0.05)

The Kruskal-Wallis test is the non-parametric analogue of one-way ANOVA for comparing three or more independent groups without assuming normality. It tests whether the groups come from the same distribution by ranking all observations together and comparing mean ranks across groups. It is appropriate for ordinal data, small samples, heavy-tailed distributions, or when ANOVA assumptions are severely violated. A significant result means at least one group has a different distribution. Post-hoc pairwise comparisons use Dunn's test with multiple-comparison corrections. The test is less powerful than ANOVA when normality holds but more robust to distributional violations.

from scipy import stats
stat, p_val = stats.kruskal(group1, group2, group3)

The chi-squared test of independence assesses whether two categorical variables are associated or independent. It compares observed cell frequencies in a contingency table to expected frequencies under the independence assumption (expected = row_total * col_total / grand_total). The chi-squared statistic sums (observed - expected)^2 / expected across all cells. Assumptions: expected frequency >= 5 in each cell; use Fisher's exact test for smaller samples. Effect size: Cramér's V = sqrt(chi2 / (n * min(r-1, c-1))). Common applications: marketing (does gender affect product preference?), medicine (is treatment associated with outcome?), and data quality checks.

from scipy import stats
contingency = [[30, 10], [20, 40]]
chi2, p_val, dof, expected = stats.chi2_contingency(contingency)

Fisher's exact test computes the exact probability of observing a contingency table as extreme as or more extreme than observed, under the null hypothesis of independence. Unlike chi-squared which uses a large-sample approximation, Fisher's test is exact for any sample size. Use it when: any cell has an expected count below 5, the total sample is small (N < 20), or a 2x2 table structure is used. It is computationally intensive for large tables; chi-squared is preferred for large samples where the approximation is accurate. Fisher's test can be extended to larger tables (Fisher-Freeman-Halton test) at high computational cost.

from scipy import stats
odds_ratio, p_val = stats.fisher_exact([[8, 2], [1, 5]])

The two-sample KS test compares the empirical cumulative distribution functions (ECDFs) of two samples, testing whether they come from the same distribution. The KS statistic is the maximum absolute difference between the two ECDFs. It detects differences in location, scale, and shape, making it more general than the t-test which only detects mean differences. It is non-parametric and requires no distributional assumptions. Limitations: less sensitive to differences in the tails than tests like Anderson-Darling; for very large samples, small trivial differences become significant. Useful for model output distribution monitoring and data drift detection in ML pipelines.

from scipy import stats
stat, p_val = stats.ks_2samp(sample_old, sample_new)
print(f'KS stat: {stat:.3f}, p-value: {p_val:.4f}')

A Bayes factor (BF) is the ratio of the marginal likelihood of data under the alternative hypothesis to the marginal likelihood under the null. BF > 1 supports the alternative; BF < 1 supports the null. Unlike p-values, Bayes factors quantify evidence for both hypotheses, can support the null (not just fail to reject it), and update prior beliefs to posterior beliefs via Bayes theorem. Jeffreys' scale: BF of 3-10 is moderate evidence, 10-30 is strong, >100 is decisive. Bayes factors depend on the prior specification for the alternative, which is a limitation requiring careful choice. BayesFactor R package and pingouin in Python compute them.

# pingouin Bayesian t-test
import pingouin as pg
result = pg.ttest(group_a, group_b, correction=True)
print(result['BF10'])  # Bayes factor

Traditional fixed-sample tests require committing to a sample size before data collection; peeking at results and stopping early inflates Type I error (optional stopping problem). Sequential testing designs allow valid interim analyses with pre-specified stopping rules. Methods include: Sequential Probability Ratio Test (SPRT) by Wald, which computes a likelihood ratio and stops when it crosses upper (reject null) or lower (accept null) boundaries; Alpha-spending functions (O'Brien-Fleming, Pocock) that allocate the significance budget across planned looks; and Always-valid confidence intervals (e-values framework). These are essential for real-time A/B testing platforms where early stopping is operationally necessary.

When testing m hypotheses simultaneously, the probability of at least one false positive increases rapidly (familywise error rate, FWER) even with alpha = 0.05 per test. FWER control ensures the probability of any false positive across all tests is at most alpha (Bonferroni, Holm). FDR (False Discovery Rate) controls the expected proportion of rejected nulls that are false positives — less strict, allowing more discoveries. FDR control (Benjamini-Hochberg) is preferred in exploratory science where some false positives are acceptable. FWER control is preferred in confirmatory settings or when any false positive has serious consequences. Choice depends on the cost of false positives vs false negatives.

from statsmodels.stats.multitest import multipletests
# Holm (FWER)
reject_holm, _, _, _ = multipletests(p_vals, method='holm')
# BH (FDR)
reject_bh, _, _, _ = multipletests(p_vals, method='fdr_bh')

Minimum Detectable Effect (MDE) is the smallest true effect size that a test can reliably detect (with specified power, typically 80%) given a fixed sample size and alpha. It defines the practical resolution of your experiment. MDE is calculated by inverting the sample size formula: given N, alpha, and power, solve for the effect size. If the MDE is larger than the practically important effect (e.g., experiment can only detect a 15% lift but a 5% lift is business-meaningful), you need more data. MDE communicates practical constraints to stakeholders and helps determine if an experiment is feasible within available traffic or time.

from statsmodels.stats.power import tt_ind_solve_power
# Given N, find MDE
mde = tt_ind_solve_power(nobs1=500, alpha=0.05, power=0.80)
print(f'MDE (Cohen d): {mde:.3f}')

Statistical significance (p < alpha) indicates that an observed effect is unlikely under the null hypothesis given the sample size. Practical significance (effect size) indicates whether the effect is large enough to matter in the real world. With large samples, even tiny, practically irrelevant effects become statistically significant (e.g., p < 0.001 for a 0.01% conversion rate lift). With small samples, large important effects may not reach statistical significance. Always report both: the p-value and the effect size with confidence interval. Business decisions should be based on practical significance (will this effect generate enough value to justify rollout?) not just statistical significance.

Welch's ANOVA (Welch's F-test for multiple groups) is a generalization of Welch's t-test to three or more groups, testing for mean differences without assuming equal variances. It is the recommended alternative to standard one-way ANOVA when Levene's test indicates heteroscedasticity (unequal variances) or when group sizes are unequal. Unlike the standard F-test, it adjusts degrees of freedom to account for variance inequality. Post-hoc comparisons after Welch's ANOVA use Games-Howell test (which also does not assume equal variances). pingouin.welch_anova() provides a convenient implementation.

import pingouin as pg
result = pg.welch_anova(data=df, dv='score', between='group')
# Post-hoc: Games-Howell
ph = pg.pairwise_gameshowell(data=df, dv='score', between='group')

The Wilcoxon signed-rank test is the non-parametric paired equivalent of the paired t-test. It tests whether the median of paired differences is zero, using both the sign and magnitude of differences (unlike the sign test which only uses signs). It ranks the absolute differences and compares the sum of positive-rank vs negative-rank sums. It assumes the differences are symmetrically distributed around the median (not necessarily normal). Use it for before/after studies with small samples, ordinal outcomes, or severely non-normal difference distributions. It has about 95% efficiency compared to the paired t-test under normality.

from scipy import stats
stat, p_val = stats.wilcoxon(before, after, alternative='two-sided')

False Positive Rate (FPR, or Type I error rate) = FP / (FP + TN) — the proportion of true null hypotheses incorrectly rejected. It is controlled per test by alpha. False Discovery Rate (FDR) = FP / (FP + TP) — the proportion of rejected hypotheses that are actually false positives among all rejections. FDR only makes sense in the context of multiple testing: if you reject 100 hypotheses and control FDR at 5%, you expect at most 5 false discoveries. FPR control (FWER) is more stringent; FDR control allows more discoveries at the cost of some false positives. In genomics, controlling FDR at 5% is standard for 100,000+ tests.

A 95% confidence interval (CI) means that if you repeated the experiment many times and computed a CI each time, approximately 95% of those intervals would contain the true population parameter. It does NOT mean there is a 95% probability the true parameter lies in the computed interval (that interval either contains it or doesn't). The width of the CI reflects precision: narrower CIs come from larger samples or lower variance. CIs are more informative than p-values because they quantify the range of plausible effect sizes. A CI that excludes zero is equivalent to a two-tailed significant test at the corresponding alpha.

from scipy import stats
cl, cu = stats.t.interval(0.95, df=len(data)-1,
                           loc=data.mean(),
                           scale=stats.sem(data))

Tukey's Honest Significant Difference (HSD) controls the family-wise error rate for all pairwise comparisons after a significant ANOVA, but is less conservative than Bonferroni when all pairwise comparisons are being made simultaneously. Tukey's HSD uses the studentized range distribution and assumes equal group sizes and equal variances. For all pairwise comparisons: prefer Tukey HSD. For a subset of comparisons or non-pairwise contrasts: prefer Bonferroni or Scheffé. When variances or group sizes are unequal: use Games-Howell. Tukey HSD provides simultaneous confidence intervals for all pairwise differences, making it informative beyond just p-values.

from statsmodels.stats.multicomp import pairwise_tukeyhsd
result = pairwise_tukeyhsd(endog=df['score'],
                            groups=df['group'],
                            alpha=0.05)
result.plot_simultaneous()

A two-tailed hypothesis test at significance level alpha is equivalent to checking whether the (1-alpha) confidence interval excludes the null value. If a 95% CI for the difference in means does not include zero, the two-tailed t-test is significant at alpha = 0.05. This duality means CIs and tests are informationally equivalent but CIs communicate more — they show the range of plausible effect sizes, not just a binary reject/fail-to-reject decision. This equivalence holds for symmetric tests and intervals. For asymmetric tests or one-tailed tests, the correspondence requires care. Bayesian credible intervals do NOT have this exact equivalence with frequentist tests.

Data peeking (optional stopping) means repeatedly checking p-values as data accumulates and stopping when p < 0.05. This severely inflates the Type I error rate: with alpha = 0.05 and unlimited peeking, the probability of a false positive approaches 1. The fundamental issue is that each peek is an implicit hypothesis test, and running multiple tests on growing data without correction violates the assumption underlying the p-value calculation. Solutions: pre-register the sample size and analysis plan (confirmatory), use sequential testing methods (SPRT, alpha spending) that explicitly account for interim looks, or use always-valid confidence intervals. Many published A/B test results are biased by peeking.

The sign test is the simplest non-parametric test for paired data, assessing whether the number of positive differences exceeds negative differences (testing median difference = 0). It uses only the sign (direction) of each paired difference, ignoring magnitude, making it robust to outliers and appropriate for ordinal or heavily skewed data. It follows a binomial distribution under the null of p = 0.5. Its weakness is very low power compared to the Wilcoxon signed-rank test (which uses magnitudes) or the paired t-test. Use the sign test when only ordinal direction is reliable (e.g., preference surveys where magnitudes are meaningless) or when the assumption of symmetric differences for Wilcoxon is violated.

from scipy import stats
diffs = [a - b for a, b in zip(group_a, group_b)]
pos = sum(d > 0 for d in diffs)
p_val = stats.binomtest(pos, len(diffs), p=0.5).pvalue

The Friedman test is the non-parametric equivalent of repeated-measures ANOVA, used when the same subjects are measured under three or more conditions and normality cannot be assumed. It ranks observations within each subject (block) and tests whether the mean ranks differ across conditions. Appropriate for: ordinal data across multiple conditions, small samples with repeated measures, or when sphericity (equal variance of differences) assumption is violated. Post-hoc comparisons use Nemenyi test or Wilcoxon signed-rank tests with Bonferroni correction. Common in preference ranking studies, clinical crossover trials, and ML model comparison across multiple datasets.

from scipy import stats
stat, p_val = stats.friedmanchisquare(condition1, condition2, condition3)

Eta-squared (η²) measures the proportion of total variance in the outcome explained by the grouping factor: η² = SS_between / SS_total. Partial eta-squared (η²_p) is the proportion explained by the factor relative to the factor's variance plus error variance, excluding variance explained by other factors: η²_p = SS_effect / (SS_effect + SS_error). Partial eta-squared is the standard effect size for ANOVA reported in psychological and medical literature. Benchmarks: small = 0.01, medium = 0.06, large = 0.14. Omega-squared (ω²) is a less biased estimator recommended for small samples. These measures communicate practical significance beyond the F-test p-value.

import pingouin as pg
aov = pg.anova(data=df, dv='score', between='group', detailed=True)
print(aov[['Source','F','p-unc','np2']])  # np2 = partial eta^2

Equivalence testing (TOST — Two One-Sided Tests) tests whether an effect is practically negligible rather than simply absent. It inverts the hypothesis structure: the null hypothesis is that the effect is outside a pre-specified equivalence margin [–delta, +delta]; rejecting the null means the effect is within the margin (equivalence). Two one-sided t-tests check H0: effect <= –delta and H0: effect >= +delta; rejecting both confirms equivalence. Used to demonstrate that a new treatment is as effective as a standard, that a simplified model performs equivalently to a complex one, or that a system change does not degrade a metric. Equivalence margins must be defined before data collection.

import pingouin as pg
# TOST equivalence test
result = pg.ttest(group_a, group_b)
# Check if CI is within equivalence bounds [-delta, +delta]

A likelihood ratio test (LRT) compares two nested statistical models — a restricted (null) model and an unrestricted (alternative) model — by computing the ratio of their maximum likelihoods. The test statistic is –2 * ln(L_null / L_alt), which follows a chi-squared distribution with degrees of freedom equal to the number of additional parameters in the alternative model. LRT is the most powerful test for nested models (Neyman-Pearson lemma). It is used in logistic regression, mixed models, and generalized linear models to test whether adding predictors significantly improves model fit. For non-nested models, AIC/BIC are alternatives.

from scipy import stats
import numpy as np
# LR stat
lr_stat = -2 * (log_lik_null - log_lik_alt)
p_val = stats.chi2.sf(lr_stat, df=delta_params)

False Omission Rate (FOR) = FN / (FN + TN) — the proportion of predicted negatives that are actually positive (missed positives among all non-discoveries). It complements FDR (which focuses on false positives among discoveries). In hypothesis testing, FOR relates to Type II errors: when you fail to reject the null, how often is the null actually false? High FOR indicates low power — you are missing many real effects. Balancing FDR and FOR requires thinking about the prior probability of effects being real, the cost of false positives vs false negatives, and optimal decision thresholds. Bayesian decision theory formalizes this trade-off.

Dunnett's test is a post-hoc test designed specifically for comparing multiple treatment groups against a single control group, rather than all pairwise comparisons. It controls FWER for only the k comparisons of interest (each treatment vs control), making it more powerful than Bonferroni for this specific design. Unlike Tukey HSD which tests all k*(k-1)/2 pairs, Dunnett's tests only k comparisons. It is standard in drug trials (multiple doses vs placebo), feature experiments (multiple variants vs baseline), and quality control (multiple machines vs a reference). The test uses the multivariate t-distribution and assumes equal variances.

# Using statsmodels
from statsmodels.stats.multicomp import MultiComparison
mc = MultiComparison(df['score'], df['group'])
result = mc.allpaircomp()  # or Dunnett via R interface

Pre-registration is the practice of publicly committing to a study's hypothesis, design, sample size, primary outcome metric, and analysis plan before data collection begins. It prevents: p-hacking (testing multiple metrics and reporting only significant ones), HARKing (Hypothesizing After Results are Known — presenting post-hoc findings as a priori), and selective reporting. Pre-registered studies are confirmatory; exploratory analyses of the same data are labeled as such. Pre-registration is standard in clinical trials (ClinicalTrials.gov), increasingly required in psychology and social science (OSF.io), and being adopted in tech industry experimentation platforms to ensure A/B test validity.

Power is always calculated against a specific alternative hypothesis value, not a general 'alternative is true' statement. For a t-test, power against delta = 0.5 SD difference is much higher than against delta = 0.1 SD with the same N. The power curve plots power as a function of the true effect size, showing how the test performs across the full range of alternative values. Power analysis should specify the minimum effect size worth detecting (MDE), not an arbitrary 'expected effect.' Power increases with N, decreases with alpha (more conservative = less power for same N), and increases with effect size. Power is typically reported as a curve, not a single number.

from statsmodels.stats.power import TTestIndPower
analysis = TTestIndPower()
analysis.plot_power(dep_var='nobs', nobs=np.arange(10,300),
                    effect_size=np.array([0.2,0.5,0.8]))

The mid-p value adjusts exact discrete test p-values (from binomial, Fisher's exact, or Poisson tests) by subtracting half the probability of the observed outcome: mid-p = p_upper – 0.5 * P(X = observed). Standard exact tests are conservative (true Type I error well below alpha) because discrete distributions cannot achieve exactly alpha. The mid-p correction makes tests less conservative and better calibrated (actual Type I error closer to nominal alpha) without being liberal on average. It is recommended when exact tests are overly conservative and missing detections is costly, but is not universally accepted since it can occasionally exceed the nominal alpha.

Randomization is the foundation of valid causal inference in experiments. Random assignment of subjects to treatment and control ensures that confounding variables (observed and unobserved) are balanced in expectation across groups, making the treatment the only systematic difference. Without randomization, observed differences may be due to selection bias rather than the treatment. Randomization also justifies the use of probability-based p-values under the randomization (permutation) distribution, without requiring distributional assumptions. Stratified randomization ensures balance on key confounders. Cluster randomization is used when individual randomization is not possible. The validity of every hypothesis test in an A/B experiment rests on this foundation.

The Jarque-Bera test assesses normality by testing whether sample skewness and excess kurtosis match those of a normal distribution (skewness=0, excess kurtosis=0). The test statistic combines squared skewness and squared excess kurtosis, following a chi-squared distribution with 2 degrees of freedom asymptotically. It is appropriate for large samples and is commonly used in econometrics and finance to test residual normality in regression models. Compared to Shapiro-Wilk, it is less powerful for small samples but computationally fast for large datasets. A significant result indicates non-normality due to skewness, heavy tails, or both.

from scipy import stats
stat, p_val = stats.jarque_bera(residuals)
print(f'JB stat: {stat:.3f}, p: {p_val:.4f}')

P-hacking (data dredging) refers to any practice that exploits researcher degrees of freedom to achieve p < 0.05: testing multiple outcomes and reporting only significant ones, segmenting data until a significant subgroup is found, adding or removing covariates until significance is achieved, stopping early when p dips below threshold, or transforming variables selectively. Each additional test inflates the actual Type I error rate well above the nominal alpha. P-hacking explains much of the replication crisis in psychology and medicine. Defenses: pre-registration, pre-specified analysis plans, correcting for multiple comparisons, reporting all tests conducted, and requiring replication in an independent dataset.

Cochran's Q test is a non-parametric test for comparing binary outcomes across three or more related groups (repeated measures design with dichotomous response). It is the extension of McNemar's test (which handles two groups) to multiple groups. Subjects are measured under k conditions, and the test assesses whether the proportion of 'successes' differs significantly across conditions. It uses a chi-squared approximation. It is appropriate for: comparing click-through rates across multiple UI designs on the same users, testing whether treatment preferences differ across multiple sessions, or evaluating binary outcomes in crossover studies.

from statsmodels.stats.contingency_tables import cochrans_q
result = cochrans_q(np.array([
    [1,1,0], [1,0,0], [0,0,1], [1,1,1]
]))

Null hypothesis significance testing (NHST) frames inference as a binary decision: reject or fail to reject the null. It answers whether an effect exists but says little about its magnitude or precision. The estimation approach frames inference as quantifying the effect size with uncertainty: report the point estimate (mean difference, relative risk, odds ratio) and its confidence interval. The estimation approach is richer: a CI that is entirely above zero and narrow (e.g., [0.15, 0.20] percentage points) tells a very different story than one that spans [0.01, 0.80]. Modern statistical practice increasingly emphasizes estimation over binary decisions, as promoted by the New Statistics movement and the ASA's 2019 statement on p-values.

Pearson correlation measures the linear relationship between two continuous variables, assuming normality and sensitivity to outliers. Spearman rank correlation (rho) measures the monotonic relationship between two variables using their ranks, making it non-parametric and robust to outliers and non-linear monotonic relationships. Use Spearman when: data is ordinal, the relationship is monotonic but not linear, or outliers are present. A hypothesis test for rho = 0 uses a t-approximation. Spearman rho = 1 means perfect monotonic relationship. Kendall's tau is an alternative rank correlation with better statistical properties for small samples. Both are interpretable as effect size measures.

from scipy import stats
rho, p_val = stats.spearmanr(x, y)
tau, p_val = stats.kendalltau(x, y)

The one-sample KS test compares an empirical distribution to a specified theoretical distribution (e.g., normal, exponential), measuring the maximum absolute difference between the ECDF and the theoretical CDF. The null hypothesis is that the sample was drawn from the specified distribution. Unlike Shapiro-Wilk (specialized for normality), the KS test works for any theoretical distribution when parameters are known. When parameters are estimated from the data, the Lilliefors correction adjusts critical values (using estimated parameters inflates the p-value). For normality testing specifically, Shapiro-Wilk is more powerful than KS. For testing exponential or uniform fit, KS is appropriate.

from scipy import stats
# Test if data is normally distributed
stat, p_val = stats.kstest(data, 'norm',
                           args=(data.mean(), data.std()))

Testing a difference in proportions (e.g., conversion rate A vs B) requires different formulas than means testing. The null distribution of the z-statistic for proportions uses the pooled proportion estimate. Sample size calculation: n = (z_alpha/2 + z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p1-p2)^2. Effect size for proportions (Cohen's h) is h = 2*arcsin(sqrt(p)) applied to each proportion and differenced. The baseline conversion rate critically determines required sample size: low baseline rates (1%) need much larger samples for the same absolute lift than higher rates (20%), because variance is proportional to p*(1-p) which peaks at p=0.5.

from statsmodels.stats.proportion import proportion_effectsize, zt_ind_solve_power
h = proportion_effectsize(0.05, 0.055)  # baseline vs target rate
n = zt_ind_solve_power(effect_size=h, alpha=0.05, power=0.80)

Circular hypothesis testing (also called double dipping or selection bias) occurs when the same data is used to both identify a hypothesis and test it. For example, exploring data to find the most significant metric, then testing that exact metric as if it were pre-specified. The resulting p-value is invalid because the null distribution assumes the test was pre-specified. The true p-value must account for the selection process — it is much larger than the naive p-value. Solutions: split data into exploration and confirmation sets, use cross-validation, apply corrections for the number of metrics explored, or pre-register before looking at any data.

The exact binomial test computes the exact probability of observing k or more successes in n trials under the null proportion p0, using the binomial CDF. It requires no large-sample approximations and is valid for any N. The z-test for proportions approximates the binomial with a normal distribution (valid when n*p0 >= 5 and n*(1-p0) >= 5). For small samples or extreme proportions, the exact binomial is preferred. scipy.stats.binomtest implements it. The one-proportion z-test can use continuity correction to improve approximation. For two-proportion comparisons, Fisher's exact test provides the exact equivalent.

from scipy import stats
# Exact binomial test: 60 successes in 100, null p=0.5
result = stats.binomtest(k=60, n=100, p=0.5, alternative='two-sided')
print(result.pvalue)