Hypothesis Testing quiz

Question 1

What is a p-value and what does it actually mean?

Accepted Answer

The p-value is the probability of observing results at least as extreme if the null is true

Answer

The p-value is the probability that the null hypothesis is true

Answer

The p-value is the probability the result occurred by chance

Answer

A p-value below 0.05 proves the alternative hypothesis

Question 2

What is the difference between Type I and Type II errors?

Accepted Answer

Type I error (false positive) rejects a true null hypothesis

Answer

Type I error is failing to reject a false null hypothesis

Answer

Type II error is rejecting a true null hypothesis

Answer

Reducing alpha always increases power

Question 3

What is Cohen's d and how is it interpreted?

Accepted Answer

Cohen's d is computed as the difference in means divided by the pooled standard deviation

Answer

A Cohen's d of 0.2 is considered large

Answer

Cohen's d increases with larger sample sizes for the same true effect

Answer

A Cohen's d of 0 means the groups have identical standard deviations

Question 4

What is the Bonferroni correction and when is it too conservative?

Accepted Answer

Bonferroni adjusts the significance threshold to alpha divided by m

Answer

Bonferroni correction controls the false discovery rate

Answer

Bonferroni sets the threshold to alpha multiplied by m

Answer

Bonferroni is the most powerful correction for correlated tests

Question 5

What is a one-tailed versus two-tailed test and when do you choose each?

Accepted Answer

Two-tailed tests are the standard default for detecting effects in either direction

Answer

One-tailed tests are always more powerful than two-tailed tests

Answer

Two-tailed tests detect effects only in the direction specified by the researcher

Answer

One-tailed tests split the rejection region across both tails

Question 6

What is the Shapiro-Wilk test and when should you use it?

Accepted Answer

Shapiro-Wilk tests whether a sample is drawn from a normal distribution

Answer

The null hypothesis of Shapiro-Wilk is that data is non-normal

Answer

Shapiro-Wilk is the best normality test for very large samples

Answer

A non-significant Shapiro-Wilk test proves normality

Question 7

What is one-way ANOVA and what are its assumptions?

Accepted Answer

One-way ANOVA tests whether at least one group mean differs from the others

Answer

One-way ANOVA requires at least 5 groups to be valid

Answer

A significant F-test identifies which specific group pairs differ

Answer

ANOVA requires that sample sizes be equal across all groups

Question 8

What is Fisher's exact test and when do you use it instead of chi-squared?

Accepted Answer

Fisher's exact test computes exact probabilities without large-sample approximations

Answer

Fisher's exact test is only valid for large sample sizes

Answer

Fisher's exact test uses a normal approximation to compute p-values

Answer

Fisher's exact test is less accurate than chi-squared for 2x2 tables

Question 9

What is sequential hypothesis testing and why is it needed in A/B testing?

Accepted Answer

Sequential testing allows interim analyses without inflating Type I error

Answer

Optional stopping (peeking) does not affect Type I error rates

Answer

SPRT requires fixing the sample size before the experiment begins

Answer

Alpha-spending functions increase the significance threshold at each look

Question 10

What is the difference between statistical significance and practical significance?

Accepted Answer

Statistical significance and practical significance are independent concepts

Answer

A statistically significant result always has practical business importance

Answer

A practically significant result always achieves statistical significance

Answer

Effect size measures are irrelevant when p < 0.05

Question 11

What is the false discovery rate and how does it differ from false positive rate?

Accepted Answer

FDR is the expected proportion of rejected nulls that are false positives

Answer

FDR is the proportion of all true nulls that are incorrectly rejected

Answer

FDR and FPR are identical concepts

Answer

FDR is always larger than FPR for the same dataset

Question 12

What is a confidence interval and how should it be interpreted?

Accepted Answer

A 95% CI means if the study were repeated, 95% of computed CIs would contain the true parameter

Answer

A 95% CI means the true parameter has a 95% probability of being in the interval

Answer

A 95% CI means 95% of sample observations fall within the interval

Answer

Confidence intervals are equivalent to Bayesian credible intervals

Question 13

What is the issue with data peeking in hypothesis tests?

Accepted Answer

Optional stopping inflates Type I error toward 1 with unlimited looks

Answer

Peeking at results does not affect p-value validity

Answer

Stopping early when p < 0.05 is equivalent to applying a Bonferroni correction

Answer

Sequential testing methods eliminate the need for pre-registration

Question 14

What is eta-squared and partial eta-squared in ANOVA?

Accepted Answer

Partial eta-squared excludes variance from other factors in its denominator

Answer

Eta-squared equals 1 minus the F-statistic

Answer

Eta-squared and partial eta-squared are always identical in one-way ANOVA

Answer

A partial eta-squared of 0.14 indicates a small effect

Question 15

What is pre-registration in hypothesis testing and why does it matter?

Accepted Answer

Pre-registration commits to analysis plan before data collection, preventing p-hacking

Answer

Pre-registration prevents researchers from analyzing data before committing to hypotheses

Answer

Pre-registration is only required for medical studies

Answer

Pre-registration guarantees statistical significance of results

Question 16

What is the role of randomization in hypothesis testing validity?

Accepted Answer

Randomization ensures confounders are balanced in expectation across groups

Answer

Randomization only controls for observed confounders

Answer

Randomization guarantees equal group sizes in A/B tests

Answer

Stratified randomization is only needed when sample sizes are very large

Question 17

What is p-hacking and how does it invalidate results?

Accepted Answer

P-hacking exploits multiple testing to artificially achieve p < 0.05

Answer

P-hacking increases statistical power

Answer

P-hacking is only possible when using parametric tests

Answer

Pre-registration prevents all forms of p-hacking

Question 18

What is the Spearman rank correlation and when do you use it instead of Pearson?

Accepted Answer

Spearman uses ranks and is robust to outliers and non-linear monotonic relationships

Answer

Pearson correlation is always preferred over Spearman

Answer

Spearman correlation detects linear relationships; Pearson detects monotonic relationships

Answer

Spearman correlation requires normally distributed variables

Question 19

What is an exact binomial test and how does it differ from a z-test for proportions?

Accepted Answer

The exact binomial test computes exact p-values valid for any sample size

Answer

The exact binomial test requires large-sample normal approximation

Answer

The z-test for proportions is always more accurate than the exact binomial test

Answer

Both tests produce identical p-values for large samples

Test yourself