Testing and validating ML systems quiz

Question 1

What is data leakage in the context of ML model validation, and why does it inflate offline metrics?

Accepted Answer

Using future or target-derived information unavailable at prediction time, inflating offline metrics

Answer

Training on a dataset larger than the test set

Answer

Using a deep model when a linear one suffices

Answer

Storing model weights in an unencrypted file

Question 2

Why must preprocessing transforms like StandardScaler be fit only on the training fold inside cross-validation?

Accepted Answer

To prevent statistics from validation data leaking into training

Answer

To make training faster

Answer

Because scalers cannot handle the full dataset in memory

Answer

To increase model variance deliberately

Question 3

When should you use TimeSeriesSplit (forward-chaining) instead of standard k-fold cross-validation?

Accepted Answer

Whenever data is temporally ordered and you predict the future from the past

Answer

Whenever the dataset is large

Answer

Only for classification, never regression

Answer

When classes are imbalanced

Question 4

What is nested cross-validation and what bias does it correct?

Accepted Answer

It gives an unbiased estimate by separating tuning (inner) from evaluation (outer)

Answer

It removes the need for a test set

Answer

It always increases accuracy

Answer

It is identical to repeated k-fold

Question 5

Why is stratified k-fold preferred over plain k-fold for imbalanced classification?

Accepted Answer

It preserves class proportions per fold for stable estimates

Answer

It trains faster on imbalanced data

Answer

It oversamples the minority class automatically

Answer

It guarantees higher recall

Question 6

What is group k-fold (GroupKFold) and when is it required to avoid leakage?

Accepted Answer

When correlated samples share a group that must not span train and test

Answer

When you want balanced classes

Answer

Only for image data

Answer

When the dataset is small

Question 7

In ML system testing, what distinguishes an invariance test from a directional expectation test (as in CheckList)?

Accepted Answer

Invariance expects no change; directional expects a known-direction change

Answer

They are synonyms

Answer

Invariance tests latency; directional tests memory

Answer

Both require labeled test data only

Question 8

What is metamorphic testing and why is it useful when ground-truth labels are unavailable?

Accepted Answer

It checks relations between related inputs/outputs without needing per-input labels

Answer

It requires fully labeled data

Answer

It is only for computer vision

Answer

It measures training loss

Question 9

What is train-serve skew and how do you detect it?

Accepted Answer

Mismatch between training and serving feature logic/distributions

Answer

Difference in GPU usage between train and serve

Answer

A type of overfitting to the validation set

Answer

The gap between train and test accuracy

Question 10

Distinguish data drift from concept drift in production ML monitoring.

Accepted Answer

Data drift changes P(X); concept drift changes P(y|X)

Answer

Data drift changes P(y|X); concept drift changes P(X)

Answer

They are identical

Answer

Both only occur during training

Question 11

Why is accuracy a misleading offline validation metric for highly imbalanced problems, and what should you use instead?

Accepted Answer

Because a majority predictor scores high while ignoring the minority class

Answer

Because accuracy needs probabilities

Answer

Because accuracy cannot be computed for >2 classes

Answer

Because accuracy double-counts true negatives

Question 12

What does it mean for a classifier to be well-calibrated, and how do you test calibration?

Accepted Answer

When predicted probabilities match observed outcome frequencies

Answer

When accuracy exceeds 90%

Answer

When the ROC AUC equals 1

Answer

When training and test loss are equal

Question 13

What is slice-based (subgroup) evaluation and why is overall accuracy insufficient?

Accepted Answer

Reporting metrics per meaningful subgroup to catch hidden failures

Answer

Evaluating on the entire dataset only

Answer

Slicing the model into smaller networks

Answer

Reducing the test set size

Question 14

In an online A/B test for a recommender, what is a guardrail metric and why include one?

Accepted Answer

A secondary metric that must not regress while optimizing the primary

Answer

The metric you most want to improve

Answer

A statistical correction for multiple testing

Answer

The minimum sample size needed

Question 15

What is the purpose of a power analysis before running an ML A/B experiment?

Accepted Answer

To compute the sample size needed to detect a target effect reliably

Answer

To choose the learning rate

Answer

To measure GPU power consumption

Answer

To pick the random seed

Question 16

Why can repeatedly evaluating many models on the same validation set cause overfitting to it, and how do you mitigate this?

Accepted Answer

Repeated decisions adapt to validation noise; use a once-only test set or nested CV

Answer

It cannot happen with a fixed validation set

Answer

It only happens with deep learning

Answer

It is solved by increasing batch size

Question 17

How do you build a confidence interval around a model's test metric without distributional assumptions?

Accepted Answer

Bootstrap-resample the test set and take metric percentiles

Answer

Use the training loss curve

Answer

Assume a normal distribution and use the train accuracy

Answer

Run more epochs

Question 18

What is a golden / regression test set in ML CI, and how does it differ from the held-out test set used for model selection?

Accepted Answer

A curated CI set of high-value cases with expected outputs to catch regressions

Answer

It is the same as the held-out test set

Answer

A set used only to train the final model

Answer

A synthetic dataset for data augmentation

Question 19

What is test-set contamination for LLM benchmarks and how do you check for it?

Accepted Answer

When benchmark data leaked into pretraining, inflating scores via memorization

Answer

When the test set is too small

Answer

When the model is quantized

Answer

When prompts are too long

Question 20

What are the main risks of using an LLM-as-a-judge for evaluation, and how do you mitigate them?

Accepted Answer

Position, verbosity and self-preference biases; mitigate with order swaps and human calibration

Answer

LLM judges are unbiased and need no validation

Answer

They only fail on math problems

Answer

They require GPUs to run, which is the main risk

Question 21

Why is offline evaluation often insufficient for ranking/recommendation systems, requiring online tests?

Accepted Answer

Logged data has feedback/presentation bias and cannot show user reaction to new items

Answer

Offline metrics are always wrong

Answer

Online tests are cheaper than offline

Answer

Offline evaluation requires no data

Question 22

What is the difference between shadow deployment and canary deployment when validating a new model?

Accepted Answer

Shadow logs predictions without serving them; canary serves a small live fraction

Answer

They are identical strategies

Answer

Canary never sees real traffic

Answer

Shadow always serves all users

Question 23

What is a data validation schema check (e.g., TFDV / Great Expectations) and what failures does it catch before training?

Accepted Answer

They assert feature types/ranges/categories to catch bad data early

Answer

They tune hyperparameters

Answer

They compress the dataset

Answer

They replace cross-validation

Question 24

What is the ML test pyramid analogy and how do its layers map to model systems?

Accepted Answer

Many fast unit tests at the base, few costly end-to-end/online tests at the top

Answer

All ML tests should be end-to-end

Answer

Only online A/B tests matter

Answer

The pyramid is inverted for ML (mostly e2e)

Question 25

How do you write a meaningful unit test for a model itself (as opposed to data), per the ML Test Score rubric?

Accepted Answer

Assert invariants like loss-decreases-on-step and ability to overfit a tiny batch

Answer

Only assert the test accuracy is above 90%

Answer

Check that the file size is small

Answer

Verify the GPU temperature

Question 26

What statistical pitfall does the Bonferroni (or Benjamini-Hochberg) correction address in ML experimentation?

Accepted Answer

Inflated false positives from many simultaneous tests

Answer

Overfitting to the training set

Answer

Slow model inference

Answer

Class imbalance

Question 27

Why is 'peeking' / continuous monitoring of a fixed-horizon A/B test invalid, and what fixes it?

Accepted Answer

Repeated looks inflate false positives; use fixed horizon or sequential tests

Answer

Peeking is fine if the sample is large

Answer

It only matters for Bayesian tests

Answer

It makes the test underpowered, not biased

Question 28

What is backtesting and what is look-ahead bias when validating a time-dependent (e.g., trading or forecasting) model?

Accepted Answer

Simulating decisions through time; look-ahead bias is using not-yet-available info

Answer

Backtesting trains on the future; look-ahead bias is harmless

Answer

Look-ahead bias means using too little data

Answer

Backtesting requires shuffling data

Question 29

How does adversarial / robustness testing differ from standard accuracy evaluation, and what does it reveal?

Accepted Answer

It probes worst-case failures under crafted perturbations, exposing brittleness

Answer

It measures average accuracy faster

Answer

It only applies to tabular data

Answer

It is the same as cross-validation

Question 30

What is mutation testing applied to ML pipelines, and what does it assess?

Accepted Answer

It injects faults to measure whether the test suite catches them

Answer

It mutates the model weights to improve accuracy

Answer

It generates adversarial examples

Answer

It augments the training data

Test yourself