Experiment tracking (MLflow, W&B) quiz

Question 1

What is the difference between an MLflow Run and an MLflow Experiment?

Accepted Answer

An Experiment groups related Runs; a Run is one execution recording params/metrics/artifacts

Answer

A Run groups multiple Experiments for comparison

Answer

They are synonyms for the same tracking object

Answer

An Experiment is a deployed model and a Run is its inference call

Question 2

In MLflow, what is the distinction between log_param and log_metric, and why does it matter?

Accepted Answer

Params are static config (string, immutable); metrics are numeric, can have steps/time series

Answer

Params are numeric time series; metrics are static strings

Answer

Both are identical and interchangeable

Answer

Metrics can only be logged once while params support steps

Question 3

What does mlflow.autolog() do, and what is a common pitfall when using it?

Accepted Answer

It auto-captures params/metrics/model for supported libs but can conflict with manual logging or silently miss unsupported versions

Answer

It auto-deploys the best model to production

Answer

It only works after training completes

Answer

It encrypts all logged artifacts automatically

Question 4

What is the MLflow Model Registry and what are its core stages?

Accepted Answer

A centralized versioned model store with stages (None/Staging/Production/Archived) and lineage to runs

Answer

A feature store for training data versioning

Answer

A container registry for Docker images only

Answer

A log aggregator for metrics across experiments

Question 5

In MLflow, why are model 'aliases' increasingly preferred over the legacy 'stages' for promotion?

Accepted Answer

Aliases are mutable named pointers to any version enabling champion/challenger; stages are a rigid single-version-per-stage enum

Answer

Aliases are encrypted while stages are plaintext

Answer

Stages support more versions per stage than aliases

Answer

There is no functional difference; aliases are just renamed stages

Question 6

What is an MLflow Model 'flavor' and why is the pyfunc flavor important?

Accepted Answer

A flavor is a loading convention; pyfunc is a universal predict() interface across frameworks

Answer

A flavor is a hyperparameter group; pyfunc is the optimizer

Answer

A flavor is a UI theme for the tracking server

Answer

A flavor is a metric type and pyfunc is a chart type

Question 7

What is an MLflow model signature, and what problem does it solve at serving time?

Accepted Answer

It declares input/output schema for validation and type coercion at inference

Answer

It signs the model cryptographically for integrity

Answer

It records who trained the model for audit

Answer

It defines the REST endpoint URL

Question 8

What are the four backend components of an MLflow Tracking deployment?

Accepted Answer

Tracking server, backend metadata store (DB), artifact store (object storage), client API

Answer

Only a single SQLite file holds everything

Answer

A Kafka queue, Redis cache, GPU cluster, and UI

Answer

Frontend, GraphQL layer, vector DB, and feature store

Question 9

For a production MLflow server, why should the backend store and artifact store be configured separately?

Accepted Answer

Because metadata needs fast DB queries while large artifacts belong in scalable object storage, enabling direct uploads

Answer

Because the law requires encryption separation

Answer

Because artifacts cannot be versioned otherwise

Answer

There is no reason; a single store is always better

Question 10

In W&B, what is the relationship between a Project, a Run, and a Sweep?

Accepted Answer

A Project groups Runs; a Sweep orchestrates many Runs via a search strategy

Answer

A Sweep contains Projects which contain Runs

Answer

Run and Sweep are the same; Project is unrelated

Answer

A Project is one Run and a Sweep is one metric

Question 11

What W&B search strategies are available for hyperparameter sweeps, and when would you use Bayesian over grid/random?

Accepted Answer

grid, random, bayes; use Bayesian when runs are expensive to focus on promising regions

Answer

Only grid search is supported

Answer

Only gradient-based search

Answer

random and genetic only; Bayesian is unsupported

Question 12

What is the W&B early-termination feature 'Hyperband' and what does it accomplish in a sweep?

Accepted Answer

It terminates underperforming runs early at milestones to reallocate compute

Answer

It speeds up data loading

Answer

It compresses logged artifacts

Answer

It schedules runs across time zones

Question 13

What are W&B Artifacts and how do they support reproducibility and lineage?

Accepted Answer

Artifacts are versioned, content-hashed objects that build a data-to-model lineage DAG with immutable versions and aliases

Answer

Artifacts are temporary logs deleted after each run

Answer

Artifacts are only chart images

Answer

Artifacts store hyperparameters, not files

Question 14

In distributed (multi-GPU) training, how should you log to W&B to avoid duplicate or conflicting runs?

Accepted Answer

Log from rank-0 only, or use grouped runs per process; avoid concurrent writes to one run

Answer

Always log from every process to the same run

Answer

Disable logging entirely in distributed mode

Answer

Log only after all processes finish, from a separate machine

Question 15

What is the purpose of the 'step' argument in metric logging, and what happens if you log out-of-order steps?

Accepted Answer

step sets the x-axis; must be non-decreasing per run in W&B or points are dropped

Answer

step is just a label with no ordering rules

Answer

step deletes earlier metrics

Answer

step controls the learning rate

Question 16

How does MLflow support nested runs, and when are they useful?

Accepted Answer

mlflow.start_run(nested=True) creates child runs ideal for HPO trials/CV folds under a parent

Answer

Nested runs are forbidden in MLflow

Answer

Nested runs require a separate experiment per child

Answer

Nesting automatically averages child metrics into the parent

Question 17

What environment variable or API sets the MLflow tracking destination, and what are valid targets?

Accepted Answer

MLFLOW_TRACKING_URI / set_tracking_uri; targets: local dir, DB URI, http server, databricks

Answer

MLFLOW_HOME only, pointing to a Docker image

Answer

WANDB_API_KEY sets it

Answer

Only an S3 bucket path is valid

Question 18

How do you ensure a failed or crashed MLflow run is marked correctly rather than left 'RUNNING'?

Accepted Answer

Use the with-statement (or end_run with status) so it marks FINISHED/FAILED; killed processes leave RUNNING

Answer

MLflow auto-detects crashes via heartbeat always

Answer

Failed runs are auto-deleted

Answer

Status cannot be set programmatically

Question 19

What is the recommended way to log evaluation tables/plots (e.g., confusion matrices, sample predictions) in W&B?

Accepted Answer

Use wandb.Table and wandb.plot/Image for interactive, queryable, rich-media logging

Answer

Only log raw PNG files

Answer

Tables are not supported; use CSV download

Answer

Encode everything as a single metric value

Question 20

What is the difference between resuming a run with WANDB_RESUME='must' versus 'allow' versus 'never'?

Accepted Answer

must=requires existing id; allow=resume-or-create; never=always new and errors if id exists

Answer

They all create a new run regardless

Answer

must=never resumes; allow=always errors

Answer

They control sweep methods, not resuming

Question 21

How can you compare hundreds of runs efficiently in experiment-tracking UIs?

Accepted Answer

Use filtered/sorted runs tables, grouping, and parallel-coordinates plots; query programmatically

Answer

Open each run individually and read logs

Answer

Export all to a spreadsheet manually each time

Answer

Comparison across runs is not possible

Question 22

What is mlflow.evaluate and what does it provide for model validation?

Accepted Answer

It computes task metrics + diagnostics (confusion matrix, SHAP) and can gate via validation thresholds

Answer

It trains the model from scratch

Answer

It only logs the model file

Answer

It deploys the model to an endpoint

Question 23

Why is logging the full environment (e.g., conda.yaml / requirements.txt / Docker image) important in experiment tracking, and how does MLflow handle it?

Accepted Answer

MLflow saves conda.yaml/python_env.yaml/requirements.txt with the model to recreate the runtime reproducibly

Answer

Environment capture is unnecessary if code is committed

Answer

It only stores the Python version

Answer

It bundles the entire OS image by default

Question 24

What is the difference between MLflow's git commit / source tracking and W&B's code saving for reproducibility?

Accepted Answer

MLflow tags the git commit/source; W&B can save actual code plus the git diff/patch

Answer

Neither tracks code

Answer

Both store the full OS snapshot

Answer

Only W&B records anything about code

Question 25

For LLM/GenAI applications, what experiment-tracking capabilities do MLflow Tracing and W&B Weave provide?

Accepted Answer

They capture nested LLM/agent traces (prompts, tool calls, tokens, cost) and support LLM-judge evaluation

Answer

They only track training loss

Answer

They are unrelated to GenAI

Answer

They replace the LLM provider's API

Question 26

What is a robust pattern for tracking the dataset version used in an experiment?

Accepted Answer

Log a dataset digest/version + source via mlflow.data.log_input or W&B versioned Artifacts

Answer

Always paste raw data into params

Answer

Store only the row count

Answer

Dataset tracking is impossible in these tools

Question 27

What is the W&B Model Registry (Registered Models / linked artifacts) and how does it relate to artifact lineage?

Accepted Answer

A registered collection of model artifact versions with aliases and full lineage back to run/data/code

Answer

A place to store only metrics

Answer

A billing dashboard

Answer

A replacement for the sweep controller

Question 28

How does MLflow Projects (MLproject file) contribute to reproducible experiment execution?

Accepted Answer

It declares entry points/params/environment so 'mlflow run' executes code reproducibly from a git URI

Answer

It is a billing config file

Answer

It stores metrics

Answer

It is only for model serving

Question 29

When logging metrics at high frequency (e.g., every training step), what tracking-overhead concern arises and how do you mitigate it?

Accepted Answer

Batch/reduce logging frequency and use async/background uploads to limit IO and storage overhead

Answer

High-frequency logging has zero cost

Answer

Disable all metrics and rely on print statements

Answer

Increase logging frequency to fix it

Question 30

What is the role of tags versus params in experiment tracking, and give an example of an appropriate tag.

Accepted Answer

Params are fixed config inputs; tags are mutable organizational metadata (e.g., team, ticket) addable after the run

Answer

Tags and params are identical

Answer

Tags must be numeric only

Answer

Params can be changed anytime but tags cannot

Test yourself