Data labeling and annotation quiz

Question 1

What is the key difference between Cohen's kappa and Fleiss' kappa for measuring inter-annotator agreement?

Accepted Answer

Cohen's handles 2 raters while Fleiss' generalizes to any number of raters

Answer

Cohen's is for continuous labels, Fleiss' is for categorical

Answer

Fleiss' ignores chance agreement while Cohen's corrects for it

Answer

Cohen's requires balanced classes while Fleiss' does not

Question 2

Why is Krippendorff's alpha often preferred over kappa statistics for real-world annotation projects?

Accepted Answer

It supports missing data, any number of raters, and multiple measurement levels

Answer

It always produces higher scores than kappa

Answer

It does not require chance correction

Answer

It only works for two annotators

Question 3

In weak supervision with Snorkel, what does the generative label model learn from multiple labeling functions?

Accepted Answer

The estimated accuracies/correlations of labeling functions to produce probabilistic labels

Answer

The exact ground-truth labels from a held-out set

Answer

A fixed majority vote with equal weights

Answer

The optimal neural architecture for the task

Question 4

What problem does the Dawid-Skene algorithm solve in crowdsourced labeling?

Accepted Answer

It estimates annotator confusion matrices and true labels via EM

Answer

It schedules annotation tasks to minimize cost

Answer

It detects PII in raw text before labeling

Answer

It augments data with synthetic labels

Question 5

How does confident learning (e.g., the Cleanlab approach) identify mislabeled examples?

Accepted Answer

By estimating the joint of noisy and true labels from out-of-sample predicted probabilities

Answer

By retraining until training loss reaches zero

Answer

By comparing annotators with Cohen's kappa

Answer

By removing all low-frequency classes

Question 6

In object detection annotation, how is Intersection over Union (IoU) computed and why does it matter for label quality?

Accepted Answer

Area of overlap divided by area of union of two boxes

Answer

Sum of box areas divided by their difference

Answer

Distance between box centers normalized by diagonal

Answer

Number of matching pixels divided by image size

Question 7

What is the purpose of 'honeypot' (gold) items in a crowdsourcing labeling pipeline?

Accepted Answer

Hidden items with known answers used to monitor annotator accuracy

Answer

Decoy data injected to confuse competitors

Answer

Items reserved exclusively for the test set

Answer

Synthetic samples for data augmentation

Question 8

In active learning, how does uncertainty sampling decide which unlabeled examples to send for annotation?

Accepted Answer

By choosing instances where the model is least confident (e.g., highest entropy)

Answer

By choosing instances the model classifies most confidently

Answer

By selecting a uniform random subset each round

Answer

By selecting the most frequent class examples

Question 9

Why can pure uncertainty sampling in active learning lead to redundant or biased labeled sets, and what mitigates it?

Accepted Answer

It ignores diversity; combine it with density/diversity-aware batch selection

Answer

It always undersamples the majority class; use SMOTE

Answer

It overfits gold items; remove honeypots

Answer

It needs balanced kappa; add more annotators

Question 10

What distinguishes hard labels from soft labels in a training dataset, and when are soft labels useful?

Accepted Answer

Soft labels are probability distributions capturing ambiguity/disagreement

Answer

Soft labels are simply lowercase class names

Answer

Hard labels include confidence scores, soft labels do not

Answer

They are identical except for storage format

Question 11

In RLHF preference data collection, why is pairwise comparison annotation often preferred over absolute scalar ratings?

Accepted Answer

Pairwise judgments are more consistent than calibrated absolute scores

Answer

Pairwise data is cheaper to store than scalar scores

Answer

Absolute ratings cannot train reward models at all

Answer

Pairwise comparison eliminates the need for any guidelines

Question 12

What is model-assisted (pre-)labeling and what is its main quality risk?

Accepted Answer

Pre-annotating with a model; risk is annotators rubber-stamping model errors

Answer

Labeling only synthetic data; risk is overfitting

Answer

Encrypting labels; risk is data loss

Answer

Removing rare classes; risk is imbalance

Question 13

When using an LLM as an annotator (LLM-as-judge or auto-labeling), what is a critical validation step before trusting its labels at scale?

Accepted Answer

Benchmark it against a human gold set for agreement and bias before scaling

Answer

Assume parity with humans since LLMs are large

Answer

Use it only on the test set

Answer

Disable any human review to avoid contamination

Question 14

What is span-level (token-level) annotation in NER, and how is agreement typically measured given boundary disagreements?

Accepted Answer

Marking exact entity boundaries; agreement via span-based F1 due to dominant O class

Answer

Labeling whole documents; agreement via accuracy

Answer

Sentence classification; agreement via RMSE

Answer

Pixel masks; agreement via IoU

Question 15

Why are clear, versioned annotation guidelines critical, and what should be tracked when they change mid-project?

Accepted Answer

They ensure consistency; track versions to map labels to rules and re-annotate affected data

Answer

They are optional once annotators are trained

Answer

They only matter for image tasks

Answer

They should never change once published

Question 16

What is label leakage in the context of dataset construction, and how can the annotation process inadvertently introduce it?

Accepted Answer

Target-correlated info unavailable at inference leaks into the dataset via annotation artifacts

Answer

Annotators leaking data to competitors

Answer

Storing labels in plaintext

Answer

Using soft labels instead of hard labels

Question 17

For an ordinal labeling task (e.g., 1-5 severity), why is plain accuracy or nominal kappa a poor agreement/eval metric?

Accepted Answer

They ignore ordering; use weighted kappa or ordinal Krippendorff's alpha

Answer

They overweight rare classes; use F1

Answer

They require balanced data; use ROC-AUC

Answer

They cannot handle two annotators; use Fleiss'

Question 18

In a multi-label annotation setting, why is per-item majority voting across annotators insufficient, and what aggregation is more appropriate?

Accepted Answer

Per-label thresholded voting; single-class majority discards co-occurring labels

Answer

Always pick the rarest label

Answer

Average the label indices numerically

Answer

Use IoU on the label strings

Question 19

What is the trade-off in choosing the number of redundant annotators per item, and how can it be optimized?

Accepted Answer

Quality vs cost; allocate extra annotators adaptively to disputed/hard items

Answer

Always use exactly three annotators regardless of item

Answer

More annotators always reduce quality

Answer

Redundancy only affects storage cost

Question 20

How does class imbalance affect annotation strategy, and what sampling approach helps capture rare classes efficiently?

Accepted Answer

Use stratified/model-guided mining to surface rare-class candidates

Answer

Label only the majority class to save budget

Answer

Duplicate majority examples before labeling

Answer

Ignore imbalance; random sampling suffices

Question 21

What is the difference between semantic segmentation and instance segmentation annotation?

Accepted Answer

Instance segmentation separates individual objects; semantic only labels pixel classes

Answer

They are identical

Answer

Semantic uses boxes, instance uses polygons only

Answer

Instance segmentation ignores class labels

Question 22

What is the MACE (Multi-Annotator Competence Estimation) model used for in annotation aggregation?

Accepted Answer

Estimating annotator competence and true labels while detecting spammers

Answer

Generating synthetic annotations

Answer

Scheduling annotators by timezone

Answer

Encrypting annotation payloads

Question 23

How should personally identifiable information (PII) be handled in a text-labeling pipeline under regulations like GDPR?

Accepted Answer

Redact/pseudonymize PII before annotation and control access with audit logs

Answer

Share full PII so annotators have context

Answer

PII only matters after model deployment

Answer

Store PII unencrypted for convenience

Question 24

What is annotator bias and how can demographic composition of an annotation pool affect a model?

Accepted Answer

Systematic subjective-label skew from annotator backgrounds; mitigate with diverse pools

Answer

A statistical artifact of large batch sizes

Answer

Bias only in image bounding boxes

Answer

An error fully removed by majority voting

Question 25

In data programming, what are labeling functions and what trade-off do they exhibit between coverage and accuracy?

Accepted Answer

Noisy heuristic labelers trading coverage against accuracy, combined by a label model

Answer

Manual per-item labels with perfect accuracy

Answer

Loss functions used in training

Answer

Functions that delete mislabeled data

Question 26

Why is a held-out, expertly-labeled gold/benchmark set important even when most labels come from crowds or weak supervision?

Accepted Answer

It gives an unbiased benchmark to validate noisy labels and detect drift

Answer

It is used to train the largest model

Answer

It replaces the need for guidelines

Answer

It should be merged into the training noisy labels

Question 27

What is the consensus vs. expert-adjudication trade-off for resolving annotation disagreements?

Accepted Answer

Consensus is cheap but can entrench shared errors; adjudication is accurate but costly

Answer

Adjudication is always cheaper than consensus

Answer

Consensus guarantees correctness on hard items

Answer

They produce identical labels by definition

Question 28

When auto-labeling with a model and then training on those labels, what failure mode resembles 'model collapse' or confirmation feedback?

Accepted Answer

Self-training on own errors amplifies bias; inject human labels to break the loop

Answer

Overfitting cured by more epochs

Answer

Class imbalance fixed by oversampling

Answer

Leakage from the test set

Question 29

What is the rationale for measuring and acting on annotator throughput and edit-rate metrics, not just accuracy?

Accepted Answer

They reveal rushing, rubber-stamping, or guideline issues beyond spot-checked accuracy

Answer

They only measure payroll cost

Answer

They replace honeypots entirely

Answer

Throughput is irrelevant to label quality

Question 30

In hierarchical/taxonomic labeling, what advantage does enforcing the label hierarchy give during annotation and aggregation?

Accepted Answer

Annotators can back off to parent nodes and aggregation uses hierarchy-aware distances

Answer

It forces every item to the leaf level always

Answer

It eliminates the need for agreement metrics

Answer

Hierarchy only matters for image masks

Test yourself