Data labeling and annotation

Cohen's kappa measures agreement between exactly two raters on categorical labels, correcting for chance agreement. Fleiss' kappa generalizes this to any fixed number of raters (and does not require the same raters for every item), making it suitable for crowd settings where each item may be labeled by a different subset of annotators. Both yield 1 for perfect agreement and ~0 for chance-level agreement.

Krippendorff's alpha handles any number of annotators, missing data (incomplete annotations), and multiple measurement levels (nominal, ordinal, interval, ratio) within one unified framework. Kappa variants assume complete data and are restricted to nominal categories, so they break down when annotators skip items or labels are ordinal. This flexibility makes alpha the de facto standard for heterogeneous crowdsourced datasets.

The label model estimates the accuracies and correlations of the noisy, conflicting labeling functions without access to ground truth, then combines their votes into probabilistic (soft) training labels. It exploits agreements and disagreements among labeling functions to infer their reliability, similar to a Dawid-Skene-style latent-variable model. These probabilistic labels then train a downstream (discriminative) model that generalizes beyond the labeling functions' coverage.

Dawid-Skene jointly estimates each annotator's confusion matrix (their reliability per class) and the true latent label of each item using expectation-maximization, given only the observed noisy labels. This is more robust than simple majority voting because it down-weights unreliable annotators and accounts for systematic per-class error patterns. It is a foundational model for aggregating redundant crowd annotations into consensus labels.

Confident learning estimates the joint distribution between noisy given labels and latent true labels using out-of-sample predicted probabilities and per-class confidence thresholds. Examples whose model confidence for a different class consistently exceeds the threshold are flagged as label errors. It is model-agnostic and provides theoretical guarantees for noisy-label identification without needing clean ground truth.

IoU is the area of overlap between a predicted/annotated box and the reference box divided by the area of their union. It quantifies bounding-box localization accuracy; annotation QA often requires annotator boxes to reach an IoU threshold (e.g., 0.7-0.9) against gold boxes. Loose IoU tolerance introduces localization noise that degrades downstream detector mAP.

def iou(a, b):
    x1,y1 = max(a[0],b[0]), max(a[1],b[1])
    x2,y2 = min(a[2],b[2]), min(a[3],b[3])
    inter = max(0,x2-x1)*max(0,y2-y1)
    area = (a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter
    return inter/area if area else 0.0

Honeypots are tasks with known correct answers secretly mixed into the annotation stream to continuously measure each worker's accuracy. Workers falling below a quality threshold can be down-weighted, retrained, or removed, and their prior work re-reviewed. They provide ongoing quality control without requiring full re-annotation of every item.

Uncertainty sampling selects the unlabeled instances on which the current model is least confident, commonly measured by least-confidence, margin (gap between top two class probabilities), or entropy of the predictive distribution. Labeling these maximally informative points tends to reduce model error faster per label than random sampling. It targets the decision boundary where labels are most valuable.

Uncertainty sampling can repeatedly pick near-duplicate uncertain points clustered in one region, ignoring data diversity and over-representing certain modes. Diversity- or density-aware strategies (e.g., core-set selection, BADGE, clustering-based batch selection) combine informativeness with representativeness to avoid redundant batches. This is especially important when labeling in batches rather than one item at a time.

Hard labels assign a single discrete class per example, while soft labels assign a probability distribution across classes (e.g., 0.7 cat, 0.3 dog). Soft labels capture genuine annotator disagreement, inherent ambiguity, or teacher-model knowledge (as in distillation), and can improve calibration and robustness. They are valuable when items are legitimately ambiguous rather than forcing a possibly incorrect single label.

Humans are more consistent and reliable at judging which of two responses is better than at assigning calibrated absolute scores, which suffer from scale drift and inter-annotator variance. Pairwise preferences feed directly into a Bradley-Terry/reward-model objective and reduce noise from differing personal rating scales. This yields more learnable, lower-variance signal for training reward models.

Model-assisted labeling uses a model to generate candidate labels (pre-annotations) that human annotators then verify or correct, dramatically increasing throughput. The main risk is automation/confirmation bias: annotators tend to accept plausible-looking model suggestions, propagating systematic model errors into the 'ground truth'. Mitigations include hiding low-confidence predictions, blind re-labeling of samples, and tracking accept/edit rates.

Validate the LLM annotator against a human-labeled gold set to measure agreement, per-class accuracy, and systematic biases (e.g., position bias, verbosity bias, self-preference). Only after confirming acceptable alignment and characterizing failure modes should it be scaled, ideally with human review on low-confidence or high-stakes items. Calibrating prompts and adding self-consistency or ensembling further reduces noise.

Span-level NER annotation marks the exact token boundaries and type of each entity mention within text. Because annotators may disagree on exact boundaries, agreement is usually measured with span-based F1 (exact or partial/overlap match) rather than per-token kappa, since the negative 'O' class dominates and inflates chance-corrected scores. Partial-match and type-only metrics help diagnose whether disagreement is about boundaries or categories.

Guidelines define the label ontology and edge-case rules, ensuring consistency across annotators and over time; ambiguity is the leading cause of low inter-annotator agreement. Guideline versions must be tracked so that labels can be tied to the rule version that produced them, enabling re-annotation of affected items and preventing silent label drift. Without versioning, mixing pre/post-change labels corrupts the dataset.

Label leakage occurs when information correlated with the target (but unavailable at inference time) becomes embedded in features, inflating offline metrics. Annotation can cause it when annotators see or use proxy artifacts (e.g., a post-event field, filename patterns, or model pre-labels derived from future data) that encode the answer. Auditing what signals annotators have access to and separating train/test by time or source helps prevent it.

Nominal metrics treat all disagreements equally, so a 1-vs-2 disagreement counts the same as 1-vs-5, ignoring the ordering. Weighted kappa (linear or quadratic weights) or Krippendorff's alpha with an ordinal/interval distance penalizes larger ordinal gaps more, reflecting that adjacent-rank disagreements are less severe. This gives a truer picture of annotation reliability for ranked scales.

In multi-label tasks each item can carry several labels, so collapsing to a single majority class loses co-occurring labels and treats labels as mutually exclusive. Aggregation should be per-label (e.g., threshold each label by the fraction of annotators selecting it) or use multi-label-aware models, and agreement measured with Jaccard/F1 per label or example-based metrics. This preserves the set structure of the annotations.

More annotators per item reduce label variance and improve consensus quality but increase cost and slow throughput, with diminishing returns. Adaptive redundancy assigns extra annotators only to items showing disagreement or high difficulty (e.g., via Dawid-Skene confidence or model uncertainty), concentrating budget where it matters. This dynamic allocation beats fixed uniform redundancy on cost-quality efficiency.

With severe imbalance, random sampling yields very few rare-class examples, wasting annotation budget on the majority. Strategies like stratified sampling, model-guided mining (using a weak classifier to surface likely rare-class candidates), or active learning targeted at minority regions increase rare-class yield. This ensures the labeled set has enough positives to train and evaluate reliably.

Semantic segmentation assigns a class label to every pixel without distinguishing separate objects of the same class, so two adjacent cars share one 'car' mask. Instance segmentation additionally separates individual object instances, giving each car its own mask and id. Panoptic segmentation unifies both, labeling every pixel with a class and, for countable 'things', an instance id.

MACE is a Bayesian/EM item-response model that infers the most likely true label per item while estimating each annotator's competence (trustworthiness) and detecting spammers who label randomly or with fixed strategies. By modeling annotator behavior with latent variables, it outperforms majority voting especially when some annotators are adversarial or careless. It outputs both consensus labels and per-annotator reliability scores.

PII should be detected and redacted or pseudonymized before raw data reaches annotators, with access controlled on a need-to-know basis and data-processing agreements in place. Annotation platforms should log access, support data-subject deletion requests, and minimize the data retained to what the task requires. Failing to redact exposes the organization to legal liability and the annotators to sensitive data unnecessarily.

Annotator bias arises when annotators' backgrounds, cultures, or beliefs systematically shape subjective labels (e.g., toxicity, sentiment, offensiveness), embedding those perspectives into the dataset and resulting model. A non-representative pool can encode majority-group norms and harm under-represented users. Mitigations include diverse recruitment, collecting annotator metadata, modeling disagreement explicitly, and reporting per-group label distributions.

Labeling functions are heuristics, rules, patterns, or model outputs that programmatically emit noisy labels (or abstain) for unlabeled data. Broad functions achieve high coverage but lower accuracy, while narrow precise functions are accurate but cover few examples; the label model combines many to balance the two. The goal is high aggregate coverage with manageable, estimable noise rather than perfection per function.

def lf_contains_refund(x):
    return SPAM if 'free money' in x.text.lower() else ABSTAIN

A small, high-quality gold set provides an unbiased yardstick to evaluate aggregation quality, calibrate labeling functions, validate LLM annotators, and detect drift, without contaminating it with the same noisy process. Because the bulk labels are noisy, model evaluation on them would be unreliable. The gold set must be kept separate from training and re-checked periodically.

Consensus (majority/aggregation) is cheap and scalable but can entrench shared annotator misconceptions and fail on genuinely hard items. Expert adjudication routes disagreements to a senior reviewer for authoritative resolution, improving accuracy on hard cases at higher cost and lower throughput. A hybrid escalates only high-disagreement or high-stakes items to experts, reserving consensus for the rest.

If a model labels data, is trained on its own (errorful) labels, and the cycle repeats, its systematic biases are amplified and reinforced, narrowing the learned distribution and degrading on rare or hard cases. This self-reinforcing loop (a form of confirmation bias / model collapse) is mitigated by injecting human-verified labels, keeping fresh real data, and monitoring distribution shift. Diversity of supervision sources breaks the loop.

Throughput and edit/accept rates surface operational issues: unusually fast annotators may be rushing (low quality), high edit rates on pre-labels indicate poor model assistance or hard data, and low edit rates can signal rubber-stamping. Combined with honeypot accuracy, these signals identify training needs, fraud, or guideline ambiguity early. Quality alone, sampled sparsely, misses these process dynamics.

Enforcing the taxonomy lets annotators label at the most specific level they are confident in and roll up uncertain cases to a parent node, reducing forced errors. Aggregation can then exploit hierarchy-aware distances (a sibling confusion costs less than a cross-branch one) and partial credit, improving consensus and evaluation fidelity. It also keeps labels internally consistent (a child implies its ancestors).

Inter-annotator agreement measures consistency across different annotators, while intra-annotator (test-retest) agreement measures whether the same annotator labels the same item the same way over time. Low intra-annotator agreement signals task ambiguity or annotator inconsistency, setting an upper bound on achievable inter-annotator agreement and model performance. Measuring both separates individual reliability from cross-annotator disagreement.

With imbalance, a labeler that always predicts the majority class can score high overall accuracy while completely missing minority classes that matter. Per-class precision/recall, macro-F1, and confusion matrices reveal where the auto-labels fail, and calibration checks assess confidence reliability for thresholding. These guide where human review should focus.