Data Quality & Validation

Completeness measures the fraction of expected data values that are actually present—missing fields reduce completeness. Accuracy measures whether values correctly represent real-world facts—a price field with the wrong value is inaccurate even if present. Consistency measures whether data is free of contradictions across records or systems—a customer's age as 30 in one table and 45 in another is inconsistent. Timeliness measures whether data is current enough for its intended use—yesterday's stock prices may be stale for real-time trading. ISO 8000 defines additional dimensions: uniqueness (no duplicates), validity (values conform to defined formats/ranges). AI/ML pipelines are particularly sensitive to all five.

Great Expectations (GX) is an open-source Python library that defines Expectations—assertions about data properties (expect_column_values_to_not_be_null, expect_column_values_to_be_between). Expectations are organized into Expectation Suites and validated against DataSources (Pandas, Spark, SQL). Validation results produce Data Docs—HTML reports showing pass/fail counts per expectation. GX implements data contracts by defining expected data schema, value ranges, null rates, uniqueness, and referential integrity checks that serve as formal agreements between data producers and consumers. If validation fails, pipelines can halt and alert. GX Cloud adds scheduling, collaborative expectation management, and observability.

import great_expectations as gx
context = gx.get_context()
batch = context.sources.pandas_default.read_dataframe(df)
results = batch.validate(
    expectation_suite_name="my_suite"
)

Data drift occurs when the statistical distribution of input features changes between training time and production, degrading model performance. Types: (1) covariate shift—input X distribution changes; (2) concept drift—the relationship P(Y|X) changes; (3) label drift—target Y distribution changes. Detection methods: (1) statistical tests—Kolmogorov-Smirnov (KS) test for continuous features, chi-squared for categorical; (2) Population Stability Index (PSI)—measures distribution shift magnitude; (3) Maximum Mean Discrepancy (MMD)—kernel-based distributional distance. Tools: Evidently AI, Whylogs, Arize Phoenix, Fiddler. Alert thresholds: PSI > 0.2 indicates significant drift requiring retraining.

A schema definition specifies data structure: field names, types, nullable constraints. A data contract is a broader formal agreement between data producers and consumers covering: schema, semantics (what each field means), SLAs (freshness, availability, error rates), ownership (who maintains the data), and quality expectations (null rates, value ranges, row counts). Data contracts are enforced via automated validation in pipelines and serve as a contractual commitment—breaking a contract triggers alerts and escalation. Tools: Soda Core, Schemata (Confluent), Atlan, Monte Carlo. The data mesh movement promotes data contracts as the mechanism for decentralized data quality ownership.

Deduplication removes duplicate or near-duplicate records. Exact deduplication uses content hashing (MD5/SHA256) to identify identical records. Near-duplicate detection uses MinHash LSH (Locality-Sensitive Hashing) which generates signature sketches of documents and groups similar signatures using banding—efficient for large corpora. N-gram Jaccard similarity measures token overlap. For LLM training data, deduplication is critical because: (1) duplicates cause memorization—models memorize repeated sequences and can regurgitate them verbatim; (2) duplicates inflate effective training data size while reducing actual diversity; (3) The Pile deduplication reduced data 7–15× while improving model quality. SlimPajama applied aggressive deduplication to Red Pajama improving downstream benchmarks.

Data lineage tracks the origin, transformations, and movement of data through a system—a directed acyclic graph of data assets and transformations. For ML debugging: when a model's performance degrades, lineage enables tracing back through transformations to identify where quality issues were introduced (a bad join, a stale source, an incorrect filter). Tools: Apache Atlas, OpenLineage (standard), dbt's lineage graph, Marquez, Datahub. OpenLineage provides a vendor-neutral standard for emitting lineage events from Spark, Airflow, dbt. Column-level lineage tracks specific field derivations, enabling precise impact analysis when a source field's definition changes.

PSI measures the shift between a baseline (training) distribution and a current (production) distribution for a single feature. Calculation: bucket the feature into N bins (typically 10–20), compute the proportion of records in each bin for both distributions, then PSI = Σ [(P_current - P_expected) × ln(P_current / P_expected)]. Interpretation: PSI < 0.1 = no significant drift; 0.1–0.2 = moderate drift, investigate; >0.2 = significant drift, retraining likely needed. PSI is widely used in banking risk models (Basel compliance) and production ML monitoring. It is asymmetric with respect to swap of current/expected, so convention consistently uses training as expected.

def psi(expected, actual, bins=10):
    import numpy as np
    breakpoints = np.linspace(0, 100, bins+1)
    exp_perc = np.histogram(expected,
        bins=np.percentile(expected, breakpoints))[0] / len(expected)
    act_perc = np.histogram(actual,
        bins=np.percentile(expected, breakpoints))[0] / len(actual)
    exp_perc = np.where(exp_perc == 0, 0.0001, exp_perc)
    act_perc = np.where(act_perc == 0, 0.0001, act_perc)
    return np.sum((act_perc - exp_perc) * np.log(act_perc / exp_perc))

Schema evolution occurs when the structure of incoming data changes—new fields added, existing fields renamed, types changed, fields removed. In streaming pipelines (Kafka, Kinesis), schema changes can break consumers. Strategies: (1) Schema Registry (Confluent/AWS Glue) enforces backward/forward/full compatibility rules before a producer can publish a new schema version; (2) Avro/Protobuf/JSON Schema handle backward-compatible changes (adding optional fields) gracefully; (3) Schema migration transforms old-format events to current schema using transformation functions; (4) Versioned topics—different Kafka topics per schema version with consumer migration. Breaking changes (removing required fields) require coordinated producer-consumer deployment.

dbt provides built-in generic tests: unique (no duplicate values), not_null, accepted_values (allowlist check), relationships (foreign key integrity). Custom tests are SQL queries where a non-empty result indicates a test failure. dbt-expectations (based on Great Expectations) adds statistical tests: expect_column_values_to_be_between, expect_column_mean_to_be_between. Tests run after model materialization and fail the pipeline if violations exceed configured thresholds. dbt's test severity levels—warn (log but continue) vs. error (halt pipeline)—allow tiered response. Source freshness tests alert when upstream data is stale. Integration with data observability tools like Monte Carlo enables anomaly detection beyond rule-based tests.

# dbt test example in schema.yml
# models:
#   - name: orders
#     columns:
#       - name: order_id
#         tests:
#           - unique
#           - not_null
#       - name: status
#         tests:
#           - accepted_values:
#               values: ['placed','shipped','returned']

Label noise—incorrect annotations in training data—degrades model accuracy disproportionately: even 10% label noise can reduce accuracy 5–20% depending on model and task complexity. Models may memorize mislabeled examples or learn incorrect decision boundaries. Detection methods: (1) Confident Learning (Northcutt et al., CleanLab)—identifies likely mislabeled examples by finding samples where a trained model's confident prediction disagrees with the assigned label; (2) Cross-validation disagreement—train multiple models and flag examples where predictions consistently disagree with labels; (3) Influence functions—measure how removing each training example affects model predictions; (4) KNN consistency—flag labels that differ from k-nearest neighbors in embedding space.

Data observability (coined by Barr Moses, Monte Carlo) provides end-to-end visibility into data health across the data stack—analogous to software observability (logs, metrics, traces). It covers five pillars: freshness (when was data last updated), volume (expected row counts), distribution (statistical properties of values), schema (structural changes), and lineage (upstream dependencies). Traditional data quality monitoring checks explicit rules at scheduled intervals. Data observability uses ML-based anomaly detection to automatically learn normal patterns and alert on deviations without manual rule definition. Tools: Monte Carlo, Anomalo, Bigeye, Acceldata. Observability catches issues like silent data drops that rule-based systems miss.

Class imbalance (e.g., 99% negative, 1% positive in fraud detection) causes models to predict the majority class trivially. Handling strategies: (1) Oversampling minority class—SMOTE generates synthetic minority examples by interpolating between existing ones; (2) Undersampling majority class—random or Tomek links; (3) Class weights in loss function—penalize misclassification of minority class proportionally; (4) Threshold calibration—adjust decision threshold based on desired precision-recall tradeoff. Metrics that account for imbalance: F1 score, AUROC, AUPRC (area under precision-recall curve, more sensitive for extreme imbalance), Matthew's Correlation Coefficient. Accuracy is misleading—95% accuracy trivially achieved by predicting majority class with 95:5 imbalance.

Dataset versioning tracks exact dataset state at each training run, enabling reproducibility of past results and rollback when data quality issues are discovered. Tools: DVC (Data Version Control) uses git-like commands to version large files stored in remote storage (S3, GCS, Azure Blob), tracking dataset versions alongside model code in git. Delta Lake and Apache Iceberg provide table-level time travel—querying data as of a past timestamp. Hugging Face Datasets uses git-lfs-backed versioning. Without versioning: a data quality fix may improve current accuracy but break comparison with historical baselines; debugging production model degradation requires knowing exactly what data the model was trained on.

For tabular data: outliers detected via statistical methods (Z-score >3, IQR fences), isolation forests (unsupervised tree-based), DBSCAN (density-based clustering), or Local Outlier Factor. For text in LLM training: (1) perplexity filtering—compute perplexity of each document under a reference model; documents with very high perplexity are likely garbled, non-English, or spam; (2) embedding-space outliers—compute document embeddings and flag examples far from cluster centroids; (3) rule-based heuristics—flag documents with extreme compression ratios (repetitive text), too many special characters, or non-UTF-8 encoding. The Dolma and FineWeb datasets apply combination of perplexity and rule-based filtering.

The data validation pyramid prioritizes: (1) Schema/format checks (bottom)—fast, cheap, catches structural issues; column existence, data types, null constraints. Run on 100% of records; (2) Range/constraint checks—value bounds, enum validation, referential integrity; (3) Statistical/distribution checks—mean, std, histogram comparisons vs. historical baselines; computationally heavier, may run on samples; (4) Business rule checks—complex multi-table consistency rules, derived metric sanity (revenue > 0, refunds < sales); (5) ML-specific checks (top)—feature distribution drift, label distribution, prediction score drift. Lower layers run more frequently (per pipeline run); upper layers may run on schedules (daily/weekly).

LLM fine-tuning data quality filters: (1) Length filters—minimum/maximum token counts per example (typically 10–2048 tokens); (2) Repetition detection—flag documents with >30% repeated n-grams or paragraphs; (3) Language detection (fastText or langdetect)—ensure target language only; (4) Deduplication—exact and near-duplicate removal; (5) Perplexity filtering—score with a small reference LM, remove very high or very low perplexity examples; (6) Toxicity filtering—NSFW classifier scores; (7) PII detection—flag/redact personal information; (8) Quality model scoring—train a quality classifier on human-preferred vs. dispreferred examples and filter on score. DataComp showed aggressive filtering yields better models than large unfiltered corpora.

Data freshness SLA defines the maximum acceptable delay between when data is generated and when it is available in the target system (e.g., 'event data must be available within 15 minutes of occurrence'). Monitoring approaches: (1) watermark-based—track the timestamp of the most recent record in the target table and alert if max(event_timestamp) is older than now - SLA_threshold; (2) row count monitoring—alert if incoming row count drops below expected minimum (indicating feed failure); (3) metadata-based—check pipeline run timestamps from orchestration system; (4) heartbeat records—synthetic records inserted every N minutes whose presence confirms pipeline health. Tools: Great Expectations source freshness tests, dbt source freshness, Monte Carlo. SLA violations trigger escalation.

In distributed pipelines (Data Lakes, Lakehouse), foreign key constraints aren't enforced by the storage engine. Validation approaches: (1) JOIN-based orphan detection—LEFT JOIN child to parent table and flag unmatched rows (e.g., orders with non-existent customer_id); (2) Set membership checks—compare child foreign key values against parent primary key set using broadcast joins for small parent tables or bloom filter approximations for large ones; (3) dbt relationships test—built-in test verifying all values in a column exist in a referenced model's column; (4) Delta constraints—enforce NOT NULL and CHECK constraints in Delta Lake tables (enforced at write time). Periodic (daily) referential integrity scans catch cross-table inconsistencies introduced by async pipeline delays.

Pandera is a Python library for DataFrame validation that integrates with pandas and PySpark. It defines DataFrameSchema objects specifying: column types, nullable constraints, value checks (Check.ge(0) for non-negative values), string pattern matching, and custom check functions. Validation raises SchemaError on failure with detailed error messages. Pandera supports type annotations for schema-as-code and generates synthetic test data from schemas. Unlike Great Expectations (document-centric), Pandera is code-centric with Pythonic API, making it natural for data scientists. It integrates with FastAPI for validating request/response DataFrames and supports lazy validation (collecting all errors before raising).

import pandera as pa
schema = pa.DataFrameSchema({
    "id": pa.Column(int, pa.Check.gt(0)),
    "price": pa.Column(float, pa.Check.between(0, 10000)),
    "category": pa.Column(str, pa.Check.isin(["A","B","C"]))
})
validated_df = schema.validate(df)

Data drift (covariate shift) means input feature distributions P(X) change, while the true relationship P(Y|X) remains stable. Detected by monitoring input feature distributions (KS test, PSI). Addressed by retraining on recent data or adding recent data to training. Concept drift means the true relationship P(Y|X) changes—even if inputs look the same, the correct output changes (e.g., consumer behavior shifts post-pandemic making historical purchase patterns poor predictors). Detected by monitoring model performance metrics (accuracy, AUC) against labeled current data, which requires ground truth labels—often delayed by days/weeks. Addressed by retraining on recent data with current labels or online learning. Concept drift is harder to detect because it requires delayed ground truth.

Data profiling computes descriptive statistics for each feature to understand its properties and identify quality issues. For numeric features: count, null count, min, max, mean, median, std, percentiles (p25, p75, p95, p99), skewness, kurtosis, cardinality, histogram. For categorical: count, null count, cardinality, top-K value frequencies, value distribution, samples. For text: length statistics (min/max/mean character count), word count distribution, language distribution. Libraries: pandas-profiling (now YData Profiling), Apache Spark's df.describe(), AWS SageMaker Data Wrangler. Profiling should be run on new datasets before training and periodically on production data to detect drift and guide preprocessing decisions.

Fail-fast halts the pipeline immediately when a critical data quality check fails, preventing bad data from propagating downstream to production tables, dashboards, or models. It is appropriate for: schema violations, primary key uniqueness failures, or null rates exceeding a hard threshold. The pipeline is reprocessed after the data issue is resolved. Warn-and-continue logs the violation and continues processing, appropriate for: minor distribution shifts, elevated null rates in non-critical fields, or edge cases expected in real data. Great Expectations and dbt implement both via severity levels (error vs. warn). Production systems typically use fail-fast for critical quality SLAs and warn for advisory metrics.

Annotation quality is measured via: (1) Inter-annotator agreement (IAA)—Cohen's Kappa for two annotators, Fleiss's Kappa for multiple; values >0.8 indicate strong agreement; (2) Gold standard accuracy—a subset of items with known ground truth measures individual annotator accuracy; (3) Majority vote baseline—use 3+ annotators per item and take majority, measuring disagreement rate. Quality improvement strategies: (1) Annotation guidelines and training examples; (2) Calibration rounds with feedback; (3) Annotator-level quality monitoring and removal of low-accuracy annotators; (4) Hierarchical annotation with expert review of uncertain items; (5) Active learning to prioritize uncertain examples for human review. Platforms: Scale AI, Labelbox, Prodigy, Amazon MTurk with qualification tests.

DataOps applies DevOps principles to data: (1) Version control—all pipeline code, SQL transformations, and schema definitions in git; (2) Automated testing—data quality tests run in CI/CD before deploying pipeline changes; (3) Environment parity—development, staging, and production data environments with controlled data subsets; (4) Observability—metrics, logging, and alerting on pipeline health and data quality; (5) Iterative development—incremental pipeline changes with quality gates; (6) Collaboration—shared ownership between data engineers, analysts, and scientists. Tools: dbt + GitHub Actions for SQL pipeline CI/CD; Airflow with DAG validation; Great Expectations integrated into CI. DataOps reduces mean time to detect and fix data quality issues from days to hours.

TDDA (Test-Driven Data Analysis) applies test-driven development principles to data: (1) Constraints are automatically inferred from reference data using the tdda Python library—it generates constraints file (.tdda JSON) capturing min/max, type, nullability, allowed values; (2) These constraints become automated tests verified against new data batches; (3) Violations are reported with exact row-level detail. Unlike manually written expectations, TDDA constraints are data-derived, capturing what the data actually is, not what developers think it should be. This catches subtle distribution changes that hand-crafted rules miss. Integration with pytest enables running data constraint tests alongside unit and integration tests in CI pipelines.

Training-serving skew occurs when features computed at training time differ from those computed at serving time—due to different code paths, time zones, rounding, or data sources. Feature stores (Feast, Tecton, Hopsworks, AWS SageMaker Feature Store) centralize feature computation to eliminate skew: the same feature transformation code runs for both training data retrieval and online serving. Validation includes: (1) point-in-time correctness—training features use only data available before each training example's timestamp; (2) feature schema validation—type and range checks at write and read time; (3) feature drift monitoring—compare training-time feature distributions to serving-time distributions; (4) consistency tests—validate that batch and streaming computation paths produce identical results.

Data augmentation synthetically increases training data diversity to improve model robustness. For images: random flips, rotations, crops, color jitter, Mixup (label-weighted blending of two images), CutMix. For text/NLP: back-translation (English→French→English for paraphrases), synonym replacement (EDA: Easy Data Augmentation), random insertion/deletion/swap, paraphrasing with LLMs. For speech: speed perturbation, pitch shifting, noise addition. For tabular: SMOTE (synthetic minority oversampling), Gaussian noise injection. Augmentation improves quality by: increasing data diversity, reducing memorization of specific examples, and artificially balancing class distributions. However, augmented examples must preserve semantic validity—invalid augmentations introduce noise.

Synthetic data quality validation: (1) Fidelity—statistical similarity to real data: compare marginal distributions (KS test per feature), joint distributions (correlation matrices, PCA projections), and nearest-neighbor distance ratios between real and synthetic samples; (2) Utility—train a model on synthetic data and evaluate on real test data (TSTR) vs. train on real, test on real (TRTR)—the gap measures utility; (3) Privacy—verify synthetic records don't memorize real records (membership inference attack success rate should be near random); (4) Diversity—measure generation coverage and avoid mode collapse. Tools: SDV (Synthetic Data Vault), Gretel.ai, most.ai, Syntheticus all provide built-in quality reports.

Missing data mechanisms: MCAR (Missing Completely At Random)—missingness is independent of data; MAR (Missing At Random)—missingness depends on observed variables; MNAR (Missing Not At Random)—missingness depends on the missing value itself. Imputation methods: MCAR/MAR: mean/median imputation (introduces bias in variance), multiple imputation (MICE—generates M imputed datasets, models uncertainty), KNN imputation (uses nearest neighbors), model-based (gradient boosting imputation). MNAR: requires explicit modeling of the missingness mechanism (e.g., Heckman correction). For ML: tree-based models handle NaN natively (XGBoost, LightGBM); neural networks require imputation or missingness-aware architectures.

Apache Iceberg provides: (1) ACID transactions—concurrent writes don't corrupt data, enabling reliable pipeline updates; (2) Time travel—query data as of any past snapshot, enabling rollback when quality issues are discovered; (3) Schema evolution—safely add, rename, or change column types without rewriting data; (4) Partition evolution—change partitioning strategy without data migration; (5) Row-level deletes—GDPR compliance deletes without full table rewrites; (6) Hidden partitioning—eliminates partition pruning errors from incorrect user partition paths. Iceberg's metadata layer tracks file-level statistics (min/max per column per file) enabling efficient predicate pushdown. Combined with Delta Lake or Hudi, these features form the data quality foundation of modern lakehouses.

Z-score outlier detection: for each value x, compute z = (x - μ) / σ; flag |z| > 3 as outliers. Simple and fast for normally distributed data. Failure cases: (1) Non-normal distributions—for skewed data, Z-score flags legitimate extreme values and misses true outliers; use modified Z-score with median (Iglewicz & Hoaglin) instead; (2) Small samples—mean and std are unstable, causing high false positive/negative rates; (3) Masking—multiple outliers inflate μ and σ, hiding each other ('masking effect'); (4) High dimensions—all points appear equidistant in high-dimensional spaces, making Z-score per feature miss multivariate outliers. Use IQR fence (Q3 + 1.5×IQR) or robust statistics for skewed data.

A Data Quality Score aggregates multiple dimension scores into a single 0-100 metric. Example formula: DQS = w1 × Completeness + w2 × Accuracy + w3 × Consistency + w4 × Timeliness + w5 × Uniqueness, where weights reflect business importance. Completeness = 1 - null_rate; Uniqueness = 1 - duplicate_rate; Timeliness = max(0, 1 - delay_hours/SLA_hours); Accuracy requires ground truth comparison or proxy rules. Weights are domain-specific: financial data may weight Accuracy 0.5; IoT sensor data may weight Timeliness 0.4. DQS enables: dashboard reporting, SLA compliance tracking, regression testing (DQS should not decrease between pipeline versions), and data product comparison.

Soda Core is an open-source Python library defining data quality checks in YAML (SodaCL—Soda Check Language). Checks specify: row_count > 1000, missing_percent(email) < 5%, duplicate_count(user_id) = 0, schema columns match expected list. Soda Cloud adds collaborative dashboards, alerting, and incident management. Comparison to Great Expectations: (1) Soda uses YAML configuration vs. GX's Python-centric API; (2) Soda's syntax is more readable for non-programmers; (3) GX has larger ecosystem (more DataSource integrations, more active community); (4) Both support similar check types; (5) GX's Data Docs provide richer HTML reporting; (6) Soda integrates natively with Airflow and dbt via Soda Scan. Choice depends on team preference for YAML vs. Python.

Data masking replaces sensitive values with realistic but non-identifiable substitutes: names→random names, SSNs→format-preserving random numbers, credit cards→Luhn-valid random numbers. Required for compliance: GDPR (EU), HIPAA (US healthcare), PCI-DSS (payment data), CCPA (California). ML-specific masking requirements: (1) PII in training data must be masked or pseudonymized before model training to prevent model memorization; (2) Protected attributes (race, gender, religion) should be masked in training data if used only as confounders; (3) Test data used for evaluation must be masked to prevent analysts from accessing real PII. Format-preserving encryption (FPE) maintains data structure while masking—a 16-digit number maps to a different 16-digit number.

Data quality incident response: (1) Detection—automated monitoring alerts trigger (data freshness SLA breach, null rate spike, schema change, volume anomaly); (2) Triage—data engineer assesses severity using impact matrix (how many downstream consumers affected, is production model impacted?); (3) Containment—quarantine bad data (mark partition as quarantined, halt downstream pipelines from consuming it, roll back to last known good version); (4) Root cause analysis—use lineage to trace to source, examine pipeline logs for failed transforms or upstream feed issues; (5) Remediation—fix source, backfill affected partitions, reprocess downstream; (6) Post-mortem—document timeline, root cause, fix, and process improvements; (7) Preventive controls—add validation test to catch same issue in future.

FineWeb (Hugging Face, 2024) is a 15T token open web dataset curated from 96 Common Crawl dumps with extensive quality filtering: (1) URL filtering—blocklists for adult content, spam domains; (2) Trafilatura extraction—HTML→text with quality scoring; (3) Language detection—English-only with fastText; (4) MinHash deduplication within and across dumps; (5) Quality heuristics—C4 rules (no terminal punctuation removal, no lorem ipsum), Gopher rules (word count bounds, stop word ratios); (6) Educational quality filtering using a quality classifier trained on high-quality vs. random web text. FineWeb-Edu further filters for educational content using a LLaMA-scored annotation model, producing 1.3T tokens of educational text that significantly improves math and reasoning benchmarks.

Rule-based checks use explicitly defined thresholds (null_rate < 5%, value in ['A','B','C']) that require domain knowledge and manual maintenance. They catch known issues but miss novel anomalies. Model-based anomaly detection (Isolation Forest, Autoencoder, statistical models) learns normal patterns automatically from historical data and flags deviations without explicit rules. Advantages: (1) catches unexpected anomalies (unknown unknowns); (2) adapts to seasonal patterns automatically; (3) handles multivariate anomalies that rule-based miss (individually valid values that are jointly anomalous). Disadvantages: requires historical data to learn from; false positive rates can be high; less interpretable when explaining why something is anomalous. Production systems use both: rules for known critical checks, ML anomaly detection for catch-all monitoring.

Write-Audit-Publish (WAP) is a data quality gate pattern: (1) Write—new data is written to a private/staging partition or table not visible to consumers; (2) Audit—data quality validation runs against the staging partition (row counts, null checks, distribution tests, business rules); (3) Publish—only if all audit checks pass does the pipeline atomically publish (swap) the staging partition to the production table visible to consumers. Implementations: Delta Lake and Iceberg support atomic partition swaps; dbt + dbt-audit-helper implement this with view swapping. WAP ensures consumers never see partially-processed or quality-failed data. It is the production-grade evolution of running validation after writes.

Shannon entropy H = -Σ p_i × log₂(p_i) measures uncertainty or diversity in a distribution. For categorical features: high entropy (near log₂(N) for N categories) indicates a uniform distribution across categories; low entropy (near 0) indicates concentration in one category. Data quality use cases: (1) detecting degenerate features—a feature with entropy 0 has only one value (constant, useless for ML); (2) detecting label leakage—entropy of target variable should be similar in train and test splits; (3) monitoring for distribution shifts—entropy change signals that category proportions have shifted; (4) feature quality scoring—very low entropy features may not add information beyond a constant. Relative entropy (KL divergence) measures distribution shift between time periods.

import numpy as np
from collections import Counter
def feature_entropy(values):
    counts = Counter(values)
    total = len(values)
    probs = [c / total for c in counts.values()]
    return -sum(p * np.log2(p) for p in probs if p > 0)

Active learning selectively queries the most informative examples for human annotation rather than labeling randomly. Strategies: (1) Uncertainty sampling—query examples where the model is least confident (maximum entropy or minimum margin); (2) Query-by-committee—select examples where an ensemble of models most disagrees; (3) Core-set selection—select examples that best represent the unlabeled pool geometrically (maximally diverse coverage); (4) Expected model change—select examples that would most change model parameters. Quality impact: annotators spend effort on genuinely ambiguous or representative examples rather than easy or redundant ones. Active learning reduces labeling cost 60–80% to achieve the same model accuracy as random sampling on benchmark datasets.

The medallion architecture (Databricks) organizes a lakehouse into quality-tiered layers: (1) Bronze—raw data landed from sources with minimal transformation, preserving original data exactly (ACID write, source system metadata, ingestion timestamps, no schema enforcement); (2) Silver—cleansed and validated data: deduplication, null handling, type casting, referential integrity validation, schema enforcement, PII masking; (3) Gold—business-level aggregated tables: fully validated, enriched, modeled for specific analytical or ML use cases. Each layer applies progressively stricter data quality checks. Most ML models consume silver or gold layer data. Data lineage is maintained across layers enabling rollback if upstream quality issues propagate.

Statistical tests for distribution equality: (1) Kolmogorov-Smirnov (KS) test—compares empirical CDFs; non-parametric, works for continuous distributions, sensitive to all distribution differences. Scipy: ks_2samp(sample1, sample2). (2) Anderson-Darling test—more sensitive to tails than KS. (3) Chi-squared test—for categorical features: compare observed vs. expected frequency distributions across bins. (4) Mann-Whitney U test—non-parametric test for equal medians. (5) Wasserstein distance (Earth Mover's Distance)—measures distribution shift with meaningful magnitude interpretation. For ML pipelines, two-sample KS test (p < 0.05 suggests different distributions) and PSI are most commonly used. Statistical significance must be balanced with practical significance—large datasets detect tiny differences as statistically significant.

from scipy import stats
# For continuous features:
stat, p_value = stats.ks_2samp(train_data, prod_data)
if p_value < 0.05:
    print('Distributions differ significantly')
# For categorical:
stat, p_value = stats.chisquare(f_obs=prod_counts, f_exp=train_counts)

A data catalog is a centralized metadata repository documenting available data assets, their schemas, owners, lineage, quality metrics, and access policies. Data quality governance features: (1) Data quality scores attached to each dataset (automatically updated from monitoring tools); (2) Certification tiers—data stewards certify high-quality datasets for trusted analytics; (3) Business glossary—standardizes definitions to prevent consistency issues (is 'active customer' defined consistently across teams?); (4) Lineage visualization—shows upstream dependencies and downstream impact; (5) Access controls—enforces who can read/modify each dataset. Tools: Alation, Collibra, Datahub (open-source), Atlan, AWS Glue Data Catalog. Catalogs become the single source of truth for data asset quality and trustworthiness.

Checksums (MD5, SHA-256, CRC32) verify data integrity by detecting corruption or tampering during transfer or storage. In ML pipelines: (1) file-level checksums verify dataset files match expected content after download (Hugging Face datasets store SHA256 per file); (2) chunk-level checksums in Parquet/ORC files detect partial corruption; (3) row count reconciliation verifies source-to-destination record counts match after ETL; (4) column-level hash comparison (hash all values in a column, compare between source and target) detects silent data corruption; (5) cryptographic hashes in audit logs verify logs haven't been tampered with. DVC stores SHA256 hashes of all versioned data files, enabling tamper detection when retrieving historical datasets.

Feature engineering validation strategies: (1) Unit tests on transformation functions—test edge cases (zero values, nulls, extreme values) with known expected outputs; (2) Property-based tests (Hypothesis library)—generate random valid inputs and verify invariants hold (e.g., normalized features should have zero mean ± 1e-6); (3) Distribution preservation checks—original feature and transformed feature should have expected statistical relationships (log transform of positive values should shift skew); (4) Train/test leakage detection—assert that normalization statistics (mean, std) are computed only on training set and applied to validation/test; (5) Reversibility tests—for invertible transforms (log, standardization), verify round-trip recovery within floating-point tolerance. Transformations should be wrapped in sklearn Pipelines to ensure consistent application.

In data quality validation, the quality issue detection classifier itself has precision and recall: Precision = TP / (TP + FP)—fraction of flagged records that are genuine quality issues. Recall = TP / (TP + FN)—fraction of genuine quality issues that are flagged. In validation contexts: high precision means few false alarms (wasted engineer time); high recall means few missed quality issues (data errors reaching production). The tradeoff depends on consequences: for critical financial data, prefer high recall (catch all errors, even at cost of false alarms); for informational dashboards, prefer high precision (avoid unnecessary pipeline halts). This framework extends to evaluating the quality of any anomaly detection or validation system.

Curriculum learning (Bengio et al., 2009) trains models on examples ordered from easy to hard, mimicking how humans learn. For LLMs: (1) early training on high-quality, simple, diverse text; (2) later introduction of domain-specific, complex, or long-form content; (3) final fine-tuning on task-specific data. Quality benefits: models trained with curriculum learn more stable representations early, preventing memorization of noise before the model understands concepts. DataComp experiments show ordering datasets by quality score and introducing complex examples later improves downstream accuracy 2–5% vs. uniform random sampling. Anti-curriculum (hard first) generally underperforms. LLaMA 3's training involved careful data mixing ratios and ordering across web, code, and academic text.