Feature stores and data pipelines

A feature store is a centralized system for defining, storing, and serving ML features consistently across training and inference. It eliminates duplicated feature engineering, guarantees the same transformation logic offline and online, and enables feature sharing/discovery across teams. Its dual-store design (offline for batch training, online for low-latency serving) is what distinguishes it from a plain data warehouse.

Training-serving skew is the discrepancy between feature values used during training and those computed at inference, usually caused by separate code paths or different data sources. Feature stores mitigate it by sharing a single transformation definition and ideally the same materialized values between the offline store (training) and online store (serving), so a feature computed once is reused everywhere.

A point-in-time (PIT) join retrieves each feature's value as it was known at the timestamp of each label/entity event, never using values recorded after that event. It prevents label leakage where future information contaminates training. Feature stores implement it via an as-of join between the entity/event timeline and the time-versioned feature history.

# Feast example
from feast import FeatureStore
store = FeatureStore(repo_path=".")
training_df = store.get_historical_features(
    entity_df=entity_df,  # has event_timestamp per row
    features=["txn_stats:amount_7d_avg"],
).to_df()

The offline store (e.g., warehouse, Parquet on object storage) holds large historical feature data for training and batch scoring and is optimized for throughput. The online store (e.g., Redis, DynamoDB) holds the latest feature values per entity and is optimized for millisecond-latency lookups during real-time inference. The split reconciles the conflicting needs of bulk historical access versus low-latency point reads.

Materialization is the process of computing/loading feature values from the offline store into the online store so they are available for low-latency serving. It typically runs incrementally over a time window, pushing the latest value per entity key into the online store. Without materialization the online store would be empty or stale.

from datetime import datetime, timezone
store.materialize_incremental(end_date=datetime.now(timezone.utc))

An entity is the object that features describe and the key used to join and look up features, e.g., user_id, product_id, or merchant_id. Feature views/groups are keyed by one or more entities, and online lookups provide entity key values to fetch the corresponding feature vector. Choosing the right entity granularity is foundational to the design.

A feature view is a logical group of related features that share the same entity key and data source, along with metadata like the timestamp column and TTL. It is the unit of materialization and retrieval: you select features from one or more views to build training or serving vectors. It binds a source query/transformation to its serving configuration.

TTL bounds how far back in time a feature value remains valid relative to the lookup/event timestamp. During point-in-time joins, a feature older than the TTL is treated as missing (null) rather than stale-but-valid, preventing the use of outdated values. It controls staleness semantics for training joins (and, in some implementations, online retrieval).

Batch features are precomputed on a schedule from historical data (e.g., 30-day spend). Streaming features are updated continuously from event streams for near-real-time freshness (e.g., last-5-min clicks). On-demand features are computed at request time from request payload and/or fetched features (e.g., distance between request location and a stored home location), since they cannot be precomputed.

Some features depend on inputs available only in the inference request (e.g., the current transaction amount, query string, or live GPS) and therefore cannot be precomputed and materialized. On-demand transformations compute these at serving time, often combining request data with retrieved stored features. They must use the exact same code at training time to avoid skew.

@on_demand_feature_view(
  sources=[txn_fv],
  schema=[Field(name="amt_vs_avg", dtype=Float32)],
)
def ratio(inputs):
  import pandas as pd
  df = pd.DataFrame()
  df["amt_vs_avg"] = inputs["amount"] / inputs["amount_7d_avg"]
  return df

Online stores are chosen for single-digit-millisecond point lookups by entity key, favoring key-value/in-memory systems like Redis, DynamoDB, Cassandra, or Bigtable. Trade-offs include latency vs. cost, durability, and the ability to handle high read QPS with large entity cardinality. Throughput-oriented columnar stores are unsuitable for per-request online serving.

Feature lineage tracks the upstream data sources, transformations, and jobs that produce each feature, plus the downstream models and services consuming it. It enables impact analysis (what breaks if a source changes), debugging of bad predictions, compliance/audit, and safe deprecation of features. Without lineage, governance and root-cause analysis become guesswork.

Data (feature/covariate) drift is a change in the distribution of input features P(X) over time, while concept drift is a change in the relationship P(y|X) between features and target. Monitoring feature distributions catches data drift; it does not by itself reveal concept drift, which requires label feedback and performance tracking. Both can degrade a deployed model.

An idempotent pipeline produces the same result whether a step runs once or is retried/re-run multiple times for the same input/partition. It is essential because pipelines fail and get retried; idempotency (e.g., partition overwrite, upsert by key) prevents duplicate or corrupted data. It is typically achieved by writing to deterministic partitions and using insert-overwrite or merge semantics.

Backfill recomputes historical feature values for past time ranges, e.g., when adding a new feature or fixing a bug. The key pitfall is leakage: backfilling must use only data that was actually available at each historical timestamp (point-in-time correctness), otherwise the regenerated training set contains future information and inflates offline metrics.

Most target at-least-once delivery combined with idempotent/upsert writes to achieve effectively-once results, because true end-to-end exactly-once is expensive and hard across heterogeneous sinks. By making the sink idempotent (keyed upserts), duplicates from retries are harmless. At-most-once is avoided because it can silently drop feature updates.

Event time is the timestamp when an event actually occurred (embedded in the data); processing time is when the system processes it. Using event time with watermarks yields correct, reproducible windowed aggregations despite out-of-order/late arrivals, whereas processing-time windows are simpler but non-deterministic and skew-prone. Feature aggregations should generally use event time.

A watermark is a heuristic threshold asserting that no (or few) events with timestamps earlier than it will still arrive, allowing the system to close event-time windows and emit results. It trades latency against completeness: a looser watermark waits longer for late data, a tighter one emits sooner but may drop stragglers. It is central to bounded-latency event-time aggregations.

Lambda architecture runs parallel batch and speed (streaming) layers, merging their outputs to balance accuracy and freshness, but it forces maintaining two codebases for the same logic. Kappa architecture eliminates the batch layer by treating everything as a stream and reprocessing from the log when needed, reducing duplicated logic and skew risk. Many feature platforms now lean toward unified/stream-first designs.

A schema registry validates that producers and consumers agree on field names, types, and evolution rules, preventing malformed data from silently corrupting features. It enables safe schema evolution (backward/forward compatibility) so pipelines don't break when a producer adds or changes a field. This is critical for the reliability of downstream feature computation and model inputs.

A data contract is an explicit, enforced agreement between data producers and consumers specifying schema, semantics, quality SLAs, and ownership. It shifts responsibility upstream so breaking changes are caught at the source rather than corrupting features downstream. Contracts make feature pipelines resilient to upstream changes and clarify accountability when data breaks.

Bronze holds raw ingested data as-is, silver holds cleaned/conformed/validated data, and gold holds curated, aggregated, business-ready tables (often the feature-serving layer). Each layer increases quality and structure, enabling reprocessing from raw and clear separation of concerns. Features are frequently derived from silver and published to gold/feature store.

They add ACID transactions, schema evolution, and time travel on top of object storage, so concurrent writes don't corrupt tables and you can query a feature table 'as of' a past snapshot. Time travel and snapshot isolation directly support reproducible training datasets and safe backfills. They also enable efficient upserts/merges and partition pruning.

Reuse avoids re-implementing the same feature (and its bugs/skew) in every project, saving effort and improving consistency. It requires governance: a discoverable catalog with documentation, ownership, versioning, access control, and deprecation policy so consumers can trust and safely depend on shared features. Without governance, shared features become fragile hidden dependencies.

Create a new immutable feature version (e.g., spend_30d_v2) rather than mutating the existing one in place, so already-trained models and historical training sets remain reproducible. Run old and new versions in parallel during migration, then deprecate the old version once consumers move. In-place redefinition silently breaks reproducibility and lineage.

A newly defined feature has no values in the online store until materialization/backfill populates it, so early serving requests get nulls or fail. Mitigation is to backfill the online store for active entities before routing traffic and to handle missing values with documented defaults. Streaming features may also need warm-up time to accumulate state.

Key metrics include data freshness/latency (lag between event time and availability), completeness (null/missing rates), volume/row-count anomalies, distribution stats vs. baseline, and pipeline success/SLA timeliness. Freshness lag is especially important for streaming features where staleness directly degrades predictions. These feed alerting before model performance is impacted.

A DAG (directed acyclic graph) encodes pipeline tasks as nodes with directed dependency edges and no cycles, so the orchestrator can determine execution order, parallelism, and retries. It makes data pipelines reproducible, observable, and recoverable, with scheduling, backfills, and dependency-aware reruns. Acyclicity guarantees a valid topological execution order.

Airflow is a general-purpose workflow orchestrator that schedules and runs arbitrary tasks and manages dependencies, while dbt is a SQL-centric transformation tool that builds and tests models inside the warehouse (the 'T' in ELT). They are complementary: Airflow commonly orchestrates dbt runs alongside ingestion and ML steps. dbt does not schedule infrastructure-level workflows by itself.

ELT loads raw data into a scalable warehouse/lakehouse first and transforms it there using the warehouse's compute, leveraging elastic scale and keeping raw data for reprocessing. ETL transforms before loading, which couples transformation to a separate engine and discards intermediate raw data. ELT improves flexibility, reproducibility, and the ability to redefine features later.

A DLQ captures messages that repeatedly fail processing (e.g., malformed or schema-violating events) so the main stream keeps flowing instead of blocking or dropping data silently. It enables later inspection, debugging, and reprocessing after a fix. Without a DLQ, poison messages can stall consumers or cause silent data loss.

Use techniques like hashing (the hashing trick), learned embeddings, or target/frequency encoding rather than raw one-hot, which explodes dimensionality and online lookup cost. Embeddings are often precomputed and stored as feature vectors in the online store, or generated by an embedding service. The goal is bounded, low-latency feature vectors with strong predictive signal.

A vector database serves approximate nearest-neighbor (ANN) similarity search over embeddings to retrieve semantically relevant context, whereas an online feature store serves exact key-based lookups of precomputed features by entity. They are complementary: a feature store answers 'give me this user's features', a vector DB answers 'find items most similar to this embedding'. Some modern feature platforms now integrate vector retrieval.

Define explicit, versioned imputation/default logic inside the shared feature definition so the same handling applies in both offline training and online serving. Avoid implicit framework-specific defaults that differ between pipelines, since divergent imputation is a classic source of training-serving skew. Record missingness indicators when absence is itself informative.