What is training-serving skew?

Where the gap comes from

A model is trained on features computed one way, often in a notebook or a warehouse query. In production, the same features get recomputed by different code, on a different path, under a time budget. Small differences in how a value is calculated add up to a model that reads inputs it never trained on.

Common causes

1. Two implementations of one feature. A data scientist writes the feature in Python or SQL against the warehouse. An engineer rewrites it for the serving path in another language. The two disagree on details nobody wrote down: how nulls are handled, where a day starts, whether a refund cancels an order.
2. Different freshness. Training reads values computed after the fact from complete history. Serving reads whatever was current at request time, which may lag behind what the customer just did. The model learned from values fresher than the ones it gets. Feature freshness covers this gap.
3. Values from the future. A training set built by joining labels to today’s feature values lets the model see information that did not exist when the outcome happened. It learns from data it will never have in production. A point-in-time correct training set uses the value each feature had at the moment of each label.
4. Late and out-of-order events. A streaming job counts events as they arrive. A batch job recomputes from the full table later, including events that arrived late. Over a window, the two produce different counts for the same customer.
5. Different identity. Training data is often stitched in the warehouse, so each row is a whole person. At request time the app may only have a cookie, so the model scores a fragment of that person. Identity stitching has to happen when features are served as well as when they are trained on.

Example: a churn model

An illustration. A subscription app trains a churn model on “sessions in the last seven days”, computed in the warehouse with calendar days in UTC. In production, a streaming job counts a rolling seven days in the customer’s own time zone, and drops events that arrive more than a few minutes late. Both definitions are reasonable, but they give different counts for the same customer. The model learned where leavers and stayers divide on one definition and applies it to the other, so it flags the wrong customers.

Why it is hard to catch

Skew does not throw an error. The model still returns an answer, just a worse one. Accuracy that looked good in testing slips in production, and the cause is buried in the difference between two feature pipelines rather than in the model itself.

Offline evaluation does not catch it, because it runs on the training pipeline’s values. The usual way to find it is to log the features served at request time and compare them with the values the training pipeline computes for the same customer and moment. Any difference is skew.

How feature stores address it

A feature store computes a feature from one definition for both training and serving. The value a model learns from and the value it reads in production come from the same logic, so the two cannot quietly diverge.

One definition removes the first cause. The others depend on how the store handles time and identity: whether it builds point-in-time correct training sets, how fresh the served value is, and what key the features are computed on. See online vs offline features for how the two paths relate.

How Signals handles it

Signals calls each feature an attribute. One attribute definition runs in a streaming engine for the live value and a batch engine for warehouse history, so the value served in production matches what the model trained on. Its dataset builder creates point-in-time correct training sets from the same definitions, in the warehouse.