In this article
Before you start
This page is one step on the machine learning operations (MLOps) learning path.
Why interviewers ask this
A feature store question tests whether you understand why many machine learning (ML) systems fail after the model works. The model code is a small part of the system; the data path around it is most of it. Sculley and colleagues describe the pattern in "Hidden Technical Debt in Machine Learning Systems": quick-to-build prediction systems carry large ongoing maintenance costs, including data dependencies and tangled pipelines (Sculley et al., 2015).
Training-serving skew is a gap between how a model performs in training and how it performs in serving; a common cause is that it sees different feature values or logic in the two. A weak answer draws a database and a cache. A strong answer starts from the failure the store prevents, a model that sees different feature values in production than it saw in training, then names the two stores, the join, the freshness path and the latency budget. A common prompt adds a target, such as 50,000 requests per second with a 50-millisecond p99, and asks how the design changes.
A feature store from first principles
- Feature store
- A feature store is a system that defines machine learning features once, computes and stores their values over time, and serves them for both training and low-latency prediction so the two use identical logic.
- Point-in-time join
- A point-in-time join is a join that attaches to each training example the latest feature value whose event time is at or before that example's timestamp, never a value from after it.
- Online store
- An online store is a low-latency key-value database that holds the latest value of each feature per entity so a prediction request can read it in milliseconds.
- p99 latency
- The p99 latency (99th-percentile latency) is the time within which 99 of every 100 requests finish; the slowest 1 in 100 take longer.
In plain words
A feature store is a shared pantry with a recipe book. The book says how each ingredient is prepared. The back freezer keeps every batch ever made, labelled with its date. The front shelf holds only the freshest batch, for cooks who need it fast. Everyone cooks from the same recipe, so the dish tastes the same in the test kitchen and in the restaurant. The analogy breaks here: a feature value can be corrected or back-filled later, so dates matter.
In one precise sentence: a feature store is a registry of feature definitions, a historical store of timestamped values, an online store of latest values, and the pipelines and join logic between them. Uber describes its version as a shared catalogue. Teams publish features with an owner, a description and a service level agreement (SLA). In 2017 it held about 10,000 features, updated daily (Uber Engineering, September 2017).
Step 1: Define the feature once
The registry holds one definition per feature: its name, the entity it describes, its data type, how it is computed and a time-to-live (TTL), the age after which a value is too old to use. An entity is the thing a feature describes, such as a user, an item or a merchant. The input is a piece of code or a query; the output is a named, versioned feature that both training and serving will use.
One definition is the point of the whole system. If the training pipeline computes "orders in 7 days" with one query and the serving code recomputes it with another, the two will drift. Add an owner and a description at this step so another team can find and trust the feature. What you now know: the feature has a single source of truth before any data moves.
Step 2: Compute values into the historical store
Pipelines compute the feature and write every value to the historical store with two timestamps: the event time, when the value became true, and the created time, when it was written. Batch jobs fill the history for slow features, such as a seven-day average, and stream processors update fast ones, such as a one-hour count. Uber's post describes exactly this split, with batch precomputation and near-real-time computation through Kafka, a message log, and Samza, a stream processor (Uber Engineering, September 2017).
The store keeps history, not only the latest value, because training needs to know what was true at every past moment. A data warehouse or columnar files (stored column by column, fast for large scans) are a typical home. The input is raw events; the output is an append-only table of timestamped values per entity. What you now know: the past is preserved, with dates.
Step 3: Join training sets as of the event time
To build a training set, each labelled example, such as "user 7 bought at day 4", needs the feature values that were known at that moment. A point-in-time join looks backwards from the example's timestamp and takes the latest value whose event time is at or before it. Feast's documentation describes this as scanning backwards from the entity timestamp, up to the feature view's TTL. By default it filters on event time only, so a back-fill, a value written later for an earlier event time, is still returned; an opt-in created-timestamp filter keeps such values out of training data (Feast documentation, as of October 2026).
The common mistake is the plain join on the user id that picks the newest row. In the toy test below, where values are redrawn daily, the shortcut reads a value written after the event for all 1,000 events.1 A model trained on the future looks excellent in evaluation and fails in production. What you now know: correctness in training is a question of time.
Step 4: Materialise the latest values into the online store
Materialisation is the scheduled or streaming copy of the latest value of each feature from the historical store to the online store. The online store is a key-value database keyed by entity id; Uber's post says features needed by online models are precomputed and stored in Cassandra, and reports a P95 latency below 10 milliseconds for models that fetch features from it (Uber Engineering, September 2017).
The job writes one row per entity holding all of that entity's features, which keeps one read per entity. Each write carries the event time so the serving layer can refuse values older than the TTL. The input is the history table; the output is a small, hot table of the newest values. Streaming features are written to the online store directly and to the history in the same step, so both stores receive the same value; if one write fails, the logs are used to repair it. What you now know: serving reads a copy built from the same data as training.
Step 5: Serve with batched reads, and log what you served
The serving application programming interface (API) receives an entity id or several, reads each entity's feature row in one call and returns the values to the model. Two habits keep it fast and honest. First, batch the reads: one call per entity, not one per feature. Second, log the exact features served with each prediction. Google's Rules of Machine Learning says the best way to make sure you train like you serve is to save the set of features used at serving time (Rules of ML, Rule 29, updated August 2025).
Logged features also let you check for skew by comparing them with what the historical store would have returned. The input is a request; the output is a feature vector and a log line. What you now know: the system serves quickly and records what it served.
Step-by-step description
- One definition per feature in a registry.
- Timestamped values land in the historical store.
- Training sets use a point-in-time join.
- Latest values are copied to the online store.
- An API reads and logs features at request time.
- A feature is defined once in a registry with its entity, its type and a time-to-live.
- Pipelines write timestamped values to the historical store.
- Materialisation copies the latest value of each feature to the online store.
- The serving API reads the online store and logs the features it served.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Two calculations: the join on a tiny example, then the sizing of the online store. Both use illustrative inputs, and the script is saved with the page.1
The join. A feature "orders in 7 days" is written for a user on day 1 (value 2), day 3 (value 4) and day 5 (value 9). The user buys on day 4. The correct training row uses 4, the value from day 3. The shortcut uses 9, the newest row, which was written a day after the purchase.1
Run on 1,000 random events across 200 toy entities and 30 days, with each value redrawn from 0 to 9 every day, the shortcut differed from the correct value for 912 events, 91.2%.1 With 10 equally likely values about 9 in 10 differ by construction; real features change more slowly, so the share is lower, but every shortcut row still comes from after the event.
Text description of the diagram
- The feature is written on days 1, 3 and 5 in the toy example.
- The correct training row uses 4, the value from day 3.
The sizing. Take a target of 50,000 requests per second. Assume each request reads 3 entities (user, item, merchant) with 30 features each at 8 bytes, plus 100 bytes of overhead per row: 340 bytes a row. With 100 million users, 10 million items and 1 million merchants there are 111 million rows, 37.7 GB, or 113.2 GB with three replicas (three full copies, so a node can fail).1
| Quantity | Value | How it is derived |
|---|---|---|
| Row size | 340 bytes | 30 × 8 + 100 |
| Rows | 111 million | 100M + 10M + 1M |
| Memory with 3 replicas | 113.2 GB | 111M × 340 bytes × 3 |
| Lookups per second, one per entity | 150,000 | 50,000 × 3 |
| Lookups per second, one per feature | 4,500,000 | 150,000 × 30 |
| Nodes at 100,000 lookups per second and 2x headroom | 3 batched, 90 per feature | assumed node capacity |
The latency target needs a budget too. A 50-millisecond p99 can be split into 8 for client and network, 7 for the gateway, 10 for the batched store read, 10 for transforms, 5 for decoding and 10 of slack: 50 in total.1 The 10 is the p99 of the whole 3-entity read, so each single lookup must finish within 10 milliseconds at about its 99.67th percentile.1
The fan-out, the number of parallel calls per request, matters: if lookups are independent and each is slow 1% of the time, a request that reads its 3 entities in one call each hits at least one slow lookup 2.97% of the time, and one that reads all 90 features one call each (3 × 30) does so 59.53% of the time.1
Text description of the diagram
- A request that reads its 3 entities in one call each hits at least one slow lookup 2.97% of the time.
- A request that reads all 90 features one call each hits at least one slow lookup 59.53% of the time.
Trade-offs and edge cases
A feature store adds a system to run, so the first trade-off is whether you need one. A single model with a few features computed inside the service does not. The need appears when several models share features or when training and serving code are written separately.
- Freshness against cost. Streaming features are fresh and expensive; daily batch features are cheap and stale. Decide per feature, not per system. See batch, streaming and real-time inference for the serving-mode choice.
- Consistency of the two stores. If materialisation lags, the online store serves old values. Alert on the age of the newest value per feature.
- Back-fills. As of October 2026 Feast's point-in-time join returns a corrected value written after the example's timestamp unless the created-timestamp filter is switched on, so turn it on when back-fills happen.
- Multi-tenancy. Give each team a namespace, quotas and per-tenant limits so one noisy tenant does not consume the read capacity of the rest.
- Hot keys. A very popular entity can overload one shard, the node that holds its slice of the data; replicate it or cache it in the service.
- Ownership. Features without owners rot; require an owner and a description, as Uber's registry does.
For where this sits in a wider design, see the machine learning system design overview and the data engineering hub.
How to answer it in an interview
How would you design a feature store that serves both training and online inference?
I would define each feature once in a registry, then write timestamped values from batch and stream pipelines into a historical store. Training sets come from a point-in-time join, so each example sees only values from before its timestamp. The latest values are materialised into a key-value online store, read with one batched call per entity, and I log the features served. I would size memory from rows times row size times replicas, and budget the p99 across network, store and transforms. The main risk is skew, so I would compare logged and recomputed features.
The spoken structure: registry first, then the two stores, then the join, then serving and sizing, then the risk.
Common mistakes
- "Join the latest feature value to each label." (wrong) That reads values from after the event. (right) "I use a point-in-time join on the event timestamp."
- "The online store is the source of truth." (wrong) It holds only the newest values, so it cannot rebuild a past training set. (right) "History lives in the historical store; the online store is a copy of the latest."
- "Compute features separately for training and serving." (wrong) Two codebases drift, which is training-serving skew. (right) "One definition feeds both paths."
- "Read each feature with its own call." (wrong) It multiplies lookups by the feature count and the tail risk by the fan-out. (right) "One read per entity, returning all its features."
Follow-up questions
What is a feature store in machine learning?
A feature store is a system that defines features once, stores their values with timestamps, and serves them for training and for low-latency prediction. It exists so both use the same logic, which prevents training-serving skew.
How do you populate the feature store and the online feature computation?
Batch jobs write historical values into the historical store and materialise the latest to the online store on a schedule; streaming jobs write fast-changing features to both at once. Uber describes batch precomputation for slow features and near-real-time computation for fast ones (Uber Engineering, September 2017).
How would you architect a feature store that serves both training and online inference?
Use a registry, a historical store with event and created timestamps, a point-in-time join for training, and an online key-value store fed by materialisation. Add a serving API that batches reads and logs features, so skew can be measured.
How do you design a multi-tenant feature store for 50,000 requests per second?
Size memory from rows, row size and replicas, then size throughput from lookups per second with headroom; in the worked example 3 nodes cover the batched case. Add per-tenant quotas, batch reads by entity and budget the p99 stage by stage.
What is training-serving skew and how does a feature store prevent it?
It is a gap between a model's performance in training and in serving, which Google lists as caused by mismatched data handling, changing data and feedback loops (Rules of ML, updated August 2025). A shared definition, a point-in-time join and logged serving features address the first, and the logs expose the others. Monitoring served features over time is covered in large language model (LLM) observability.
What changes at 500,000 requests per second?
Throughput, not memory: the data is still 113.2 GB, but batched reads rise to 1.5 million lookups a second and need 30 nodes at 2x headroom, against 900 with one read per feature.1 Shard by entity id and keep the same per-tenant quotas.
- Define each feature once; use it for training and serving.
- Keep timestamped history for training and the latest values for serving.
- Join training sets as of the event time, or the model learns from the future.
- Batch reads per entity; fan-out multiplies tail latency.
- Log the features you serve so skew can be measured.
Sources
- Point-in-time joins (Feast documentation) — Feast project (date = read date), 24 Oct 2026
- Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers, 25 Aug 2025
- Meet Michelangelo: Uber's Machine Learning Platform — Uber Engineering, 5 Sep 2017
- Hidden Technical Debt in Machine Learning Systems — NeurIPS 2015 (Sculley et al.), 7 Dec 2015
