How do you design the data platform under ML?

In this article

1. Before you start
2. Why interviewers ask this
3. Design the data platform under ML from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of the machine learning operations (MLOps) interview questions learning path.

Why interviewers ask this

Much of the work in a production ML system is data plumbing. Candidates report system design prompts such as a reporting pipeline, a metrics service for creators, a training pipeline fed by billions of metric points a day, and a shared dataset that teams may query without seeing raw rows. Each prompt is the same question in different clothes: how does data get from a source to a model, and how do you trust it on the way?

A weak answer draws boxes labelled "Kafka, Spark, warehouse" and stops. A strong answer says what each stage promises, where raw data is kept, and how a bad day of data is repaired. A Staff-level answer adds the failure modes: skew between training and serving, label leakage from careless joins, late events, and the cost of freshness. The interviewer is listening for those promises, not the product names.

Design the data platform under ML from first principles

Data platform under ML

The data platform under ML is the set of storage, pipeline, validation, feature and monitoring systems that carries raw data from its sources to training sets and live predictions, so a model always learns from, and runs on, data of known quality and age.

Training-serving skew

Training-serving skew is a gap between the feature values a model saw in training and the values it sees in production, caused by computing one feature through two code paths or from different data. It lowers accuracy quietly, without raising any error.

In plain words

Think of a restaurant kitchen. Deliveries arrive and go into a store room exactly as received. Cooks wash and check the produce. Chefs turn it into prepared ingredients with fixed recipes. The line serves dishes and someone tastes them. The analogy breaks in one place: a kitchen cannot un-cook a dish, but a data platform can rebuild any prepared ingredient from the store room.

In one precise sentence, the platform is a chain of layers where each layer reads only the layer below it, so any layer can be rebuilt from the one beneath.

Four stages follow. Raw data is landed append-only and kept; clean data is validated and deduplicated; features come from one shared definition; serving reads the features and monitoring watches them. Each stage has an input, a change and an output.

Step 1: Land raw data in an append-only layer so you can reprocess it

The raw layer stores every source record in its original form and only ever adds to it. Input: events, files and application programming interface (API) pulls. Output: a growing, date-partitioned copy of what the sources sent. Databricks calls this the bronze layer of what it names the medallion architecture, a set of layers that denote data quality. Its documentation, last updated September 2026, says bronze keeps the raw state of the source, is appended incrementally, and enables reprocessing and auditing by retaining all historical data. It also advises minimal validation here and storing most fields as strings or binary to survive unexpected schema changes.

This answers the reprocessing probe. Because raw data is kept, "reprocess last month" means rerunning the later stages over thirty date partitions. Each run must be idempotent: it overwrites its own day's output instead of appending to it. You now know why nothing downstream is ever the source of truth.

Step 2: Validate and clean the data in a second layer

The clean layer turns raw records into trustworthy ones. Input: the raw layer. Output: one validated, deduplicated, typed record per real-world event. The Databricks page lists what happens at this silver layer: schema enforcement, handling of nulls and missing values, deduplication, resolution of out-of-order and late-arriving data, and data quality checks. It also says not to write to silver straight from ingestion, because schema changes and corrupt records in a source would then break the pipeline.

Two rules make this layer useful. Records that fail a check go to a quarantine table with the reason, so one bad source never blocks the day. And every event keeps its event time, the time it happened, separate from its processing time, the time the system saw it. Late events then land in the right day. You now have clean records; the next step gives them meaning for a model.

Step 3: Compute features once and join them at the right time

A feature is a number or category derived from clean records that a model reads as input, such as a customer's purchase count over a window (illustrative). The feature layer computes each feature from one shared definition used by both training and serving. Two definitions of one feature is how training-serving skew starts, and a model that is accurate in testing then degrades in production with no error to point at. The feature store design page goes deep on how to share definitions.

The second hazard is leakage. To build a training row for an event on a given day, join only feature values known on that day, which is a point-in-time join. A plain join to the latest feature value lets the future leak in, and the evaluation metric on held-out data looks better than production ever will. Output: versioned training sets and the values served live.

Step 4: Serve the features and monitor the data

The last stage hands data to two consumers. Training jobs read a snapshot of the training set, stored with a version so any model can be traced to its data. Live predictions read the newest feature values from a low-latency store. Input: the feature layer. Output: training sets and live feature values.

Monitoring watches the data, not only the servers. Useful checks are freshness (when did the newest record arrive), volume (did today's count jump or collapse), null rate, and distribution change against training. A shift in distribution is the early sign of model drift, and for language models the same idea reappears as large language model (LLM) observability. Automating retraining on top of this is the subject of Google Cloud's MLOps guide, which covers continuous integration (CI), continuous delivery (CD) and continuous training (CT) for ML systems. The jobs themselves usually run on shared compute; see Kubernetes for ML infrastructure.

1 · Land raw2 · Validate3 · Features4 · Serve and monitorReplay

Step-by-step description

1. Sources land in an append-only raw layer.
2. The clean layer validates and deduplicates.
3. Features use one shared definition.
4. Serving reads features, monitoring watches them.

• Raw data is landed append-only and kept.
• Clean data is validated and deduplicated.
• Features come from one shared definition.
• Serving reads the features and monitoring watches them.

Worked example

Sizing shows which part of the platform is hard. Two prompts reported by candidates give the inputs: an e-commerce app sending 2 terabytes of clickstream a day to be served to 500 concurrent users, and a metrics pipeline handling billions of points a day. The volumes are inputs; the peak factor, bytes per point, query rate and cluster read speed below are illustrative assumptions.¹

Spread over a day, 2 terabytes is an average of 23.15 megabytes per second. With an illustrative peak factor of 3, the design target is 69.44 megabytes per second.¹ For metrics, 1 billion points a day is 11,574 points per second, and 10 billion is 115,741.¹ At an illustrative 100 bytes a point that is 1.16 and 11.57 megabytes per second, so even ten billion points a day is a modest byte rate; the count of points, not the bytes, drives the stream design.¹

Serving 500 concurrent users is smaller still. If each user runs one query every 10 seconds (illustrative), the load is 50 queries per second.¹ The expensive part is the backfill. Thirty days of raw clickstream is 60 terabytes, and replaying it at an assumed 1 gigabyte per second takes 16.67 hours, which is 43.2 times faster than the data arrived.¹ A three-API daily ingestion with a month to reprocess is 90 partition runs, each idempotent.¹

A backfill rereads 2 terabytes for one day and 60 terabytes for thirty days, and the cost scales with the window.¹

Text description of the diagram

• A backfill rereads 2 terabytes for one day and 60 terabytes for thirty days.
• The cost scales with the window.

Trade-offs and edge cases

The main trade-off is freshness against cost. The Databricks page compares three ingestion styles. Continuous ingestion costs more and has lower latency. Triggered ingestion costs less and has higher latency. Batch ingestion costs less and has the highest latency. Pick the slowest option the feature can tolerate; the batch vs streaming comparison covers the decision in depth.

Text description of the diagram

• Continuous ingestion costs more and has lower latency.
• Triggered ingestion costs less and has higher latency.
• Batch ingestion costs less and has the highest latency.

Other cases where the textbook answer bends:

• Layers are a pattern, not a law. The same page calls the layered approach a recommended practice rather than a requirement. A small team may keep two layers; the rule that raw data is kept still matters.
• Two code paths. Running a batch path and a stream path for the same feature duplicates logic and invites skew. Prefer one definition executed in both modes, or one streaming path that can replay from the raw log.
• Late and duplicate events. Decide a lateness window, reprocess affected partitions when late data arrives, and deduplicate on a stable event id.
• Schema change upstream. A renamed field should land in raw untouched and fail loudly at the clean layer, not silently become a null.
• Sensitive data. Raw layers hold the most sensitive data, so restrict access to the raw layer to the pipeline itself and let people read views built on the later layers.

At very large scale, training itself becomes the bottleneck; see distributed training for large models.

How to answer it in an interview

The spoken structure: name the layers, say what each promises, state the freshness trade-off, then answer the reprocessing probe.

Common mistakes

• "Ingest straight into the clean tables." (wrong) A corrupt source record or a schema change breaks the pipeline. (right) "Land raw first, then validate into the clean layer."
• "I compute features separately for training and serving." (wrong) That creates training-serving skew. (right) "One shared feature definition feeds both."
• "Join the latest feature value to every training row." (wrong) Future information leaks in. (right) "Use a point-in-time join on the event timestamp."
• "Streaming everywhere, so everything is fresh." (wrong) It raises cost for features that do not need it. (right) "Match freshness to each feature's need."
• "Monitor the servers and we are done." (wrong) Data can degrade while every server is healthy. (right) "Monitor freshness, volume, nulls and distribution as well."

Follow-up questions

How would you reprocess last month's data?

Rerun the clean and feature stages over the thirty date partitions of raw data, each as an idempotent overwrite, and write to a new version before switching readers. Size the cluster for the reread volume, as the worked example shows.

How would you design a training pipeline that handles billions of metric data points per day with near-real-time feedback on model quality?

Stream points into the raw layer, aggregate by event-time windows in the clean layer, and join model outputs to later outcomes to compute quality metrics per window. At ten billion points a day the rate is 115,741 points per second (illustrative input, computed above), so aggregate early and retrain on a schedule rather than on every point.

How do you let several teams query a shared dataset without exposing the raw data?

Keep raw data readable only by the pipeline and publish governed views on the later layers that hide or aggregate sensitive columns and rows. Grant each team access to its view, and log queries for audit.

How would you design a reporting solution in AWS?

Use the same layers: object storage for raw data, scheduled jobs to clean and aggregate it, and a query engine over the aggregated tables for reports. On Amazon Web Services (AWS), as elsewhere, the shape and the promises matter more than the service names.

How would you design a service that shows metrics to content creators?

Precompute per-creator aggregates in the final layer by day and by hour, serve them from a low-latency store keyed by creator, and show how fresh each number is. Raw events stay in the raw layer so a counting bug can be fixed by a backfill.

How would you build a pipeline that ranks the most popular artists per country?

Deduplicate play events in the clean layer, aggregate counts by country and artist per day, and rank within each country in the final layer. Rank over a window, and rerun a day's partition if late plays arrive.

Sources

1. What is the medallion lakehouse architecture? — Databricks documentation (accessed 2026-10-26), 11 Sep 2026
2. MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center (retrieved 2026-10-26), 26 Oct 2026
3. Hidden Technical Debt in Machine Learning Systems — NeurIPS 2015 (Sculley et al.), 7 Dec 2015