In this article
Before you start
This page belongs to the machine learning (ML) system design interview guide; the computer vision system design questions cover the perception side.
Why interviewers ask this
The data engine is the part of a driving system that decides what the model learns next, so a candidate who designs only the model has designed half the system. Candidates report questions of this family in interviews for autonomous driving and robotics roles: continuously retraining a pedestrian detector, selecting the most informative slice of driving logs for labeling, and validating a new planner against a very large body of past driving. Reports come from candidates, not from a company source.
A weak answer says "collect more data and retrain." A strong answer explains why raw volume is the wrong lever, how a small share of clips is chosen, who labels them, and how a release is checked. A Staff-level answer adds the failure mode of the loop itself: it only fixes what it can see.
The AV data engine from first principles
- Data engine
- A data engine is a loop around a deployed model that finds the cases where the model fails, gathers more examples like them, labels them, retrains the model and checks the result, so that the training set improves where the model is weakest rather than everywhere.
- Trigger
- A trigger is a rule that runs on the vehicle and marks a moment worth saving, such as low model confidence, a hard brake or a disagreement between two models, so only short clips around that moment are uploaded.
- Auto-labeling
- Auto-labeling is the use of a large model, run on recorded data after the drive with no real-time limit, to produce draft labels that humans only review and correct.
In plain words
Think of a student preparing for an exam. The student does not reread the whole textbook. They keep a list of the questions they got wrong. They find more questions like them and practise those. The analogy breaks in one place. The student knows which answers were wrong. A driving model often does not know it failed, so the system needs separate ways to notice.
In one precise sentence, the AV data engine spends a fixed labeling budget on the clips where the current model is most likely to be wrong, then proves the new model is better before it ships.
Step 1: Triggers flag failures on the vehicle
Triggers flag failures because the fleet produces far more data than anyone can upload or label. A trigger runs on the vehicle and saves a short clip when something looks wrong: the perception model is unsure, two models disagree, the driver takes over, or a hard brake happens.
Input: the live sensor stream. Output: a few short clips with metadata. A related idea is shadow mode, which means running a new model beside the production model or the driver and recording what it would have done without letting it control the vehicle. Disagreements between the two become trigger events.
Triggers are cheap and targeted, but they find only failures someone thought to write down. You now know how clips leave the car; the next step finds the ones no rule named. Because upload bandwidth is limited, on-vehicle compute matters, which the on-device ML design page covers. The perception models that raise these triggers are designed in the autonomous vehicle perception stack question.
Step 2: Mining finds rare cases that triggers cannot name
Mining finds rare cases by searching the retained archive of uploaded clips, through a lightweight index of embeddings, low-resolution frames and metadata, for clips that resemble a known failure. For a pedestrian partly hidden by a parked van, embed the clip and retrieve the nearest archive clips. Then push a new trigger or similarity query to the fleet, so vehicles upload more clips that resemble the seed failure.
One failure can become thousands of examples. Candidates also use active learning here: active learning is the family of methods in which the model helps choose which examples get labeled, as surveyed by Settles (2010). The simplest form, uncertainty sampling, ranks clips by how unsure the model is. For a two-class prediction, the entropy is 0.6931 nats (ln 2) at 50/50, 0.3251 nats at 90/10 and 0.056 nats at 99/1 (illustrative probabilities).1
Then deduplicate, because a thousand near-identical clips teach what ten would. Input: a failure example. Output: a ranked candidate set. The multimodal embedding design page explains how such a search service is built.
Step 3: Labeling mixes a large model with human review
Labeling is made affordable by auto-labeling. Qi et al. (2021) study offboard 3D object detection from point cloud sequences: a detector that sees the whole recorded sequence and has no real-time limit can use frames from the future as well as the past, which the on-vehicle model cannot.
Humans then review the draft labels and fix the hard ones instead of drawing every box. Audit a random sample of the final labels against careful human labels, which catches both reviewer and auto-labeler errors.
Input: the candidate clips. Output: labeled clips with a quality score. In the worked example below, review cuts labeling effort by a factor of ten (illustrative). You now have the data; the next step turns it into a model.
Step 4: Retraining mixes new cases with old data
Retraining mixes the newly labeled hard cases with a replay sample of older data so the model improves on the new failures without forgetting the old ones. For a pedestrian detector, the new clips might be occluded pedestrians at night.
Keep a fixed evaluation set per scenario type that never enters training, and add the new hard cases to the evaluation set only in a separate slice, so scores stay comparable over time. Split held-out hard cases by drive or scene, not by clip, so adjacent clips do not leak between training and evaluation. Input: labeled clips plus a replay sample. Output: a candidate model and per-slice scores. Retraining on a schedule or whenever a slice has enough new examples are both reasonable; say which one you choose and why.
Step 5: Validation gates the redeploy
Validation replays recorded driving through the candidate and compares it with the production model before any release. Waymax (Gulino et al., 2023) is a data-driven simulator for driving research built on logged scenarios, one example of this kind of replay.
Replay has a known limit: if the new planner chooses a different path than the logged driver, the other road users in the log do not react to it. Reactive simulated agents (Waymax supports them) and closed-loop simulation reduce this before the staged rollout. So report results per scenario slice, then roll out in stages. Input: the candidate model and a log archive. Output: a pass or fail per slice. A fix that helps one slice and hurts another is caught here. The validated model is redeployed and its failures restart the loop. The general lesson of metrics that looked good before release applies.
Step-by-step description
- Triggers flag failures on the vehicle.
- Search finds look-alike cases.
- Auto-labels plus human review.
- New cases mix with old data.
- Replay gates the release; the redeployed model's failures restart the loop.
- The loop has five steps.
- Validation gates every redeploy.
- The validated model is redeployed and its failures restart the loop.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
All figures here are illustrative assumptions, not measurements of any real fleet. Assume a fleet of 5,000 vehicles driving 4 hours a day, each recording 1 terabyte an hour, which is 20 petabytes of raw data a day. Cut into 20-second clips that is 3,600,000 clips a day.1
If triggers keep 0.1% of clips, 3,600 clips a day are uploaded; after deduplication and ranking keeps 25%, 900 clips need labels. Labeling one clip from scratch takes 30 minutes in this example and reviewing an auto-label takes 3, so the daily effort falls from 450 hours to 45.1
| Quantity | Value (illustrative) |
|---|---|
| Raw data per day | 20 petabytes |
| Clips per day | 3,600,000 |
| Uploaded by triggers (0.1%) | 3,600 |
| Kept after deduplication and ranking (25%) | 900 |
| Human labeling hours, from scratch | 450 |
| Human labeling hours, review only | 45 |
| Clips in an informative 1% of all clips | 36,000 |
Mining also adds candidates, both from the retained archive and from new fleet queries, so the 3,600 are not the only clips the engine can label. Selecting 1% of all driving, 36,000 clips a day, would need 1,800 review hours a day at 3 minutes each, so the selected share is a budget decision.1
For validation, 1,000,000 miles at an average of 25 miles per hour is 40,000 hours of driving; replayed by 1,000 workers each running 20 times faster than real time, that takes 2 hours.1 A million miles sounds hard but is a parallel batch job, and the real difficulty is choosing the scenario slices and trusting the replay.
Text description of the diagram
- Human labeling from scratch takes 450 hours a day in this example.
- Reviewing auto-labels takes 45 hours.
Trade-offs and edge cases
No single selection method is enough, so the engine combines them. Trigger rules are cheap but limited to known failures; embedding search finds look-alikes but needs a seed example; uncertainty sampling finds what the model doubts but misses what it is confidently wrong about; a random sample finds none of the rare cases but is the only unbiased control.
Text description of the diagram
- Trigger rules find only known failures.
- Random sampling is the only unbiased control.
Edge cases follow. Feedback bias: the loop improves only the failures it can observe, so keep a random slice to measure unseen errors. Label drift: auto-labelers change, so version labels with the model that wrote them. Storage: keep a searchable index (embeddings, low-resolution previews and metadata) of uploaded clips for a retention window, and keep raw sensor data long-term only for clips selected for labeling or evaluation; this matters for point clouds, which are large. Privacy: faces and plates in uploaded video need handling, as in the privacy-constrained ML design. Finally, rare-event detection resembles anomaly detection design, including its precision and recall trade-off.
How to answer it in an interview
How would you design the data engine for a self-driving fleet?
I'd treat it as a closed loop. The fleet runs the current model with triggers, such as low confidence or disagreement with the driver, and uploads short clips around those events. I add embedding search to mine rare cases that no trigger names, then deduplicate and rank by model uncertainty. Auto-labeling on the whole recorded clip, with humans reviewing, produces labels cheaply. I retrain with the new cases mixed into old data, then replay logged scenarios, sliced by scenario type, before a staged rollout. The main risk is feedback bias: the loop only fixes failures it can see, so I keep a random sample as a control.
The spoken structure: say the loop, then triggers and mining, then labeling cost, then validation, then name the feedback-bias risk.
Common mistakes
- "We should upload everything and label as much as we can." (wrong) Bandwidth and labeling budget are the limits. (right) "I select a small share of clips with triggers and mining, and spend labels where the model is weakest."
- "Shadow mode means the new model drives the car." (wrong) It does not control the vehicle. (right) "The new model runs beside the production model and I compare outputs."
- "Mining is just uncertainty sampling." (wrong) A model can be confidently wrong. (right) "I add embedding search and a random control sample."
- "If the replay passes, ship it." (wrong) Replayed traffic does not react to the new planner. (right) "I report per slice, then roll out in stages."
Follow-up questions
How do you build a pipeline that continuously retrains a pedestrian detection model from fleet data?
Run the five steps on a schedule: pedestrian-specific triggers, mining for look-alikes of past misses, auto-labels with human review, retraining with replay of old data, and a per-slice gate before release. Track recall on rare slices such as occlusion and night, not only the average.
How would you validate a new perception model against petabytes of driving logs?
Do not rerun everything: stratify the logs into scenario slices, run the candidate and the production model over each slice in a parallel batch, and compare per-slice metrics. Keep a fixed evaluation set so scores stay comparable across versions.
How would you validate a new motion planning version against 1 million miles of driving data?
Replay the logged scenarios in a simulator (Waymax is one public example), score safety and comfort metrics per slice, and flag scenarios where the planner diverges from the logged path. The worked example shows that the compute is a parallel job of a few hours under stated assumptions.
How do you select the most informative 1% of driving logs for labeling?
Combine signals: model uncertainty, trigger events, novelty in embedding space and diversity after deduplication, then rank and keep the top slice. Audit it against a random sample to check that you are not just collecting duplicates.
How do you mine interesting images from a corpus too large to label?
Embed every image, cluster or search the embeddings, and rank by distance from what the model has already seen so that novel images surface first. The harmful-content detection design uses the same pattern of mining before labeling.
How do you collect, store and sample multi-modal data for imitation learning?
Record each modality with timestamps; video and joint states arrive at different rates, so resample or align them to a shared clock. Store clips with a metadata index (task, scenario, outcome), and sample by slice so rare behaviours are not drowned out. Keep the raw files only for clips that were selected.
- The AV data engine is a loop: detect failures, mine similar cases, label, retrain, validate.
- Spend labels where the model is weakest, not on raw volume.
- Triggers find known failures; mining finds the ones nobody named.
- Auto-labeling on whole recordings, plus human review, cuts labeling effort.
- Validate per slice before release, and keep a random sample to catch what the loop cannot see.
Sources
- Offboard 3D Object Detection from Point Cloud Sequences (arXiv 2103.05073) — arXiv (Qi et al., Waymo), 8 Mar 2021
- Active Learning Literature Survey (Computer Sciences Technical Report 1648) — University of Wisconsin-Madison (Settles), 26 Jan 2010
- Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research (arXiv 2310.08710) — arXiv (Gulino et al., Waymo), 12 Oct 2023
