How do you design an autonomous-vehicle perception stack?

In this article

1. Before you start
2. Why interviewers ask this
3. Autonomous-vehicle perception stack from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one design topic in the machine learning (ML) system design interview guide, and it assumes only the ideas below.

Why interviewers ask this

An autonomous vehicle perception system design interview question tests whether a candidate can design a real-time pipeline where a late answer is a wrong answer. Candidates report variants of it for self-driving, drone and robotics roles, such as designing a camera and lidar detection engine or a detection pipeline for a production vehicle.

A weak answer names one neural network and stops. A strong answer draws the whole chain, says what each stage consumes and produces, and states a delay budget. It also explains why several sensor types are combined and how the system is evaluated. A Staff-level answer adds the uncomfortable parts: rare situations dominate failures, sensors disagree, and a confident wrong box is worse than an honest "unsure".

Perception is the first link of a longer driving chain. Caesar et al. (2020) call robust detection and tracking of objects crucial for deploying autonomous vehicles, which is why interviewers treat it as the core of the design.

Autonomous-vehicle perception stack from first principles

Perception stack

A perception stack is the chain of software stages that converts raw sensor readings into a structured picture of the surroundings, meaning tracked objects with positions, sizes, classes and motion, plus forecast paths where prediction is included, that a planning module can use to decide how the vehicle drives.

Sensor fusion

Sensor fusion is the step that combines readings from different sensor types, such as cameras, radar and lidar, into one consistent description of the scene, so that the strengths of one sensor cover the weaknesses of another.

In plain words

Think of a person crossing a busy street. Eyes collect the scene, the brain spots a bus, links it to the bus seen a second ago, and guesses where it goes next. All of that must finish before the bus arrives. A perception stack does the same jobs in software. The analogy breaks in one place. A person can look again at will, while the stack must work on fixed snapshots from its sensors, and it must say how sure it is.

In one precise sentence, a perception stack maps synchronized multi-sensor frames to tracked 3D objects, with forecast paths where prediction is included, within a latency budget.

The reference sensor setup of the public nuScenes dataset has 6 cameras, 5 radars and 1 lidar, all with a 360 degree view. That is one dataset's rig, not a standard every vendor follows.

Step 1: Fix the delay budget and read synchronized sensors

The first design move is the delay budget, set before any model is chosen. The vehicle moves while the stack computes, so each stage's delay is distance traveled. At an illustrative 30 meters per second, a budget of 100 milliseconds means 3 meters, and a budget of 200 milliseconds means 6 meters.¹

Input: raw frames from each sensor. Output: one time-aligned bundle. Sensors run at different rates, so frames are matched by timestamp and corrected with calibration, the known position and angle of each sensor on the car. The public datasets reflect this: Waymo describes well synchronized and calibrated lidar and camera data, in scenes of 20 seconds each. A bad calibration shifts every box and no later stage can repair it.

End-to-end latency starts at capture, not when software reads the frame. The 100-millisecond budget in this page counts processing only. A spinning lidar that sweeps at an illustrative 10 hertz takes 100 milliseconds per sweep, and a camera at an illustrative 12 hertz delivers a frame that is up to about 83 milliseconds old, roughly 2.5 meters of travel at 30 meters per second.¹

Points within one sweep are captured at different moments, so the stack applies motion compensation: it shifts each point by the car's own movement during the sweep. For how a hard deadline reshapes any design, see what changes when a system must return in 100 milliseconds. You now have the clock and the input; the next step finds objects.

Step 2: Detect objects in each sensor

Detection is the stage that finds candidate objects in the sensor data and gives each a class, a position and a size. A camera detector works on pixels and is strong at classes, such as telling a pedestrian from a pole. A lidar detector works on 3D points and is strong at distance and shape. Radar adds radial speed, the speed along its line of sight, directly.

Input: one sensor's frame. Output: a list of boxes, each with a confidence score. Training data carries both kinds of labels: Waymo annotates 2D camera boxes and 3D lidar boxes, and nuScenes labels 3D boxes for 23 classes. A design answer names the output format and the class list, and keeps the confidence score, because later stages use it to weigh evidence. For detection basics, see the computer vision system design questions.

Step 3: Fuse the sensors into one scene

Sensor fusion merges what the sensors saw, and there are three common depths. Early fusion merges raw data. Late fusion merges each sensor's object lists. Fusion in a bird's-eye view (BEV) merges learned features in a shared top-down grid, where a bird's-eye view is a map of the surroundings seen from above, with the car at the center.

BEVFusion (2022) places camera and lidar features in one shared BEV space, which keeps the camera's semantic detail and the lidar's geometric structure. Late fusion is the easiest to debug. Late fusion tends to fail more gracefully, because each sensor's detector still works alone; deeper fusion can match it only if it is trained with sensors dropped or corrupted. Deeper fusion also keeps more detail but needs joint training. Input: per-sensor features or boxes. Output: one list of objects in vehicle coordinates. Say which depth you pick and why.

Text description of the diagram

• Early fusion merges raw data, late fusion merges object lists, and bird's-eye-view fusion merges learned features.
• Late fusion tends to fail more gracefully, because each sensor's detector still works alone.

Step 4: Track objects over time

Tracking is the stage that links the detections of one real object across consecutive frames, so that it keeps one identity and gains a velocity estimate. Without tracking, a pedestrian who is hidden by a van for two frames becomes a new, unknown object when they reappear.

Input: fused objects from the current frame and the tracks from earlier frames. Output: tracks, each with an identity, position, speed and an uncertainty. The usual loop matches new detections to existing tracks, updates the matched ones, starts tracks for unmatched detections and retires tracks that go unseen for a while. Matching is usually done with the Hungarian algorithm, which finds the lowest-total-cost pairing of detections to tracks, using a cost such as center distance or intersection over union (IoU), the overlap of two boxes divided by their combined area.

Each track keeps a Kalman filter, a predict-and-correct estimate of position and speed. Benchmarks score tracking separately: nuScenes defines the nuScenes detection score (NDS), built on mean average precision (mAP) with center-distance matching, and average multi-object tracking accuracy (AMOTA); Waymo defines heading-weighted average precision (APH), which credits a box only as far as its heading is right. A detector's box score alone does not describe tracking quality.

Step 5: Forecast paths and hand off to planning

Forecasting is the stage that estimates where each tracked object will be over the next few seconds. Its output is usually several candidate paths, each with a probability, because a cyclist at a junction may go straight or turn, and one averaged path would describe neither.

Input: tracks plus map and vehicle state. Output: a few weighted paths per object and an honest uncertainty. Many teams treat forecasting (behavior prediction) as a separate prediction module after perception; in an interview, say which boundary you assume. Reported interview prompts probe the training loss: a common choice pairs a best-of-K (winner-takes-all) regression, which updates only the candidate path closest to the truth, with a classification loss over the paths.

Typical metrics are minimum average displacement error (minADE), minimum final displacement error (minFDE) and miss rate, each taken on the best of K paths. The planner receives the tracks and the paths, never raw pixels. You now have the whole chain, and the delays of all five stages add up to the budget from step 1.

1 · Sensors and budget2 · Detect3 · Fuse4 · Track5 · ForecastReplay

Step-by-step description

1. Synchronized sensors start the delay clock.
2. Each sensor yields candidate objects.
3. Candidates merge into one scene.
4. Objects keep an identity over time.
5. Likely paths go to the planner.

• Cameras, radar and lidar are read in sync and calibrated.
• Detection finds objects in each sensor, and fusion merges them into one scene.
• Tracking links objects across frames, and forecasting gives their future paths to the planner.

Worked example

The budget below is an illustration, not a measurement of any real vehicle. It assumes 30 meters per second (108 kilometers per hour) and splits 100 milliseconds across five stages: 15 for synchronization and preprocessing, 40 for detection, 10 for tracking, 20 for forecasting and 15 as a safety margin. The distance is speed times time, so detection's 40 milliseconds is 1.2 meters, and all five stages together are 3 meters.¹

The same arithmetic shows why a budget is a design tool. If the total delay doubles to 200 milliseconds, the car travels 6 meters before the scene is ready; at 50 milliseconds it travels 1.5 meters.¹ An illustrative rig of 6 cameras at 1,600 by 900 pixels, 3 bytes per pixel and 10 frames per second produces 259.2 megabytes per second of raw images, which is why detection runs on a graphics processing unit (GPU) and why raw data is rarely moved between machines.¹

The last two rows use the scene counts of Waymo as published in 2020 (the dataset has been extended since) and nuScenes, and show how small the labeled public data is next to hours of driving.¹ A real-time fraud design splits its response-time allowance across stages with the same kind of table, though its cost of delay is a held payment, not distance.

Text description of the diagram

• Illustrative detection budget is 40 milliseconds, which is 1.2 meters of travel at 30 meters per second.
• The five illustrative stages total 100 milliseconds, which is 3 meters of travel.

Trade-offs and edge cases

Every choice in the stack trades delay, accuracy and reliability. The strongest design argument is to name the trade, then say what you would measure.

Sensor mix: cameras are rich in meaning, lidar is rich in geometry, and radar measures radial speed directly. Public benchmarks and the nuScenes reference rig combine cameras with range sensors; that describes one dataset's rig, and vendors differ. Fusion depth: late fusion tends to fail more gracefully because each sensor's detector still works alone, while deeper fusion keeps more detail but couples the sensors in training and matches that only if trained with sensors dropped or corrupted.

Data: a bigger dataset is not enough. Waymo's authors stress that generalization between operating regions is crucial and propose a diversity metric, so curate for rare weather, lighting and place, not only for volume. Evaluation: use detection and tracking metrics designed for 3D scenes, and slice results by condition, distance and class. Uncertainty: a detector that reports calibrated confidence lets the planner slow down when unsure.

When a sensor fails or a calibration drifts, the stack should flag it, which links to anomaly detection design. In the car, models run on a graphics processing unit (GPU), so teams quantize them (store weights in fewer bits) and profile latency stage by stage. Rare cases are mined from logs and fed back into training; see the autonomous vehicle (AV) data engine.

How to answer it in an interview

The spoken structure: budget first, then sensors, detection, fusion, tracking and forecasting, and end with evaluation and the rare cases.

Common mistakes

• "I would use one big camera network and that is the system." (wrong) It skips synchronization, tracking, forecasting and the delay budget. (right) "The network is one stage of a chain, and the chain has a budget."
• "A bigger dataset fixes generalization." (wrong) Waymo treats diversity across regions as the key issue. (right) "I would measure diversity and curate rare conditions."
• "Accuracy is just 2D mean average precision." (wrong) Mean average precision (mAP) is a score of how well predicted boxes match labeled boxes across confidence levels, and 2D boxes ignore depth and tracking. (right) "I would use 3D detection and tracking metrics, as in nuScenes."
• "Cameras and lidar always agree." (wrong) They disagree on range, occlusion and weather. (right) "Fusion must handle conflict and carry uncertainty forward."

Follow-up questions

How would you design a system for real-time object detection in autonomous vehicles?

Start from the delay budget, pick the detector that fits its detection slice, and run it on synchronized frames. Add tracking so detections keep identity, and measure with 3D metrics. The steps above give the full chain. For a camera and lidar engine in production, run a detector per sensor and fuse the results, as in step 3.

How would you design a machine learning solution for real-time object detection in drones?

The same chain applies with a smaller sensor set and tighter compute and power limits. The budget is set by drone speed and the compute on board, so choose a lighter detector and state the accuracy you give up.

Given trajectories, how would you build a model to predict future paths of cyclists?

Feed the cyclist's track, nearby objects and the map into a model that outputs several weighted paths, not one average. Train it on logged tracks, and calibrate the path probabilities so the planner can trust them.

How do you evaluate a perception stack?

Score detection and tracking with 3D metrics on held-out drives, then slice by weather, time of day, distance and class. A stack that is accurate on average can still fail on one rare slice.

Why not use cameras only, or lidar only?

Each sensor has blind spots, so combining them covers more cases, as the nuScenes rig does. A camera-only design can work in principle, but then the models must recover depth, and your answer should say how you would test that.

How is this different from other real-time ML designs?

A wrong output has a physical consequence, the deadline is hard because a late scene is a stale scene, and several sensors must be aligned in time before any model runs. A payments or recommendation service can time out and fall back; a vehicle cannot pause the world. The long tail of rare cases is handled in the AV data engine.

Sources

1. Scalability in Perception for Autonomous Driving: Waymo Open Dataset (arXiv 1912.04838) — arXiv (Sun et al., Waymo), 12 May 2020
2. nuScenes: A multimodal dataset for autonomous driving (arXiv 1903.11027) — arXiv (Caesar et al.), 5 May 2020
3. BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (arXiv 2205.13542) — arXiv (Liu et al.), 26 May 2022