In this article
Framing the problem
What do you ask first in a computer vision system design interview?
Ask what decision the output feeds, then what the system must produce: a label, boxes, a count or a mask. Next ask how much delay is allowed, which error costs more (a missed item or a false alarm), and what the images look like in deployment: camera angle, lighting, resolution and how many cameras. Write these as numbers before choosing a model, because a bin sorter and a traffic counter tolerate very different errors.
Why it is asked: it shows you start from the decision, not the model. Read the full explanation: machine learning system design guide.
How do you choose between classification, detection and segmentation?
Classification gives labels for the whole image (one, or several in multi-label classification), detection gives a box and a label for each object, and segmentation labels every pixel. Choose the cheapest one that answers the decision. "Is this a kitchen photo" is classification, counting cars needs detection, and measuring the area of a surface needs segmentation. Labels get more expensive at each step, so the simplest task that works also costs least to build.
Why it is asked: it tests whether you match the task to the decision and to the labelling budget. Read the full explanation: deep learning guide.
How do you measure a detector?
Match each predicted box to a true box with intersection over union (IoU), the overlap area divided by the area covered by either box (the union). In the computed example, a 100 by 80 pixel true box and a shifted prediction overlap by 5,600 pixels out of 10,400 combined, an IoU of 0.54.1 A detection counts as correct above a chosen IoU threshold, and average precision (one score summarising the precision and recall trade-off) then covers all score thresholds. Add a product metric, such as count error per hour.
Why it is asked: it separates model metrics from business metrics. Read the full explanation: confusion matrix.
Data and labels
How would you get labelled training data?
Start from frames of the real deployment cameras. Label a small seed set by hand, train a first model, then let it pre-label new frames and have people correct the output, spending effort on frames where the model is unsure. Public sets help with pre-training: the Common Objects in Context (COCO) dataset holds 328,000 images with 2.5 million labelled instances of 91 object types (Lin et al., 2014), but its pictures do not match your camera.
Why it is asked: labelling is usually the largest cost and schedule risk. Read the full explanation: machine learning operations (MLOps) guide.
How do you handle rare classes and the long tail?
Rare cases, such as an unusual vehicle type, are where overall accuracy hides failure. Find more of them by searching for low-confidence frames, give them extra weight or oversample them in training, and report recall per class instead of one average. Decide in advance what the system does with a low-confidence item: ignore it, count it as unknown, or send it to a person.
Why it is asked: it tests whether you look beyond the average metric. Read the full explanation: confusion matrix.
How do you check that training images match what the camera sees in production?
Compare simple statistics between a sample of production frames and the training set: brightness, resolution, class mix and the model's confidence scores. A shift in any of them is a warning. Review a small random sample of production frames by hand each week, since new weather, a moved camera or a new lens changes the input without any error message.
Why it is asked: models fail quietly when the input changes. Read the full explanation: distribution shift.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Models and tasks
Which model would you use for real-time object detection?
Use a one-stage detector, which finds boxes and classes in a single pass over the image, and choose by measured speed and accuracy on your own device and footage. The original You Only Look Once (YOLO) paper reported 45 frames per second for its base model and 155 for a smaller one on its test hardware (Redmon et al., 2015); newer models exist, so benchmark before committing. Pre-train, then adapt to your classes.
Why it is asked: it tests speed versus accuracy reasoning with evidence. Read the full explanation: deep learning guide.
How would you tag images uploaded by hosts as kitchen, bedroom or pool?
This is multi-label image classification, because one photo can show several things. Start with a model that scores an image against text such as "a photo of a kitchen". Contrastive Language-Image Pre-training (CLIP) was trained on 400 million image-text pairs (Radford et al., 2021) and can do this without task-specific labels. Check the tags on a labelled sample, adapt the model on the errors, store a confidence with each tag, and send low-confidence photos to review.
Why it is asked: it tests whether you try a pre-trained model before collecting labels. Read the full explanation: multimodal embedding systems.
How would you design a system that sorts waste by detecting paper?
Place a camera above the conveyor, classify or detect each item, and fire an air jet or arm when the item reaches it. The time budget comes from geometry: with an illustrative belt speed of 0.5 metres a second and 0.4 metres between camera and jet, an item arrives 800 milliseconds after it is seen, and after 100 for capture and 200 for the actuator, 500 remain for inference and decision.1 Add other sensors if paper and plastic look alike.
Why it is asked: it forces a latency budget from physical distances. Read the full explanation: machine learning system design guide.
How would you find videos similar to a given scene in a very large library?
Turn frames or short clips into embedding vectors (lists of numbers that capture meaning) with an image-text model, store them in a vector index, and search with an example clip or a text description such as "rainy road with a collision". Re-rank the top results with metadata like time, place and sensor, and verify with people. Sample frames at a low rate so the index stays affordable.
Why it is asked: it tests retrieval design for non-text data. Read the full explanation: similarity search.
Video and counting
How would you design a real-time vehicle counter from closed-circuit television (CCTV) footage?
Decode frames, detect vehicles, track each one across frames, count a vehicle once when its track crosses a virtual line, and store counts by class and minute. Simple Online and Realtime Tracking (SORT) is a lightweight tracker built from a Kalman filter (predicts where a box moves next) and the Hungarian algorithm (pairs boxes with tracks), and its authors report the tracker alone updates about 260 times a second (260 Hz), detection not included (Bewley et al., 2016). Keep the counting logic separate from the detector so either can change.
Why it is asked: open-ended, it shows how you steer from a vague brief to a pipeline. Read the full explanation: batch, streaming and real-time inference.
Text description of the diagram
- A counter detects vehicles, tracks them across frames, counts each track once when it crosses a line, and stores counts by class and minute.
- Counting once per track at a line keeps one vehicle from being counted in every frame.
How would you count vehicles by type, cars and bikes separately?
The detector outputs a class for every box, but the class can flicker between frames, for example car and van. Give each track one class by a majority vote or an average of its scores over its life, then count the track once when it crosses the line. Report per-class counts with their error, since small classes such as bikes are the hardest to detect.
Why it is asked: it tests that you smooth noisy frame-level output into one decision. Read the full explanation: confusion matrix.
How do you make sure the same vehicle is not counted twice?
Count each track once, at a line, not each detection. In a toy run with 12 true vehicles and 20 percent of detections missed, counting every detection gave 1,016, unique track IDs gave 21 because tracks broke after missed detections, and counting each track once at a line gave 11.1 The ByteTrack authors say discarding low-score boxes causes missed objects and fragmented trajectories, and that keeping them improves identity scores (Zhang et al., 2021).
Why it is asked: duplicates are the classic counting failure. Read the full explanation: anomaly detection system design.
Text description of the diagram
- In the toy run there are 12 true vehicles, counting each track once at a line gives 11, and unique track IDs give 21.
- Counting every detection gives 1,016.
How much hardware do you need for sixteen cameras?
Multiply cameras by frame rate, then divide by detector speed. Sixteen cameras at 30 frames per second produce 480 frames a second, which at an assumed 45 frames per second per detector (one detector means one model instance on its own accelerator) needs 11 detectors. Processing every fifth frame and letting the tracker bridge the gaps needs 3.1 Measure real speed on your hardware, because the assumed figure is only the base YOLO speed on the authors' machine.
Why it is asked: it tests back-of-envelope sizing. Read the full explanation: machine learning system design guide.
Text description of the diagram
- Sixteen cameras at 30 frames per second need 11 detectors at an assumed 45 frames per second each when every frame is processed.
- Processing every fifth frame needs 3 detectors and every tenth frame needs 2.
Serving and quality
Should the model run at the edge or in the cloud?
Compare the data you would send with the value of sending it. One uncompressed 1080p colour stream at 30 frames per second is about 186.6 megabytes a second, roughly 1.5 gigabits.1 Compression shrinks that but loses detail. Running the detector next to the camera sends only counts and a few clips, trading hardware cost and updates at many sites for bandwidth savings and less video leaving the site.
Why it is asked: it checks that you size the data path, not only the model. Read the full explanation: batch, streaming and real-time inference.
How do you monitor a deployed vision model?
Watch the inputs, the outputs and the outcomes. Inputs: brightness, resolution and frame rate. Outputs: the distribution of confidence scores and per-class counts. Outcomes: a weekly human review of a random sample against the model's answer. Alert when counts move in a way the world cannot explain, such as a count of zero at rush hour, which usually means a camera or pipeline fault.
Why it is asked: it tests whether you plan for silent failure. Read the full explanation: anomaly detection system design.
What changes if the camera moves, as on a car or a drone?
A tracker that matches boxes in image coordinates assumes a still camera. When the camera moves, every object appears to move, so you need to compensate for the camera's own motion or track in a shared world frame using the vehicle's position and sensors. Add depth from a second camera or a lidar (a laser distance sensor) when the decision needs distances.
Why it is asked: it separates fixed-camera answers from moving-platform answers. Read the full explanation: machine learning system design guide.
How do you release a new model without breaking the counts?
Run the new model beside the old one on the same live frames without acting on it, compare counts and per-class recall against a labelled sample, and switch a small share of cameras first. Keep the old model ready to restore. Version the model together with its thresholds and tracker settings, since a change in either moves the counts.
Why it is asked: it tests safe rollout for a system people rely on. Read the full explanation: MLOps guide.
For the whole system design process, see the machine learning system design guide.
Follow-up questions
How do you prepare for a computer vision system design interview?
Practise one pipeline end to end, such as the vehicle counter, and say the numbers aloud: cameras, frame rate, delay budget, detector speed and error cost. Then change one requirement, such as a moving camera, and redo the design. Prepare one example each for data, tracking and monitoring.
Should you start with the model or the data?
Start with the decision and the data. A model choice follows from the image type, the delay budget and the labels you can afford. Many designs improve more from better labels on the real camera than from a larger model.
What changes if you need 100 cameras instead of 16?
Redo the same sum with the new number. One hundred cameras at 30 frames per second produce 3,000 frames a second, which at the assumed 45 frames per second per detector needs 67 detectors on every frame, or 14 when only every fifth frame is processed.1 The cost grows in proportion to cameras, so say which lever you pull first: skip frames, use a lighter model or share accelerators. The machine learning system design guide covers how to defend that choice.
What if the counts suddenly drop to zero overnight?
Treat it as a fault until proven otherwise: check the camera feed, the frame rate and the confidence scores before blaming traffic. A real change moves neighbouring cameras too, and a fault usually moves one. The anomaly detection system design guide shows how to alert on that pattern without flooding people with false alarms.
- Start from the decision, then fix the task (classification, detection or segmentation), the delay and the cost of each error.
- Label real deployment frames first; pre-trained models and public datasets speed things up but do not match your camera.
- Count each track once at a line: in the toy run, unique track IDs gave 21 for 12 vehicles.
- Size hardware by cameras times frame rate over detector speed, and skip frames with a tracker to cut it.
- Monitor inputs, outputs and a weekly human-reviewed sample, because vision models fail quietly.
Sources
- You Only Look Once: Unified, Real-Time Object Detection — arXiv (Redmon, Divvala, Girshick, Farhadi; v1 June 2015), 8 Jun 2015
- Simple Online and Realtime Tracking (SORT) — arXiv (Bewley, Ge, Ott, Ramos, Upcroft; February 2016), 2 Feb 2016
- ByteTrack: Multi-Object Tracking by Associating Every Detection Box — arXiv (Zhang et al.; v1 October 2021), 13 Oct 2021
- Microsoft COCO: Common Objects in Context — arXiv (Lin et al.; v1 May 2014), 1 May 2014
- Learning Transferable Visual Models From Natural Language Supervision (CLIP) — arXiv (Radford et al.; v1 February 2021), 26 Feb 2021
