How do you design visual search, step by step?

In this article

1. Before you start
2. Why interviewers ask this
3. Visual search from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the machine learning (ML) system design interview path. For the neighboring vision questions, see computer vision system design interview questions.

Why interviewers ask this

A candidate who answers "train a convolutional network and compare cosine similarities" is stopped by the first follow-up: how do you search a catalog of millions of vectors inside a latency budget? The question separates people who know one model from people who can build a whole retrieval system, with data, serving, and a way to prove it works.

Candidates report being asked to design visual search for an online marketplace and for a large image-sharing platform; these are candidate reports, not confirmed by a public source. A weak answer names a model and stops. A strong answer walks the chain from query to ranked results, gives a number for the index size, and names three layers of evaluation. A Staff-level answer adds the trade-off: where to spend the latency budget, and why a single shared embedding may beat several specialized ones.

Visual search from first principles

Object detection

Object detection is a computer-vision task that finds each object of interest in an image and returns a box around it with a label, so a system can search for the one product the user cares about instead of the whole photo and its background.

Approximate nearest-neighbor index

An approximate nearest-neighbor index is a data structure that stores many vectors and returns the ones closest to a query vector by checking only a small part of them. It trades a little recall for large savings in time, so a catalog of millions of vectors can be searched within a latency budget.

Cascade

A cascade is a ranking design in which a cheap first stage cuts a huge catalog down to a few hundred candidates and one or more heavier stages re-rank only those, so the expensive models see few items and the whole request stays within its latency target.

In plain words

Think of finding a lookalike lamp in a giant shop. You do not compare your lamp with every lamp. A clerk has already written a short description card for each lamp and filed the cards by similarity. You describe yours in the same card style. Then you open the nearest drawers and compare only a few cards.

The analogy breaks in one place. The cards here are lists of numbers learned from data, and a person cannot read them.

In one precise sentence, visual search maps every catalog image and the query image into one vector space, then returns the catalog vectors nearest to the query vector.

The field is the same one the literature calls content-based image retrieval. The Pinterest team of Jing et al. (2015) showed that a small engineering team can build and maintain a cost-effective, large-scale system with cloud computing and open-source tools. Visual search does not need a research lab.

Step 1: Detect the object, then embed it

Visual search starts by finding the object the user cares about, because a whole photo mixes the product with its background. In the figure of Jing et al. (2015), object detection localizes products such as bags and shoes, and the user clicks one object to see similar-looking products. Zhai et al. (2017) describe the use of object detection and interactive interfaces as deliberate design decisions, and list them among the rationales for the system.

Input: the query photo. Output: a crop of one object, then one vector for that crop from the embedding model, with a size such as 512 numbers (illustrative). Without the crop, a red shoe on a green lawn retrieves green lawns. You now know what gets embedded; the next step is how the embedding model learns to put similar items close together.

Step 2: Train the embedding so similar items land close

An embedding model for visual search is trained on pairs of images that should match and pairs that should not, with a loss that pulls matches together and pushes non-matches apart. Matching pairs can be photos of the same product, or items that users engaged with together.

A common layout is a two-tower model. A two-tower (bi-encoder) model is a retrieval layout in which the query and the item are embedded by separate encoders, often sharing weights, with no interaction between them, so item vectors can be precomputed. In image-to-image search the two towers usually share weights and act as one encoder. They differ when the query is a user photo and the item a clean catalog shot, when the query is text, or when the item tower also reads metadata.

In practice you start from a pretrained image network, either convolutional or a vision transformer, and fine-tune it on catalog data. Each embedding is scaled to length one (L2 normalization), so cosine similarity equals the dot product and the index can use dot-product search.

Pairs are not the only recipe: contrastive or triplet losses work on pairs or triples, while classification or proxy-based softmax losses treat each group of matching items as a class, and Zhai et al. (2019) train their unified embedding with multi-task metric learning. Hard negatives, wrong items that look much like the query, teach more than random ones. Labels come from groups of photos of the same product, from engagement such as saves and clicks on result pairs, and from human raters.

Pinterest also learned a lesson about how many embeddings to train. Zhai et al. (2019) report that one unified embedding improved relevance and engagement over specialized embeddings and cut the cost of maintaining several. Input: labeled pairs. Output: a model whose distances mean similarity.

Step 3: Build the index

The catalog vectors are computed once by a batch job and stored in an approximate nearest-neighbor index. Brute-force search compares the query with every vector. That cost grows linearly with the catalog and becomes impractical at billions of vectors, or on central processing units (CPUs) within a latency budget, so the index checks only a small part and accepts a small miss rate.

Common index families are an inverted file index (IVF), which searches only the nearest clusters; product quantization (PQ), which stores a short code per vector; and a hierarchical navigable small world graph (HNSW), which walks a graph of neighbors. Compression matters because memory sets the bill: the worked example below sizes it.

Input: all catalog vectors. Output: a searchable index. When the embedding model is retrained, every vector must be recomputed and the index rebuilt.

Step 4: Retrieve candidates

At request time only the query side runs: detect, embed, then ask the index for the top few hundred neighbors. This first stage is cheap and tuned for recall, because anything it misses can never be ranked later. The detected category can filter results, and so can stock status.

The scale can be large. Hu et al. (2018) report a Bing visual search system with tens of billions of images in the index and a response time under 200 milliseconds. They use a cascaded learning-to-rank framework to meet relevance, latency and scalability together; the retrieval, ranking and reranking funnel page explains that pattern.

Input: one query vector. Output: a short list of candidate ids. You now have candidates; the last step orders them.

Step 5: Re-rank and serve

A re-ranker is a heavier model that scores only the short list, using signals the first stage does not have: visual detail, category match, popularity, availability, and personalization for the user. It runs in real time, so its cost per item sets how many candidates it can afford.

Pinterest's Shop The Look paper (2020) covers object detection, visual embeddings, serving infrastructure for real-time inference and a data-labeling process for training and evaluation data. The cascade is why serving stays fast: the expensive model sees hundreds of items, not millions. The search ranking system page goes deeper on this stage.

1 · Detect and embed2 · Train the embedding3 · Build the index4 · Retrieve5 · Re-rank and serveReplay

Step-by-step description

1. The query is cropped to the object and embedded.
2. Matching and non-matching pairs train the model.
3. Catalog vectors go into an approximate index.
4. The index returns a few hundred candidates.
5. A heavier model orders the candidates.

• A detector crops the query image to the object.
• The embedding model is trained on matching pairs.
• Catalog vectors are stored in an approximate nearest-neighbor index.
• A re-ranker orders the candidates before serving.

Worked example

Assume a catalog of 10,000,000 images (illustrative) with a 512-number vector each (illustrative). Stored as 32-bit floats, the raw vectors take 20.48 GB. With 8-bit values they take 5.12 GB, and a 64-byte code per image takes 0.64 GB, which is 32 times smaller than the float version.¹ Brute-force search would cost 5,120,000,000 multiply-adds for one query, which is heavy on a CPU and grows linearly with the catalog, so an index is used.¹ These sizes count only the vectors; the graph or inverted-file structure of an index adds overhead.

Text description of the diagram

• Raw vectors as 32-bit floats take 20.48 GB.
• Raw vectors with 8-bit values take 5.12 GB.
• A 64-byte code per image takes 0.64 GB.

For latency, Hu et al. (2018) report under 200 milliseconds end to end for Bing. The split of that 200 milliseconds across the five stages in the table is illustrative, and it adds up exactly.¹ If peak traffic is 3,000 queries per second (QPS) (illustrative) and one replica serves 400 (illustrative), 3,000 divided by 400 is 7.5, which rounds up to 8 replicas.¹ Eight replicas leave little headroom, so a real plan adds a target utilization below full load or one spare replica.

Now the metrics. Take a toy gallery with five relevant items and a ranked list of ten results where results 1, 3, 4 and 7 are relevant (illustrative). Precision at K is the share of the top K results that are relevant.

All four rows come from the toy list.¹ The fifth relevant item never appears, so recall stops at 0.8, and only a better first stage can fix that.

Trade-offs and edge cases

Visual search trades recall, memory and latency against each other. A smaller code makes the index cheaper and faster but loses recall; a bigger candidate list helps recall but eats the re-ranking budget. The right setting is the one the online results support, not the one with the best held-out number.

Edge cases that matter in a round:

• Detection misses. If the detector misses the product or crops the wrong one, the embedding describes the wrong thing. Fall back to the whole image.
• Stale index. A retrained embedding model makes old vectors incomparable, so every vector is recomputed and the index rebuilt, then switched over together.
• New items. Items listed after the last batch need an incremental path into the index, or they are not searchable.
• One embedding or several. Zhai et al. (2019) found one unified embedding better for Pinterest's visual search products, for both browsing and searching. Whether that holds in another catalog is not shown there, so treat it as a strong default, not a law.

The 2025–26 default (as of October 2026). A common starting point is a pretrained vision foundation model instead of a backbone trained from scratch, then fine-tuning it on catalog pairs. Radford et al. (2021) introduced CLIP (Contrastive Language-Image Pre-training), which trains an image encoder and a text encoder together so that a text query can search images. Oquab et al. (2023) present DINOv2, which learns general-purpose image features without labels. For the index, Johnson et al. (2017) describe a graphics processing unit (GPU) similarity-search approach that handles billions of vectors and that they open-sourced (the library later known as Faiss); product quantization, one of the index families named in Step 3, is among the methods the paper covers.

Evaluation: three layers

Evaluate in layers: metrics on a held-out set, human relevance judgments, then an online A/B test. An A/B test is an experiment that shows two versions of a system to different random groups of users and compares a behavior metric. Zhai et al. (2019) used evaluations on held-out data, user studies and online A/B experiments. Pinterest's Shop The Look paper measured system choices with held-out evaluations, human relevance judgments and online A/B experiments. Jing et al. (2015) ran live experiments showing that recommendations powered by visual search improved engagement.

Recall at K alone is not enough, as metrics that looked great before an A/B test explains.

Text description of the diagram

• Evaluate in layers: metrics on a held-out set.
• Human judgments say whether results look similar.
• An online A/B test compares a behavior metric.

How to answer it in an interview

The spoken structure: clarify, then detect and embed, then index and retrieve, then re-rank, then evaluate in layers, then name one trade-off.

Common mistakes

• "I embed the whole photo and search." (wrong) The background pollutes the vector. (right) "I detect the object first and embed the crop, as Pinterest's systems do (Jing et al., 2015)."
• "Retrieval is one model call." (wrong) One heavy model over the whole catalog misses the latency budget. (right) "A cheap first stage retrieves candidates and a heavier stage re-ranks them, as a cascade."
• "Recall at K on a test set proves it works." (wrong) It cannot show whether people engage more. (right) "I add human judgments and an A/B test."
• "One embedding per use case is best." (wrong) It multiplies maintenance. (right) "I start from one shared embedding and split only where a metric demands it."

Follow-up questions

How would you build a visual search system for an online marketplace?

Use the same chain, with the catalog's quirks in mind: seller photos vary in quality, so the detector and the embedding must handle that variety. Filter by category and stock status after retrieval, and judge success by recall at K, latency and engagement.

Should you pick a vision transformer or a two-tower model?

A vision transformer and a two-tower model are not alternatives, because one is a network and the other is a layout. A vision transformer (ViT) is an image network that cuts an image into patches and processes them with attention, and it can be the encoder inside a two-tower model, so choose the backbone by accuracy and cost. A model that reads query and item together cannot be precomputed, so keep it for the re-ranking stage.

How do you design an image search system where the query is text?

The pipeline is the same, but the query encoder takes text and must land in the same vector space as the images, as in CLIP, which is trained on image and text pairs. The multimodal embedding model page covers how that shared space is trained and served.

What business metrics matter besides recall at K?

A marketplace tracks the click-through rate (CTR) on results, conversion from visual-search sessions, the zero-result or no-click rate, and p99 latency (the latency the slowest one percent of requests exceed) as a guardrail. Jing et al. (2015) used engagement in live experiments to show their system's value. Personalization enters the re-ranker as user features, and an A/B test of personalized against non-personalized ranking on these metrics measures it.

How does visual search differ from a search ranking system?

The query is an image, so the first stage retrieves by vector distance, not by text match. The re-ranking and evaluation ideas carry over from search ranking system design. A similar staged pattern appears in typeahead design, and moderating uploads uses a related stack in harmful-content detection.

Sources

1. Visual Search at Pinterest (arXiv 1505.07647) — arXiv (Jing et al., KDD 2015), 28 May 2015
2. Visual Discovery at Pinterest (arXiv 1702.04680) — arXiv (Zhai et al., WWW 2017), 15 Feb 2017
3. Learning a Unified Embedding for Visual Search at Pinterest (arXiv 1908.01707) — arXiv (Zhai et al., KDD 2019), 5 Aug 2019
4. Web-Scale Responsive Visual Search at Bing (arXiv 1802.04914) — arXiv (Hu et al., KDD 2018), 14 Feb 2018
5. Shop The Look: Building a Large Scale Visual Shopping System at Pinterest (arXiv 2006.10866) — arXiv (Pinterest, KDD 2020), 18 Jun 2020
6. Learning Transferable Visual Models From Natural Language Supervision (arXiv 2103.00020) — arXiv (Radford et al., CLIP), 26 Feb 2021
7. DINOv2: Learning Robust Visual Features without Supervision (arXiv 2304.07193) — arXiv (Oquab et al.), 14 Apr 2023
8. Billion-scale similarity search with GPUs (arXiv 1702.08734) — arXiv (Johnson, Douze, Jegou), 28 Feb 2017