How do you design a multimodal embedding model and the service around it?

In this article

1. Before you start
2. Why interviewers ask this
3. Designing a multimodal embedding model and its service from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the machine learning system design guide, which covers requirements, data, model, serving and evaluation in order.

Why interviewers ask this

The question tests whether you can connect a model to the system that makes it useful. Candidates often describe the training objective and stop; the interviewer wants the next part too: how a user's uploaded photo, video or document becomes a vector, where the vector is stored, and what happens when two uploads are the same. It appears in several shapes in interviews at AI labs and large technology companies: how to turn text, images and video into numbers, how to train a model on paired data, and how to build the ingestion service for user uploads.

A weak answer names a model and a vector database. A strong answer defines what a matching pair is, says how each data type is encoded, explains the loss in one sentence, gives a retrieval metric, and sizes the index. The same pipeline feeds search, recommendations and the cold start problem, where item vectors stand in for missing history.

Designing a multimodal embedding model and its service from first principles

In plain words

Multimodal embedding

A multimodal embedding is a vector that places items of different kinds, such as an image and a sentence, in one shared space, so that items that match in meaning end up close together and unrelated items end up far apart.

Contrastive loss

A contrastive loss is a training objective that rewards a model for giving a high similarity to each matching pair in a batch and a low similarity to every non-matching pairing of the same batch.

Imagine one big atlas where every photo, caption and video clip has a street address. A photo of a dog and the caption "a dog on a beach" are given neighbouring addresses. Search then means walking to an address and looking at the neighbours. The analogy breaks in two places. A real atlas has two dimensions, while these spaces have hundreds. And the addresses are learned from examples, so bad examples give a bad atlas.

In one precise sentence, an encoder is a neural network that turns one input into a vector, and the system trains one encoder per data type so that matching inputs get close vectors. The Contrastive Language-Image Pre-training (CLIP) paper trained its encoders on 400 million image-text pairs collected from the internet, with the task of predicting which caption goes with which image (Radford et al., 2021).

Step 1: Define the job and the matching pairs

The first step decides what counts as a match, because the model learns exactly that and nothing else. The input is the product need, such as "find videos from a text description"; the output is a list of paired data types and a rule for pairing them. Images pair with their captions, videos with transcripts or titles, and documents with their summaries or the queries that opened them.

Each rule has a failure mode. Captions scraped from the web are noisy, and click pairs are biased toward what was already shown. Write down a retrieval metric at the same time, such as recall at one: the share of queries whose true match is the nearest result. Decide also whether users must search one data type with another. If they will not, separate spaces are simpler, as the comparison below shows. What you now know: the pairs and the metric define the project.

Step 2: Turn each kind of data into numbers

The second step gives every data type its own encoder, and keeps fields that are not content out of the shared space. Text is split into tokens and passed through a transformer encoder; images are cut into patches and passed through a vision encoder; video is sampled into frames, each embedded like an image, with the audio turned into text by automatic speech recognition and embedded like text. Start from pre-trained encoders for each type, because training one from nothing needs far more pairs.

Fields such as age or username are not content to be matched, so do not push them through the contrastive model. Use them as metadata filters in the index, or as separate input columns to a downstream model, where a number is scaled and a category is looked up in a learned table. What you now know: content goes through encoders; metadata stays beside the vector.

Step 3: Train with a contrastive loss

The input is a batch of paired examples; the output is encoders whose vectors line up. In a batch of B pairs, compute the similarity of every image with every caption, giving a B by B table; the diagonal holds the true pairs. Divide each similarity by a temperature (0.1 below) that sharpens differences, and turn each row into probabilities with a softmax. The row loss is minus the natural logarithm of the probability on the diagonal entry, a cross-entropy; columns get the same. Separate encoders per side make a two-tower model.

A guessing model scores a loss of the natural logarithm of B. That is 1.39 for 4 pairs and 10.40 for 32,768, the batch size CLIP used.¹ Sigmoid Loss for Language Image Pre-training (SigLIP) replaces the softmax with a loss on each pair on its own. Its authors tried batches up to one million and found the benefits quickly diminish, with 32,000 sufficient (Zhai et al., 2023). What you now know: bigger batches help, with diminishing returns.

Step 4: Evaluate retrieval before shipping

The fourth step measures retrieval on pairs the model never saw. The input is a held-out set of pairs; the output is recall at k for each direction, text to image and image to text, and for each data type. Report both directions, because a model can be good at one and weak at the other. Slice by data type and by content group, since an average hides a weak modality.

In the toy example below, four illustrative pairs have an image-to-text recall at one of 0.5 before training, because two of the four images retrieve the wrong caption, and 1.0 after. Text-to-image recall at one rises from 0.75 to 1.0, and the loss falls from 1.16 to 0.02.¹ A real evaluation uses thousands of pairs. Keep a small hand-checked set of hard cases, such as near-duplicates and similar scenes, to catch regressions. What you now know: ship on measured retrieval, not on training loss.

Step 5: Build the service around the model

The fifth step wraps the model in a system that accepts uploads and answers searches. The input is a file; the output is a stored vector with metadata. The path is: an ingestion application programming interface (API) that takes an idempotency key so a retried upload does not run twice; validation of type and size; preprocessing such as resizing, frame sampling and speech recognition; batching on graphics processing unit (GPU) workers to keep the hardware busy; then storage of the vector, the metadata and a content hash.

For search, use an approximate nearest-neighbour index with metadata filters. The Hierarchical Navigable Small World (HNSW) method builds layered proximity graphs and searches from the top layer down, which its authors report scales logarithmically with the size of the data (Malkov and Yashunin, 2016). For very large sets, a GPU library built a k-nearest-neighbour graph over one billion vectors in under 12 hours on four GPUs (Johnson et al., 2017). What you now know: the service, not the model, decides cost and reliability.

1 · Pairs2 · Encode3 · Train4 · Evaluate5 · ServeReplay

Step-by-step description

1. Define what counts as a match.
2. Turn each data type into numbers.
3. Pull matches together, push others apart.
4. Measure retrieval on held-out pairs.
5. Ingest, batch, index and search.

• The pairs and the metric define the project.
• The service, not the model, decides cost and reliability.

Worked example

The example uses a batch of four image-caption pairs with invented cosine similarities, printed below so you can redo every number. Row i is image i, column j is caption j, and the bold diagonal holds the true pairs. The temperature is 0.1. For one row, divide each similarity by 0.1, take the exponential of each, and the loss is minus the natural logarithm of the diagonal entry's share of the row total.¹

For image 1 before training, the share is e^3.0 divided by the sum of e^3.0, e^2.5, e^2.8 and e^2.0, which is about 0.358, so its row loss is 1.03. After training the same share is about 0.99 and the loss is 0.01.¹

The loss for a guessing model in this batch would be 1.39, the natural logarithm of 4, so the before-training loss is only slightly better than chance.¹

Next, size the index. The vectors alone take items times dimensions times bytes per value; the graph links and metadata add to that, about 1.28 GB of neighbour ids for 10 million vectors with 16 links per vector.¹ For 10 million items, 512 dimensions in 32-bit floating point need 20.48 GB, and 16-bit values halve that to 10.24 GB. At 1,024 dimensions the figures are 40.96 GB and 20.48 GB.¹ At one billion items and 512 dimensions in 32-bit floats the vectors alone take 2.05 TB, so compress them with product quantization or split them across machines.¹

Text description of the diagram

• For 10 million items, 512 dimensions in 32-bit floating point need 20.48 GB, and 16-bit values halve that to 10.24 GB.
• At 1,024 dimensions the figures are 40.96 GB and 20.48 GB.

Video adds a choice. A 120-second clip sampled every 2 seconds gives 60 frames. With one 512-dimension 16-bit vector per frame, a clip takes 61,440 bytes, so a million clips need 61.44 GB. Pooling to one vector per clip needs 1,024 bytes per clip and 1.02 GB for a million, at the cost of losing which moment matched.¹

Trade-offs and edge cases

The design trades paired data and training cost for cross-modal search, and index size for recall.

• Shared space or separate spaces. A shared space lets text find images but needs many good pairs. If users only search within one type, separate encoders and indexes avoid that cost.
• Pre-trained or trained from scratch. Pre-trained encoders get you started fast but may miss your domain, such as product photos or medical images. Adapt them on your own pairs when the metric shows a gap.
• Per-frame or pooled video vectors. Per-frame vectors answer "when" and cost 60 times more in the example; pooled vectors are cheap and blur moments.
• Approximate search. The index may miss the true nearest vector; measure its recall against exact search on a sample, and re-rank the top results with a heavier model such as a cross-encoder reranker.
• Duplicates and re-embedding. Hash the content so identical files share one vector, catch near-copies such as resized images with a perceptual hash or a similarity threshold on the vector, and store the model version with each vector, because changing the model means re-embedding everything.

There are two ways to apply a metadata filter: after the vector search, which can return too few results, or before it, which needs index support. Test both on your own data. For text queries mixed with keyword needs, see hybrid search.

Text description of the diagram

• A shared space supports cross-modal search but needs paired data.
• Separate per-modality spaces need no pairs but cannot search one data type with another.

How to answer it in an interview

The spoken structure is: the pair and the metric, the encoders, the loss, then the service and its size.

Common mistakes

1. Embedding every field. (wrong) "I would embed age and username into the same space." (right) Keep them as metadata filters or separate model inputs.
2. No pair definition. (wrong) "I would train a multimodal model." (right) State what matches what and how noisy those pairs are.
3. Judging by training loss. (wrong) "The loss went down, so it works." (right) Report held-out recall at k in both directions.
4. Ignoring the service. (wrong) "Then I put the vectors in a vector database." (right) Cover ingestion, retries, batching, hashing and monitoring.
5. Forgetting the size. (wrong) "Storage is not a concern." (right) Compute items times dimensions times bytes, add the index graph, and choose precision accordingly.

Follow-up questions

What about user age and username?

Age and username are not content, so they do not belong in the shared content space. Use them as metadata filters in the index, or as extra columns for a ranking model, with numbers scaled and categories looked up in a learned table.

For text, do you use pre-trained models or something else?

Start with a pre-trained text encoder, because it already captures language, and adapt it on your own pairs if held-out recall shows a domain gap. Train from scratch only with very large domain data.

How do you generate those embeddings for the image and video?

Images go through a vision encoder in patches. Video is sampled into frames, each embedded like an image, then pooled or stored per frame, and speech is turned into text and embedded like text.

How do you convert unstructured non-numeric data to numeric?

Pass each item through an encoder that outputs a fixed-length vector, with pairs and a contrastive loss teaching it which items should be close. See what an embedding is.

What if the index grows to one billion vectors?

The vectors alone take 2.05 TB at 512 dimensions in 32-bit floats, so compress them with product quantization or shard the index across machines, and re-check recall against exact search after either change.

How do you avoid storing the same file twice?

Hash the file contents on ingest and look the hash up before embedding; identical files share one vector. A hash only catches byte-identical files, so add a perceptual hash or a similarity threshold on the vector to catch resized or re-encoded copies. Keep the model version with each vector so re-embedding stays traceable. See the computer vision interview questions for related image pipelines.

Sources

1. Learning Transferable Visual Models From Natural Language Supervision (CLIP) — arXiv (Radford et al.; v1 February 2021), 26 Feb 2021
2. Sigmoid Loss for Language Image Pre-Training (SigLIP) — arXiv (Zhai, Mustafa, Kolesnikov, Beyer; v1 March 2023), 27 Mar 2023
3. Billion-scale similarity search with GPUs — arXiv (Johnson, Douze, Jegou; February 2017), 28 Feb 2017
4. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — arXiv (Malkov, Yashunin; v1 March 2016), 30 Mar 2016