What is multi modality in AI, and how does CLIP learn it?

In this article

1. Before you start
2. Why interviewers ask this
3. Multi modality from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

The how large language models are trained overview gives the wider picture; this page assumes the transformer idea.

Why interviewers ask this

Machine learning interviews use multi modality to test whether a candidate can explain how text and pictures end up in the same mathematics. The consequence of a vague answer is a follow-up you cannot survive: what exactly is compared, and what is the loss?

A weak answer says "the model understands images and text together." A strong answer names the two encoders, the shared space, the similarity grid and the loss that pulls true pairs together and pushes the rest apart. A Staff-level answer adds what the design cannot do: it matches, it does not write. It also covers what it costs, since large paired datasets are often not public and training needs heavy compute.

Multi modality from first principles

Modality

A modality is a kind of input data that a system takes in, such as text, an image or audio, each with its own format and its own way of being turned into numbers before a model can use it.

Shared embedding space

A shared embedding space is a single vector space into which separate encoders map different modalities, so that a text vector and an image vector can be compared directly and matching items land close together.

Zero-shot transfer

Zero-shot transfer is the use of a pretrained model on a new task without any labelled examples of that task, with the task described in words instead.

In plain words

Picture two translators who each read a different language, one for photos and one for sentences. Both must write their summary on the same ruler, a position between zero and one hundred. A photo of a dog and the sentence "a dog on grass" should get nearby positions. The analogy breaks in two ways. The real "ruler" has hundreds of dimensions, not one. The translators also learn their habits together, from examples, not from a rulebook.

In one precise sentence: CLIP learns an image encoder and a text encoder so that the cosine similarity of a true image-caption pair is higher than that of any mismatched pair. This page covers that matching design in depth; systems that generate text from images are a different family, mapped in the section on other kinds of multimodal models below.

Step 1: Two encoders turn each modality into one vector

CLIP has two encoders, an image encoder and a text encoder, and a linear projection maps the output of each into a shared embedding space (Figure 3 of the paper). The text side is a transformer. In the DisCo-CLIP setup the image side is a Vision Transformer (ViT), images are resized to 224 by 224 pixels, and text is capped at 76 tokens; the CLIP paper also trained ResNet image encoders.

A ViT cuts the image into square pieces. A patch token is a vector that stands for one square piece of the image. ViT-B/32 uses 32-pixel patches, because the number after the slash is the patch size, so a 224-pixel image gives 7 by 7, so 49 patch tokens, plus one extra class token.¹

Input: raw image or text. Output: one vector each. You now know the model's shape: two encoders, no contact between them yet.

Step 2: Cosine similarity compares every image with every caption

The two vectors meet in one operation. For a batch of N image-text pairs, the system computes the cosine similarity of every image against every caption, which gives an N by N grid. The CLIP paper scores all pairings in the batch this way, and a survey chapter describes the same step as computing the similarity for all possible pairs in a minibatch.

The diagonal cells are the true pairs, the positives. Every other cell is a mismatched pair, a negative. With four pairs that is 16 cells, 4 positives and 12 negatives; with a batch of 1,024 it is 1,048,576 cells, 1,024 positives and 1,047,552 negatives.¹

Input: N image vectors and N text vectors. Output: an N by N grid of scores. Nothing was labelled by hand: the supervision is the natural pairing of images and captions, which CLIP collected as 400 million web pairs.

Step 3: A symmetric contrastive loss trains both encoders

Training turns the grid into a loss. A symmetric cross-entropy is applied over the rows and the columns of the similarity grid (Figure 3 pseudocode). Reading a row asks: for image $i$, which of the N captions is right? With $s_{ij}$ the cosine similarity of image $i$ and caption $j$, the row loss is:

A column reading swaps the roles: for this caption, which image is right? The total averages the row and column versions. The softmax inside turns scores into probabilities that sum to one. Dividing the scores by a number called temperature, written $\tau$, sets how sharply close competitors are punished. CLIP learns the temperature as a parameter, initialised to 0.07, with the logits clipped at a scale of 100 (§2.5); the 0.5 used below is chosen for readability.

The two readings can differ: one direction can be easy while the other is hard, as pair 2 of the worked example shows. Averaging both trains both encoders from the same grid. You now know where the learning signal comes from: the off-diagonal cells.

Step 4: Language becomes the interface for zero-shot use

After pretraining, natural language references learned visual concepts, which enables zero-shot transfer to downstream tasks. Class names written as text enable zero-shot use. To classify an image into classes such as "dog" or "cat", you encode each class name as a short text, encode the image, and pick the text with the highest cosine similarity. No labelled examples of the new task are needed.

Zero-shot does not mean untrained. The model was pretrained on the 400 million pairs; only the new task needs no labels. The same shared space also allows retrieval, finding images for a text query or the reverse. It does not generate anything: the output is a score.

1 · Two encoders2 · Similarity grid3 · Contrastive loss4 · Zero-shot useReplay

Step-by-step description

1. Each modality becomes one vector.
2. Every image meets every caption.
3. Rows and columns train both encoders.
4. Class names in text act as labels.

• An image encoder and a text encoder each produce one vector.
• A grid of cosine similarities compares every image with every caption in a batch.
• A contrastive loss over rows and columns trains both encoders.
• Class names written as text enable zero-shot use.

What other kinds of multimodal models are there?

Three common families are these. The first is the dual encoder matched by a contrastive loss, which is CLIP. The second fuses the modalities inside a language model: Flamingo (Alayrac et al., 2022) feeds visual features into a language model through cross-attention layers, and a survey chapter notes that it uses a CLIP-like contrastive vision encoder. The third attaches a vision encoder to a language model through a projector that turns image features into tokens the language model reads, as in LLaVA (Liu et al., 2023). The last two families write text; CLIP only scores.

A later variant of the first family is SigLIP (Zhai et al., 2023), which replaces the batch-wide softmax with a sigmoid loss on each image-text pair. It removes the normalisation over the whole batch, so it scales batch size more cheaply.

Worked example

Four image-caption pairs show the loss by hand. Each image and caption is a toy two-number vector, so the numbers are illustrative; the cosine grid, the losses and the counts are computed.¹

At temperature 0.5 (illustrative), reading by rows gives per-pair losses of 0.3159, 0.3914, 0.3823 and 0.2029, a mean of 0.3231. Reading by columns gives 0.3192, 0.4917, 0.3159 and 0.1425, a mean of 0.3173. The symmetric loss is their average, 0.3202.¹ A model that could not tell the captions apart would score 1.3863, the natural logarithm of 4, so the toy encoders are far better than a guess.¹

Pair 2 shows the asymmetry: its image is placed with loss 0.3914 but its caption with 0.4917 (illustrative toy values),¹ because two images, image 1 (0.4042) and image 3 (0.4122), both resemble caption 2, while image 2 has only one close rival among the captions (caption 1, 0.3914). A row-only loss would never see that.

Text description of the diagram

• Pair 2 has loss 0.3914 by row and 0.4917 by column.
• Reading by rows and columns gives different losses for each pair.

Trade-offs and edge cases

Contrastive matching trades generation for efficiency. In the CLIP paper, predicting a caption's exact words learned 3x slower than predicting its bag of words, and swapping that for the contrastive objective gave a further 4x gain in zero-shot ImageNet transfer efficiency (§2.3, Figure 2). The result is a scorer, not a writer.

Text description of the diagram

• Predicting a caption's exact words learned 3x slower than predicting its bag of words.
• Contrastive matching gave a further 4x gain in zero-shot ImageNet transfer efficiency.
• Contrastive matching outputs similarity scores, not new text.

Edge cases follow from the mechanism. Two images of the same thing in one batch make one caption fit both, yet the off-diagonal cell counts as a negative, a false negative; mask duplicates where you can. A caption describes only part of an image, so a pair is a coarse signal. A fixed text length, such as the 76 tokens above, truncates long descriptions. And data is a barrier: CLIP used 400 million pairs, and ALIGN (A Large-scale ImaGe and Noisy-text embedding; Jia et al., 2021) used 1.8 billion; Li et al. note that most such datasets are not public.

There are three common ways to adapt CLIP to a task. Zero-shot prompting, described in Step 4, needs no training. A linear probe fits a simple linear classifier on frozen image features, a route the CLIP paper evaluates (§3.2). Full fine-tuning updates the encoders themselves and costs the most compute and memory. The same trade-off appears in large language model (LLM) fine-tuning, where parameter-efficient methods such as low-rank adaptation (LoRA) update only a small part of the weights, and the fine-tuning versus prompt engineering page covers the prompt-or-train choice.

How to answer it in an interview

The spoken structure: define the term, name the two encoders and the shared space, give the loss, then state the limit.

Common mistakes

• "Multimodal means one giant model that reads everything." (wrong) In the CLIP design the modalities have separate encoders joined by a shared space. (right) "Separate encoders, one shared embedding space."
• "CLIP generates captions or images." (wrong) It scores image-text match. (right) "It enables zero-shot classification and retrieval, not generation."
• "Zero-shot means no training." (wrong) The model is pretrained on 400 million pairs. (right) "Zero-shot means no labelled examples of the new task."
• "Contrastive learning needs hand labels." (wrong) The supervision is the natural pairing of images and captions. (right) "The pairing is the label."

Follow-up questions

Is CLIP a generative model?

No. CLIP computes a similarity between an image and a text, so it supports classification and retrieval but does not write text or draw images.

What does zero-shot mean in CLIP?

Zero-shot means the model handles a new task without labelled examples of it, because the task is described in text and compared in the shared space. The pretraining on image-text pairs still happened.

Why do the two encoders need a shared space?

A dot or cosine comparison only means something if both vectors live in the same space. Training with the contrastive loss is what aligns the two spaces so that matching items land close together.

What happens to the loss when the batch gets larger?

Each image meets more negatives: 1,023 at a batch of 1,024 instead of 3 at a batch of 4. The task gets harder, and a guesser's loss rises from 1.3863 at four pairs to 6.9315 at 1,024.¹ Larger batches generally help CLIP-style training, though the SigLIP authors report that the benefit quickly diminishes and that about 32k is sufficient (Zhai et al., 2023). Memory is the cost: DisCo-CLIP reports that CLIP fits a batch of 32,768 on 64 A100 40GB graphics processing units (GPUs) at 27.4GB each, and a batch of 65,536 runs out of memory.

What data does multi modality training need?

Large collections of paired data, such as 400 million image-text pairs for CLIP, which Li et al. note are mostly not public. How large language models are trained places this next to the other training stages.

Sources

1. Learning Transferable Visual Models From Natural Language Supervision (arXiv 2103.00020) — arXiv (Radford et al., OpenAI), 26 Feb 2021
2. Multimodal Foundation Models: From Specialists to General-Purpose Assistants (arXiv 2309.10020) — arXiv (Li et al., Microsoft), 18 Sep 2023
3. Multi-modal Generative Models in Recommendation System (arXiv 2409.10993) — arXiv survey chapter, 17 Sep 2024
4. DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training (arXiv 2304.08480) — arXiv, 17 Apr 2023
5. Learning Transferable Visual Models From Natural Language Supervision, full text (arXiv 2103.00020, §2.3, §2.5, §3.2, Figures 2 and 3) — arXiv (Radford et al., OpenAI), 26 Feb 2021
6. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv 2010.11929) — arXiv (Dosovitskiy et al.), 22 Oct 2020
7. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (arXiv 2102.05918) — arXiv (Jia et al., ALIGN), 11 Feb 2021
8. Flamingo: a Visual Language Model for Few-Shot Learning (arXiv 2204.14198) — arXiv (Alayrac et al.), 29 Apr 2022
9. Visual Instruction Tuning (arXiv 2304.08485) — arXiv (Liu et al., LLaVA), 17 Apr 2023
10. Sigmoid Loss for Language Image Pre-Training (arXiv 2303.15343) — arXiv (Zhai et al., SigLIP), 27 Mar 2023