What are multimodal models, and how do CLIP and BLIP-2 work?

In this article

1. Before you start
2. Why interviewers ask this
3. Multimodal models from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the deep learning learning path and uses softmax and log-probabilities, which the statistics and probability guide covers.

Why interviewers ask this

Candidates report questions on Contrastive Language-Image Pre-training (CLIP), Bootstrapping Language-Image Pre-training (BLIP), in its second version BLIP-2, and the Vision Transformer (ViT) in deep-learning rounds at several large companies. The questions test whether you know where two modalities meet inside a model, because that choice decides cost, flexibility and failure modes.

A weak answer says "CLIP aligns images and text." A strong answer says the two encoders are separate, that they meet only at a dot product of final embeddings, and that training is a symmetric cross-entropy over a batch where every other pair is a negative. A Staff-level answer adds where this breaks: tasks such as counting or satellite imagery, and the long-document case where a model that reads one image well fails on many pages. The same ideas return in deep learning interview questions.

Multimodal models from first principles

Patch token

A patch token is a vector that stands for one small square piece of an image, so that a transformer, which reads sequences, can process the image the way it processes a sentence of word tokens.

Contrastive pre-training

Contrastive pre-training is a way of training two encoders on matched pairs, such as an image and its caption, so that matching pairs get similar vectors and every mismatched pairing in the batch gets dissimilar ones, without any class labels.

Zero-shot classification

Zero-shot classification is the use of a trained model on a classification task it was never trained on, by writing the class names as text and picking the class whose text vector best matches the input, with no task-specific labeled examples.

Q-Former

A Q-Former, short for Querying Transformer, is a small trainable transformer with a fixed set of learned query vectors that reads the output of a frozen image encoder and passes a short summary to a frozen language model.

In plain words

Imagine two translators who never speak to each other. One turns photos into a short code. The other turns sentences into the same kind of code. If they were trained well, a photo of a dog and the sentence "a dog on a beach" get nearly the same code. Matching is then just comparing codes. The analogy breaks in one place: the translators learn their code together, from millions of examples, rather than being given it.

In one precise sentence, a multimodal model is a network that learns one function per modality. In one design, related inputs land close together in one vector space. In the other, one modality becomes extra tokens for a language model.

Step 1: Each modality gets its own encoder

An encoder is a network that turns raw input into vectors. For text it reads word pieces. For images, a ViT cuts the picture into patches and treats each patch token like a word; Dosovitskiy et al. (2020) show that a pure transformer applied directly to sequences of image patches can do image classification well, without a convolutional network. For a 224 by 224 pixel image cut into 16 by 16 pixel patches (illustrative sizes), that gives 14 × 14 = 196 patch tokens.¹

Input: raw pixels or raw words. Output: vectors of one fixed size. You now know the parts; the next step is how they are taught to agree. Older image models used convolutional networks in this slot.

Step 2: Contrastive training pulls true pairs together and pushes wrong pairs apart

CLIP trains an image encoder and a text encoder on 400 million image and text pairs from the internet. Within a batch of N pairs, the N real pairs should get a high cosine similarity (a dot product after scaling each vector to length one) and the N² − N other pairings a low one. An in-batch negative is any other item in the same batch, which serves as a wrong answer for free. The loss is a symmetric cross-entropy: a softmax over each row (image to text) and over each column (text to image).

The scale is the point. CLIP uses a minibatch of 32,768, so one batch holds 32,768 right pairs and 1,073,709,056 wrong ones.¹ A learned temperature divides the similarities before the softmax; the paper clips it so it never scales logits by more than 100.

Step 3: Zero-shot classification turns class names into a classifier

After pre-training, CLIP classifies without task training. You write each class as a sentence, such as "a photo of a dog", encode all sentences with the text encoder, encode the image, and pick the class whose text vector scores highest. Class names written as text turn the model into a zero-shot classifier. The paper reports that its best model reaches 76.2% on ImageNet, matching the original ResNet-50 without using any of the 1.28 million labeled training images.

Prompt wording matters: the authors report that prompt engineering plus ensembling improved ImageNet accuracy by almost 5%, and that ensembling 80 prompts added about 3.5% over one default prompt. "Zero-shot" means no task-specific labels, not no supervision: the 400 million captions were the supervision. For how class scores become probabilities, see activation functions.

Step 4: A Q-Former bridges a frozen image encoder and a language model

CLIP can match but not talk. BLIP-2, from Bootstrapping Language-Image Pre-training, adds a language model. Li et al. (2023) keep both the image encoder and the large language model (LLM) frozen and train only a lightweight Q-Former, which contains 188M parameters, in two stages: the first learns from the frozen image encoder, the second from the frozen language model.

The Q-Former uses 32 learned queries of dimension 768, so its output Z is 32 × 768, a bottleneck much smaller than the 257 × 1,024 frozen ViT-L/14 features (24,576 against 263,168 numbers, 10.7 times fewer).¹ A linear layer then projects the 32 vectors to the language model's embedding width (Li et al., §3.3), so the language model reads 32 soft visual tokens instead of 257, which is 8.03 times fewer.¹ The authors report beating Flamingo80B by 8.7% on zero-shot visual question answering with 54 times fewer trainable parameters.

1 · Two encoders2 · Similarity matrix3 · Zero-shot4 · Q-FormerReplay

Step-by-step description

1. Image and text are encoded separately.
2. True pairs score high, wrong pairs low.
3. Class names become a classifier.
4. Queries feed a frozen language model.

• An image encoder and a text encoder are separate networks.
• They meet only at a similarity score in a batch matrix.
• Class names written as text turn the model into a zero-shot classifier.
• A Q-Former passes image features to a frozen language model.

Worked example

A batch of four image-caption pairs shows the loss by hand. The cosine similarities below are illustrative: entry (i, j) compares image i with caption j, and the diagonal holds the true pairs. Rows are (0.9, 0.2, 0.1, 0.0), (0.3, 0.8, 0.2, 0.1), (0.1, 0.2, 0.5, 0.4) and (0.0, 0.1, 0.3, 0.7).¹

At temperature 1 the symmetric loss is 1.0071, close to the 1.3863 of a model that scores all captions equally (the natural log of 4), because dividing by 1 leaves the scores close together. At an illustrative temperature of 0.1 the image-to-text loss is 0.0987, the text-to-image loss is 0.0614 and the symmetric mean is 0.0801.¹ Image 3 is hardest: its true caption scores 0.5 against 0.4 for a wrong one, so its row loss is 0.3618 at temperature 0.1. A smaller temperature sharpens the softmax, which is why CLIP learns it instead of fixing it.

Text description of the diagram

• Frozen image features of 257 by 1,024 hold 263,168 numbers.
• The Q-Former output of 32 by 768 holds 24,576 numbers, about 10.7 times fewer.

Trade-offs and edge cases

Dual encoders are fast and flexible but shallow. Because CLIP's encoders meet only at a final dot product, you can precompute every image vector once and search them, but the model cannot compare a phrase with one region of the image. The paper itself notes the poor capability of zero-shot CLIP on more complex tasks: satellite images, lymph node tumor detection, counting objects in synthetic scenes, German traffic signs and distance to the nearest car.

A single-stream design feeds both modalities into one transformer, which allows fine interaction but prevents precomputing one side. VisualBERT is the standard example. A query bridge such as the Q-Former trains only a small module between frozen models, yet it has limits. The BLIP-2 authors did not see better visual question answering when the language model was shown in-context examples, which they attribute to pre-training data with one image-text pair per sample. Because the encoder is frozen, it can also hold more samples per GPU (graphics processing unit), and the authors use in-batch negatives for that reason.

Text description of the diagram

• A dual encoder such as CLIP meets only at a dot product.
• A single-stream design feeds both modalities into one transformer.
• A query bridge such as the Q-Former trains only a small module between frozen models.

Two later developments change the picture. Zhai et al. (2023) replace the softmax contrastive loss with a sigmoid loss that scores each image-text pair on its own, so it needs no batch-wide normalization. And many later vision-language models drop the Q-Former: Liu et al. (2023) train a projection matrix that maps CLIP ViT-L/14 features into the language model's word-embedding space.

A model that handles one image can fail on a long document, because visual tokens per page times pages overflow the context; the last follow-up below gives the numbers and the fixes. CLIP also feeds image generation: Ramesh et al. (2022) generate images from CLIP embeddings, and diffusion models cover the generator side. Stable training of such deep stacks leans on residual connections and normalization layers.

How to answer it in an interview

The spoken structure: name the two encoders, say where they meet, give the loss, then contrast with the bridge design and name one limit.

Common mistakes

• "CLIP uses cross-modal attention between image and text." (wrong) Its encoders are independent and meet only at the dot product. (right) "CLIP is a dual encoder; cross-attention appears in the BLIP-2 Q-Former."
• "BLIP-2 trains the whole vision-language model." (wrong) Both the image encoder and the language model stay frozen. (right) "Only the Q-Former is trained."
• "Zero-shot means the model saw no text supervision." (wrong) It means no task-specific labels. (right) "The supervision was 400 million web image-text pairs."
• "Zero-shot CLIP is good at everything." (wrong) The paper shows weakness on counting, satellite and medical images. (right) "It is strong on general object recognition and weak on specialized tasks."

Follow-up questions

What is the role of cross-modal attention in models like CLIP?

CLIP has none: its image and text encoders only meet at the dot product of their final embeddings. VisualBERT, by contrast, feeds text tokens and image-region features into one transformer, so self-attention mixes the modalities at every layer (Li et al., 2019). In BLIP-2 the Q-Former's learned queries attend to the frozen image features.

What does the Q-Former in BLIP-2 do?

It bridges the modality gap: 32 learned queries read the frozen image features and output a short sequence that a frozen language model accepts as input. It is trained in two stages, first against the image encoder, then against the language model.

What are the three losses used to train the Q-Former?

Image-Text Contrastive Learning (ITC) aligns image and text representations, Image-Text Matching (ITM) learns fine-grained alignment, and Image-grounded Text Generation (ITG) trains it to generate text from the image. They share parameters and differ in the attention mask between queries and text.

How does ViT work?

A ViT cuts the image into patches, turns each into a patch token and runs a standard transformer over the sequence, as Dosovitskiy et al. show. It needs no convolutions, though it replaces the convolutional network only when there is enough training data.

How do you evaluate a multimodal document-QA system, and tell a perception failure from a reasoning one?

Test perception alone by asking the model to transcribe or describe the page that holds the answer. Then give it the correct text and ask the question: if it fails there, the problem is reasoning. If it transcribes wrongly, the problem is perception.

Your VLM answers single-image questions well but falls apart on a long PDF. How do you fix it?

A vision-language model (VLM) takes a picture and text, and a portable document format (PDF) of many pages makes the input long. Visual tokens per page times pages overflow the context and dilute attention: for an illustrative 80-page PDF at 257 tokens per page, that is 20,560 visual tokens, against 2,560 with a 32-query bridge.¹ The remedies are page-level retrieval over page embeddings, using the PDF's text layer or optical character recognition (OCR) where it exists; per-page extraction, then answering over the extracts; and page-level evaluation to find where it fails. Measure on your own documents before choosing.

Sources

1. Learning Transferable Visual Models From Natural Language Supervision (arXiv 2103.00020) — arXiv (Radford et al., OpenAI; ICML 2021), 26 Feb 2021
2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (arXiv 2301.12597) — arXiv (Li et al., Salesforce; ICML 2023), 30 Jan 2023
3. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv 2010.11929) — arXiv (Dosovitskiy et al., Google; ICLR 2021), 22 Oct 2020
4. VisualBERT: A Simple and Performant Baseline for Vision and Language (arXiv 1908.03557) — arXiv (Li et al.), 9 Aug 2019
5. Hierarchical Text-Conditional Image Generation with CLIP Latents (arXiv 2204.06125) — arXiv (Ramesh et al., OpenAI), 13 Apr 2022
6. Sigmoid Loss for Language Image Pre-Training (arXiv 2303.15343) — arXiv (Zhai et al., Google), 27 Mar 2023
7. Visual Instruction Tuning (arXiv 2304.08485) — arXiv (Liu et al.), 17 Apr 2023