Multimodal AI (vision + language) quiz

Question 1

In CLIP, what training objective aligns image and text representations, and why is it effective for zero-shot transfer?

Accepted Answer

A symmetric contrastive (InfoNCE) loss aligning matched image-text pairs in a shared space

Answer

A pixel reconstruction loss that regenerates images from captions

Answer

A next-token cross-entropy loss predicting captions autoregressively

Answer

A triplet margin loss using hand-labeled hard negatives only

Question 2

What is the role of the temperature parameter in CLIP's contrastive loss, and how is it typically handled?

Accepted Answer

A learnable logit scale that sharpens or softens the similarity softmax

Answer

A fixed dropout rate applied to image patches

Answer

The learning-rate warmup duration for the text encoder

Answer

A regularization weight on the embedding L2 norm

Question 3

How do BLIP-2's Q-Former and the Querying Transformer bridge a frozen vision encoder and a frozen LLM?

Accepted Answer

Learnable queries cross-attend to frozen image features and feed the LLM as soft prompts

Answer

It fine-tunes both the vision encoder and LLM end-to-end

Answer

It converts images to text captions that are fed as plain prompts

Answer

It uses diffusion to translate pixels into token embeddings

Question 4

In LLaVA-style models, how are visual features injected into the language model?

Accepted Answer

Projected patch embeddings are concatenated as tokens in the LLM input sequence

Answer

Images are described by a separate captioner whose text is prompted

Answer

Cross-attention layers replace the LLM's self-attention entirely

Answer

The LLM weights are swapped per modality at inference

Question 5

What architectural mechanism does Flamingo use to condition a frozen LLM on visual inputs?

Accepted Answer

Gated cross-attention layers interleaved in the frozen LLM, fed by a Perceiver Resampler

Answer

Replacing all FFN layers with vision adapters

Answer

Concatenating raw pixels to the token stream

Answer

Training a brand-new decoder from scratch on image-text pairs

Question 6

Why is the Perceiver Resampler important in vision-language models that handle high-resolution images or video?

Accepted Answer

It compresses variable-length visual features into a fixed small set of tokens

Answer

It upsamples low-res images to a canonical resolution

Answer

It performs OCR before encoding

Answer

It replaces positional embeddings with learned coordinates

Question 7

What is the modality gap observed in contrastively trained models like CLIP?

Accepted Answer

Image and text embeddings cluster in separate regions of the shared space

Answer

The model cannot encode text longer than 77 tokens

Answer

Vision and text encoders share no parameters

Answer

Embeddings collapse to a single point

Question 8

How does CoCa combine contrastive and generative (captioning) objectives in one model?

Accepted Answer

A split decoder doing contrastive on pooled text plus autoregressive captioning

Answer

Two fully separate models ensembled at inference

Answer

Contrastive pretraining followed by GAN fine-tuning

Answer

Masked image modeling combined with masked language modeling only

Question 9

What is the difference between early fusion and late fusion in multimodal architectures?

Accepted Answer

Late fusion combines high-level features; early fusion mixes modalities deep in the network

Answer

Early fusion always uses contrastive loss; late fusion uses generative

Answer

Late fusion requires paired data; early fusion does not

Answer

They are identical except for naming

Question 10

Why do many VLMs (e.g., LLaVA-1.5) use a two-stage training recipe of pretraining then instruction tuning?

Accepted Answer

Align the projector cheaply first, then instruction-tune for following queries

Answer

Train the vision encoder, then discard the LLM

Answer

Do RLHF first, then supervised alignment

Answer

Pretrain on text only, then on images only, never jointly

Question 11

What problem does the 'AnyRes' / image-splitting strategy (e.g., in LLaVA-NeXT) solve for high-resolution inputs?

Accepted Answer

Tile the image into patches plus a global view to preserve fine detail

Answer

Use a diffusion super-resolution model before encoding

Answer

Crop to the salient object only

Answer

Reduce token count by averaging patches

Question 12

How does SigLIP differ from CLIP in its contrastive loss, and what is the benefit?

Accepted Answer

A pairwise sigmoid loss instead of batch-softmax InfoNCE, easing small-batch training

Answer

A diffusion-based alignment loss

Answer

An added triplet loss on hard negatives

Answer

A reconstruction loss replacing contrastive learning

Question 13

What is cross-modal retrieval and which metric is standard for evaluating it?

Accepted Answer

Recall@K (R@1/R@5/R@10) for both retrieval directions

Answer

BLEU score on captions

Answer

Pixel PSNR between query and result

Answer

Perplexity of the text encoder

Question 14

In Vision Transformers used by VLMs, how are images converted into a token sequence?

Accepted Answer

Split into patches, linearly project each, add positional embeddings

Answer

Run a CNN and take the final softmax logits

Answer

Apply OCR and tokenize the extracted text

Answer

Convert pixels directly into BPE subwords

Question 15

What causes object hallucination in vision-language models, and name one mitigation.

Accepted Answer

Language priors override weak visual grounding; mitigate with contrastive decoding

Answer

Too small a vocabulary; mitigate by enlarging the tokenizer

Answer

Float16 rounding; mitigate by using int8

Answer

Positional embeddings; mitigate by removing them

Question 16

What is the POPE benchmark designed to measure?

Accepted Answer

Object hallucination via yes/no presence questions

Answer

Image generation fidelity

Answer

Latency of the vision encoder

Answer

Caption fluency via BLEU

Question 17

How does visual grounding differ from image captioning?

Accepted Answer

Grounding localizes a region for a phrase; captioning describes the whole image

Answer

They are identical tasks

Answer

Captioning outputs boxes; grounding outputs text

Answer

Grounding requires no image input

Question 18

What is the purpose of an image-text matching (ITM) head in models like BLIP?

Accepted Answer

Fine-grained binary match classification on fused features with hard negatives

Answer

Generating the next caption token

Answer

Predicting masked image patches

Answer

Computing the temperature parameter

Question 19

Why is CLIP's text encoder limited to 77 tokens, and what is a practical implication?

Accepted Answer

Fixed 77-token training context truncates long prompts

Answer

It uses character-level encoding with a hard cap

Answer

The image encoder dictates the text length

Answer

Token limits only apply at inference, not training

Question 20

What is the key idea behind contrastive decoding for reducing VLM hallucination at inference?

Accepted Answer

Subtract logits from a distorted/no-image pass to amplify visually grounded tokens

Answer

Average logits across multiple beams

Answer

Lower the temperature to zero

Answer

Mask the text prompt during decoding

Question 21

How do interleaved image-text training data benefit models like Flamingo or Idefics?

Accepted Answer

They enable multi-image context and in-context few-shot multimodal learning

Answer

They reduce the vocabulary size needed

Answer

They eliminate the need for a vision encoder

Answer

They are only useful for OCR

Question 22

What does VQAv2 add over the original VQA dataset to reduce language-prior bias?

Accepted Answer

Balanced image pairs giving different answers to the same question

Answer

Higher-resolution images only

Answer

Open-ended generation instead of multiple choice

Answer

Removal of yes/no questions

Question 23

In multimodal LLMs, what is a common cause of the vision encoder becoming a bottleneck, and how is it addressed?

Accepted Answer

Low encoder resolution loses detail; fix with higher-res/tiling/dynamic resolution

Answer

The text tokenizer is too large; shrink it

Answer

Cross-attention is too deep; remove layers

Answer

Use of float32; switch to bf16

Question 24

What is the difference between a dual-encoder and a fusion-encoder vision-language model?

Accepted Answer

Dual-encoder compares separate embeddings; fusion-encoder jointly cross-attends

Answer

Dual-encoder uses cross-attention; fusion-encoder uses dot products

Answer

They differ only in parameter count

Answer

Fusion-encoders cannot do VQA

Question 25

How are bounding boxes typically represented so an autoregressive VLM can output them as text (e.g., in Kosmos-2 / PaliGemma)?

Accepted Answer

Quantized coordinates emitted as special location tokens in the vocabulary

Answer

Floating-point regression heads only

Answer

RLE masks stored outside the text stream

Answer

Polar coordinates relative to image center

Question 26

What is the benefit of using an MLP projector instead of a single linear layer in LLaVA-1.5?

Accepted Answer

More capacity to map visual features into the LLM space, improving alignment

Answer

It reduces the number of visual tokens

Answer

It removes the need for a vision encoder

Answer

It enables image generation

Question 27

Why can naive fine-tuning of a VLM's LLM on multimodal data cause catastrophic forgetting, and how is it mitigated?

Accepted Answer

Overwriting language skills; mitigate via LoRA, freezing, and mixing text data

Answer

Vision tokens corrupt embeddings; mitigate by removing them

Answer

Too low resolution; mitigate by upscaling

Answer

Temperature drift; mitigate by clipping logits

Question 28

What distinguishes the MMMU benchmark from VQAv2 for evaluating modern multimodal models?

Accepted Answer

Expert-level multi-discipline reasoning over diagrams vs short everyday VQA

Answer

Image generation quality vs classification

Answer

Latency benchmarking vs accuracy

Answer

OCR-only vs captioning

Question 29

How does cross-attention conditioning differ from token-concatenation (prefix) conditioning in VLMs, regarding sequence length and compute?

Accepted Answer

Concatenation lengthens the sequence; cross-attention keeps it fixed via separate KV

Answer

Both increase sequence length equally

Answer

Cross-attention requires no new parameters

Answer

Concatenation is impossible with decoder LLMs

Question 30

What is the role of negative sampling quality (hard negatives) in contrastive vision-language training?

Accepted Answer

Hard/close negatives provide stronger gradients and sharpen discrimination

Answer

Negatives should always be random for stability

Answer

Hard negatives cause the temperature to vanish

Answer

More negatives reduce accuracy

Test yourself