Advanced Multimodal Models quiz

Question 1

What is the core architectural difference between early-fusion and late-fusion multimodal models?

Accepted Answer

Early-fusion merges raw features before encoding; late-fusion merges high-level representations after encoding

Answer

Early-fusion uses attention; late-fusion uses convolutions

Answer

Early-fusion is only for vision-language; late-fusion is for audio-vision

Answer

They are equivalent architecturally

Question 2

Explain the role of the CLIP contrastive loss in training vision-language models.

Accepted Answer

InfoNCE loss maximizes similarity of correct pairs and minimizes similarity to in-batch negatives

Answer

CLIP uses cross-entropy loss over fixed image categories

Answer

CLIP trains only the text encoder while freezing the image encoder

Answer

CLIP uses triplet loss with hard negative mining

Question 3

What is the Perceiver IO architecture and how does it handle arbitrary multimodal inputs and outputs?

Accepted Answer

Perceiver IO uses latent cross-attention to decouple compute from input/output size

Answer

Perceiver IO applies separate CNNs per modality and concatenates outputs

Answer

Perceiver IO is a recurrent model with LSTM cells

Answer

Perceiver IO uses sparse transformers on raw pixel values

Question 4

How do video-language models handle temporal information that static image-text models cannot?

Accepted Answer

Video-language models use temporal positional encodings and attention across frame tokens

Answer

Video-language models simply concatenate all frame pixels into one long image

Answer

Video-language models use only audio to infer temporal order

Answer

Video-language models duplicate the static VLM for each frame independently

Question 5

Explain the difference between encoder-decoder VLMs (like BLIP-2) and decoder-only VLMs (like LLaVA).

Accepted Answer

Encoder-decoder VLMs use a Q-Former bridge and seq2seq LLM; decoder-only VLMs project visual tokens into a causal LLM

Answer

Encoder-decoder VLMs are always larger and slower

Answer

Decoder-only VLMs require an external OCR module

Answer

Both architectures are identical in practice

Question 6

What evaluation benchmark is used to measure general multimodal understanding capability across diverse tasks?

Accepted Answer

MMBench, MME, and MMMU are standard multimodal benchmarks covering perception, reasoning, and domain knowledge

Answer

BLEU and ROUGE scores are the primary multimodal benchmarks

Answer

ImageNet top-1 accuracy is the gold standard for VLMs

Answer

VQA v2 is the only accepted multimodal benchmark

Question 7

How do you prevent catastrophic forgetting when fine-tuning a pretrained VLM on a new visual task?

Accepted Answer

LoRA adapters and freezing LLM weights while training only the visual projector are the most practical approaches

Answer

Full fine-tuning on new data alone is sufficient

Answer

Reducing learning rate to zero prevents forgetting

Answer

Catastrophic forgetting does not affect multimodal models

Question 8

How does the Qwen-VL model handle multiple images in a single conversation turn?

Accepted Answer

Qwen-VL prefixes each image with an index tag and compresses it to 256 visual tokens for interleaved multi-image context

Answer

Qwen-VL concatenates all images into a single image before encoding

Answer

Qwen-VL processes only one image per conversation turn

Answer

Qwen-VL uses a separate model instance for each image

Question 9

What is contrastive captioners (CoCa) and how does it unify contrastive and generative objectives?

Accepted Answer

CoCa combines contrastive loss on CLS embeddings with cross-entropy captioning loss in one model

Answer

CoCa trains two separate models and ensembles their outputs

Answer

CoCa uses only generative loss and discards contrastive objectives

Answer

CoCa replaces CLIP by using BERT as the image encoder

Question 10

What hardware considerations are most important when deploying a large VLM (e.g., 70B parameters) in production?

Accepted Answer

Multi-GPU tensor parallelism, KV-cache management for visual tokens, and int4/int8 quantization are critical for 70B VLM deployment

Answer

VLMs only need CPUs because vision is not compute-intensive

Answer

A single A100 GPU is always sufficient for any VLM

Answer

KV-cache is irrelevant for multimodal models

Question 11

What is the model architecture of PaliGemma and what makes it notable?

Accepted Answer

PaliGemma combines SigLIP vision encoder with Gemma LLM, optimized for fine-tuning on specialized tasks

Answer

PaliGemma is a proprietary model with no open weights

Answer

PaliGemma uses a Q-Former bridge and FlanT5 backbone

Answer

PaliGemma is exclusively designed for audio-visual tasks

Question 12

How do you evaluate spatial reasoning capability in a VLM?

Accepted Answer

CVBench, What's Up, and SpatialBench evaluate spatial reasoning with accuracy on positional relationship questions

Answer

BLEU score is used to evaluate spatial reasoning

Answer

Spatial reasoning is evaluated using bounding box IoU alone

Answer

VLMs achieve human-level spatial reasoning on all current benchmarks

Question 13

How does the POPE benchmark test for object hallucination and what are its three sampling strategies?

Accepted Answer

POPE uses random, popular, and adversarial sampling strategies for non-existent object selection in yes/no probing

Answer

POPE measures caption BLEU score against reference captions

Answer

POPE uses only adversarial sampling from ImageNet categories

Answer

POPE requires human annotators to score each response

Test yourself