Multimodal AI (vision + language)

CLIP uses a contrastive objective: given a batch of N image-text pairs, it maximizes cosine similarity for the N matching pairs and minimizes it for the N²−N non-matching pairs via a symmetric InfoNCE loss over images and texts. This learns a shared embedding space, so at inference you can classify by comparing an image embedding to text-prompt embeddings of candidate classes, enabling open-vocabulary zero-shot recognition without task-specific fine-tuning.

labels = torch.arange(N)
logits = (image_emb @ text_emb.T) * temp
loss = (ce(logits, labels) + ce(logits.T, labels)) / 2

The temperature scales the logits (similarities) before the softmax, controlling how sharply the model separates positives from negatives; a low temperature sharpens the distribution and penalizes hard negatives more. In CLIP it is a learnable log-parameterized scalar that is optimized jointly and clipped to prevent it from collapsing to extreme values that destabilize training.

Q-Former is a lightweight transformer with a fixed set of learnable query embeddings that cross-attend to the frozen image encoder's features, distilling the most language-relevant visual information into a small set of output tokens. These query outputs are linearly projected and prepended as soft visual prompts to the frozen LLM, so only the Q-Former (and projection) is trained, drastically cutting compute versus fine-tuning the encoder or LLM.

A pretrained vision encoder (e.g., CLIP ViT) produces patch embeddings that pass through a learned projection (originally a linear layer, later an MLP) mapping them into the LLM's token embedding space. These projected visual tokens are concatenated with the text token embeddings and fed to the LLM, which is trained (often with the projector first, then the LLM) on instruction-following multimodal data.

Flamingo inserts gated cross-attention dense (GATED XATTN-DENSE) layers between the frozen LLM blocks; these attend from the text to visual features produced by a Perceiver Resampler that compresses variable-length image/video features into a fixed number of tokens. A tanh gating mechanism initialized at zero ensures the pretrained LLM behavior is preserved at the start of training and visual conditioning is added smoothly.

Raw vision encoders emit a large, variable number of patch/frame features, which would make cross-attention or token concatenation expensive and length-dependent. A Perceiver Resampler uses a fixed set of learned latent queries to cross-attend over those features and output a constant, small number of visual tokens, decoupling downstream compute from input resolution or video length.

The modality gap is the empirical finding that image embeddings and text embeddings occupy two separated cones/clusters in the shared space rather than being fully intermixed, even though matched pairs are pulled together. It arises from initialization geometry combined with the contrastive temperature, and can affect downstream tasks; some methods explicitly close or exploit the gap (e.g., by shifting embeddings).

CoCa (Contrastive Captioner) splits the text decoder into a unimodal part that produces a pooled text embedding for a contrastive loss against the image embedding, and a multimodal part with cross-attention to image features trained with a captioning (autoregressive) loss. Jointly optimizing both gives strong retrieval/zero-shot alignment plus generative and fine-tunable captioning/VQA capabilities from a single pretraining run.

Late fusion encodes each modality independently and combines high-level representations (e.g., concatenation or similarity) only near the output, which is modular and good for retrieval but limits cross-modal interaction. Early fusion mixes modalities at the token/feature level inside a shared transformer, enabling fine-grained interaction (better for grounding/VQA) at higher compute cost and less modularity.

Stage 1 trains only the vision-to-language projector on large noisy image-caption data to align modalities while keeping the LLM frozen, which is cheap and stabilizes the connection. Stage 2 fine-tunes the projector and LLM on smaller, high-quality multimodal instruction data so the model learns to follow user queries and reason over images without catastrophically forgetting language skills.

Vision encoders trained at a fixed low resolution (e.g., 224 or 336) lose fine detail like small text when an image is downsized. AnyRes splits a high-resolution image into multiple grid tiles, encodes each tile plus a downsized global view, and concatenates all visual tokens, giving the LLM both global context and local detail at the cost of more visual tokens.

SigLIP replaces the softmax-based InfoNCE (which normalizes over the whole batch) with a pairwise sigmoid loss that treats each image-text pair independently as a binary classification of match/no-match. This removes the need for a global all-gather to compute the softmax normalizer, enabling efficient training at smaller batch sizes and better scaling across devices.

Cross-modal retrieval ranks items of one modality (e.g., images) given a query of another (e.g., a text caption) using embedding similarity in a shared space. It is evaluated with Recall@K (R@1, R@5, R@10) for both image-to-text and text-to-image directions, measuring whether the correct match appears in the top-K retrieved results.

The image is divided into fixed-size non-overlapping patches (e.g., 16×16); each patch is flattened and linearly projected into an embedding, then positional embeddings are added to encode spatial location. The resulting sequence (optionally with a [CLS] token) is processed by transformer encoder layers, producing patch features that VLMs consume as visual tokens.

patches = rearrange(img,'b c (h p1)(w p2)->b (h w)(p1 p2 c)',p1=16,p2=16)
tokens = patch_embed(patches) + pos_embed

Object hallucination occurs when a VLM describes objects not present in the image, driven by strong language priors (co-occurrence statistics) overriding weak visual grounding, plus biased instruction-tuning data. Mitigations include grounded/contrastive decoding that downweights the language-only prior, higher-resolution visual tokens, and evaluation/curation with benchmarks like POPE to penalize hallucinated objects.

POPE (Polling-based Object Probing Evaluation) measures object hallucination in VLMs by asking simple yes/no questions about whether specific objects are present in an image, including objects sampled by popularity and co-occurrence. It reports accuracy/F1, exposing models that answer 'yes' due to language bias rather than actual visual evidence.

Image captioning generates a free-form text description of the whole image, while visual grounding (referring expression localization) takes a text phrase and outputs the specific image region (bounding box or mask) it refers to. Grounding requires aligning language to spatial locations, whereas captioning only requires producing globally coherent descriptive text.

The ITM head performs a fine-grained binary classification of whether an image and text actually correspond, using fused multimodal features (cross-attention), unlike the coarse contrastive loss that uses pooled unimodal embeddings. It is typically trained with hard negatives mined from the contrastive similarities, sharpening the model's ability to detect subtle mismatches for retrieval reranking.

CLIP was trained with a fixed context length of 77 tokens (including start/end tokens), so its positional embeddings and training distribution only cover short prompts. In practice, long captions or detailed prompts get truncated, degrading retrieval/conditioning quality, which motivates long-context CLIP variants or sentence chunking and embedding aggregation.

Contrastive decoding (e.g., VCD) generates next-token logits conditioned on the real image and also on a distorted or absent image, then subtracts the latter to amplify tokens that genuinely depend on visual evidence. This suppresses tokens driven purely by the language prior, reducing hallucinated content without retraining.

Interleaved data (documents with images and text in their natural reading order) teach the model to handle multiple images, maintain context across them, and learn in-context/few-shot multimodal patterns. This enables capabilities like answering about the second image given the first, and few-shot prompting with image-text examples, which paired single-image captions cannot provide.

VQAv2 introduces complementary/balanced pairs: for each question, two similar images yield different answers, so a model cannot guess correctly from the question text alone. This forces reliance on visual content, reducing the dataset's exploitable language priors and making accuracy a more faithful measure of grounded reasoning.

A frozen low-resolution vision encoder (e.g., CLIP at 336px) limits the model's ability to read fine text and small details, bottlenecking OCR-heavy and document tasks. Remedies include higher-resolution or tiling strategies (AnyRes), stronger or unfrozen encoders, dynamic resolution (e.g., NaViT-style packing), and adding multiple encoders or higher token budgets.

A dual-encoder (like CLIP) encodes image and text separately into a shared space and compares with a dot product, which is fast and scalable for retrieval but has no deep cross-modal interaction. A fusion-encoder jointly attends across modalities (cross-attention) for richer reasoning (VQA, ITM) but is far more expensive because it must process each image-text pair together rather than precomputing embeddings.

Coordinates are quantized into a fixed grid of location bins and represented as special location tokens (e.g., <loc0123>) added to the vocabulary, so the model emits boxes/points as a token sequence. This unifies detection and grounding with text generation under one autoregressive objective, letting the model interleave words and coordinates.

Replacing the linear vision-language connector with a two-layer MLP (with a nonlinearity) increases the projector's capacity to map visual features into the LLM embedding space, improving alignment and downstream benchmark accuracy. It is a cheap change since the projector is small relative to the LLM, yet it meaningfully boosts representational expressiveness.

Fine-tuning all LLM weights on a narrower multimodal distribution can overwrite language knowledge and instruction-following ability learned during text pretraining. Mitigations include freezing or partially freezing the LLM, using parameter-efficient methods (LoRA), mixing in text-only data during multimodal tuning, and lower learning rates for pretrained layers.

MMMU (Massive Multi-discipline Multimodal Understanding) tests college-level, expert-domain reasoning across subjects (science, medicine, engineering) requiring figures, charts, and diagrams plus deliberate reasoning, whereas VQAv2 asks short factual questions about everyday natural images. MMMU therefore stresses knowledge and multimodal reasoning, exposing gaps that VQAv2 accuracy hides.

Token concatenation prepends visual tokens to the text sequence, so they add to the LLM's context length and incur full self-attention cost (quadratic in total length) but require no architectural changes. Cross-attention keeps visual features in separate keys/values attended to by dedicated layers, so text sequence length is unchanged and visual cost scales with the number of cross-attention layers, but it requires inserting new trainable layers.

The contrastive loss learns most from negatives that are close to the positive in embedding space; trivial negatives provide little gradient signal. Large batch sizes (more in-batch negatives) and explicitly mined hard negatives (semantically similar but mismatched pairs) sharpen the decision boundary and improve fine-grained discrimination and retrieval.

A separate OCR stage introduces error propagation, loses layout/visual context, and adds latency and language coverage constraints. OCR-free models read text directly from pixels within a single end-to-end transformer, jointly modeling layout, figures, and text, which improves robustness on complex documents and removes the brittle OCR dependency.

L2-normalization projects embeddings onto the unit hypersphere so the dot product equals cosine similarity, making the temperature-scaled contrastive loss depend only on angle, not magnitude. This stabilizes training, prevents the model from trivially increasing norms to change logits, and makes retrieval thresholds comparable across items.

img = F.normalize(img,dim=-1)
txt = F.normalize(txt,dim=-1)
sim = img @ txt.T

They must encode multiple frames and model temporal relations, typically by sampling frames, encoding each with a ViT, and then aggregating with temporal attention, pooling, or a resampler that compresses across time. Key trade-offs are token budget (frames × patches can explode) versus temporal fidelity, so techniques like frame sampling, temporal pooling, or memory tokens manage cost while preserving motion/order information.

Freezing both large components and training only the lightweight bridging module (Q-Former/projector) dramatically reduces compute, memory, and data needs while preserving the strong unimodal capabilities the encoder and LLM already learned. It also avoids catastrophic forgetting and lets you reuse the same frozen LLM across modalities, at the cost of being limited by the frozen encoder's representations.