Advanced Multimodal Models

Early-fusion combines raw or low-level feature representations from multiple modalities before the main processing network, allowing joint feature learning from the start. Late-fusion keeps modality-specific encoders separate and merges their high-level embeddings or predictions at the end. Early-fusion captures fine-grained cross-modal interactions but demands aligned inputs; late-fusion is more modular, easier to train per modality, and tolerates missing modalities gracefully, though it may miss subtle cross-modal correlations.

Flamingo freezes a pretrained LLM and a pretrained vision encoder, then inserts learnable Perceiver Resampler modules and gated cross-attention layers between the LLM transformer blocks. The Perceiver Resampler compresses variable-resolution image features into a fixed set of visual tokens. Gated cross-attention lets text tokens attend to visual tokens, with a learned gate initialized near zero so training begins from the frozen LLM baseline. This design achieves few-shot multimodal prompting without full fine-tuning.

A visual token is a discrete or continuous embedding representing a patch or region of an image, fed into the language model alongside text tokens. LLM attention is O(n²) in sequence length, so adding hundreds of visual tokens per image multiplies the KV-cache size and attention compute substantially. Models like LLaVA-1.5 use 256–576 visual tokens; GPT-4o uses dynamic token counts. Reducing visual tokens via pooling, resampling, or dynamic resolution slashing directly cuts latency and memory without proportional quality loss.

CLIP trains an image encoder and a text encoder jointly using InfoNCE contrastive loss over a batch of N image-text pairs. For each pair, the model maximizes cosine similarity between the correct image and text embeddings while minimizing similarity to all N-1 negatives in the batch. The resulting shared embedding space allows zero-shot classification by comparing image embeddings to text prompt embeddings. CLIP's representations are widely reused as frozen visual backbones in downstream VLMs, reducing the need for task-specific labeled data.

LLaVA-HD (and similar HD variants) splits high-resolution images into multiple fixed-resolution tiles, encodes each tile independently through the CLIP vision encoder, and concatenates the tile tokens with a downsampled global view token. This avoids the quality degradation of naively resizing large images to the model's native resolution (e.g., 336×336), preserving fine text and detail needed for document understanding and OCR tasks. The tradeoff is increased token count proportional to the number of tiles.

Gemini 1.5 uses a Mixture-of-Experts transformer backbone combined with efficient attention mechanisms (including sparse attention variants) and aggressive KV-cache management to extend context to 1M tokens. Video frames, audio spectrograms, and text are tokenized into a unified sequence. Long-context retrieval capability is built during training on long documents and video data, with special positional encodings that generalize across very long sequences. The MoE architecture keeps per-token compute tractable despite the large parameter count.

Perceiver IO uses a small set of latent vectors that cross-attend to arbitrary-length byte-level inputs (images, audio, text, point clouds) in an encoder, then uses task-specific query vectors that cross-attend to the latents in a decoder to produce arbitrary-length outputs. This decouples compute from input/output dimensionality, keeping the bulk of processing in the fixed-size latent space via self-attention. It scales gracefully to high-dimensional inputs without quadratic blowup on raw input length.

InstructBLIP freezes a pretrained image encoder (EVA-CLIP) and a pretrained LLM (Vicuna or FlanT5), then trains only a Q-Former module—a transformer with learnable query tokens that extract task-relevant visual features by cross-attending to image features conditioned on the instruction text. This instruction-aware visual feature extraction allows the same frozen LLM to follow diverse visual instructions without updating its weights. InstructBLIP outperforms BLIP-2 on zero-shot benchmarks by aligning visual features to the specific instruction context.

Multimodal CoT prompting asks the model to reason step-by-step over both visual and text inputs before producing a final answer—for example, describing relevant image regions before answering a science question. Research shows multimodal CoT reduces hallucination and improves accuracy on benchmarks like ScienceQA and MathVista. Multimodal-CoT (Zhang et al., 2023) trains a two-stage model: first generating a rationale from image+question, then generating the answer conditioned on the rationale, image, and question.

Video-language models sample frames at regular or adaptive intervals, encode each frame with a vision encoder, and then apply temporal aggregation—either via temporal attention across frame tokens, 3D convolutions, or positional encodings that capture frame order. Models like Video-LLaMA and VideoChat inject temporal positional encodings and use a Q-Former that aggregates across frames. Audio tracks can be encoded separately and fused. Static VLMs treat each frame independently, missing motion, causality, and temporal reasoning that video models capture through sequence-aware encoding.

Object hallucination occurs when a VLM generates text referring to objects not present in the image. POPE (Polling-based Object Probing Evaluation) tests this with yes/no questions about object presence using three sampling strategies: random, popular, and adversarial. CHAIR (Caption Hallucination Assessment with Image Relevance) computes the fraction of hallucinated objects in generated captions. Mitigation approaches include RLHF with human feedback on factuality, contrast decoding (Visual Contrastive Decoding), and OPERA which penalizes over-attending to non-visual tokens.

Encoder-decoder VLMs use a visual encoder to extract image features, a bridging module (Q-Former in BLIP-2) to compress them, then feed both visual and text tokens into an encoder-decoder LLM (like FlanT5). Decoder-only VLMs (LLaVA, InternVL) project visual features directly into the LLM's embedding space and concatenate them with text tokens, relying on a single autoregressive decoder. Decoder-only models benefit from larger, more capable LLM bases and simpler training pipelines; encoder-decoder models separate visual compression from generation.

Visual grounding (or referring expression comprehension) requires the model to localize a specific region or object in an image given a natural language description, producing a bounding box or segmentation mask. VQA produces a textual answer to a question about an image without necessarily pinpointing a region. Grounding demands spatial reasoning and precise coordinate output; VQA can be answered from global image features. Models like Grounding DINO and GPT-4o with structured output combine language and detection heads for grounding.

GPT-4V (and earlier GPT-4 with vision) used a separate vision encoder (CLIP-like) whose outputs were projected and prepended to text tokens, making it a modular two-stage system. GPT-4o is trained end-to-end on text, images, and audio simultaneously with a unified token vocabulary, enabling real-time audio in and out without a separate speech pipeline. GPT-4o thus achieves lower latency for voice interaction, richer cross-modal reasoning, and expressive voice output that conveys emotion—capabilities not possible with a bolted-on vision adapter.

MMBench is a comprehensive bilingual (English/Chinese) benchmark covering 20 sub-tasks including object localization, attribute recognition, commonsense reasoning, and image captioning, scored via ChatGPT-based choice extraction. MME measures 14 perception and cognition tasks with binary yes/no answers. MMMU tests college-level multimodal questions across 6 disciplines requiring expert knowledge. Collectively these benchmarks replace COCO captioning as gold standards for VLM capability, covering perception, reasoning, domain knowledge, and instruction following.

Audio is typically converted to mel spectrograms and encoded by a pretrained audio encoder (e.g., Whisper encoder, EnCodec, or a dedicated audio transformer). The encoder outputs continuous frame-level embeddings that are either projected linearly into the LLM's embedding dimension or quantized into discrete tokens via a vector quantizer (VQ-VAE). Discrete audio tokens allow unified autoregressive generation of speech and text. Systems like AudioPaLM and Gemini use this approach to support speech-in/speech-out without a separate TTS pipeline.

Cross-modal retrieval involves retrieving relevant items from one modality given a query in another—e.g., finding images matching a text query or finding captions for an image. Standard evaluation uses Recall@K (R@1, R@5, R@10) on datasets like MSCOCO 5K, Flickr30K, or CIRR (composed image retrieval). Metrics measure the fraction of queries where the correct match appears in the top K results. CLIP-style models are the backbone for most cross-modal retrieval systems due to their aligned embedding spaces.

MoME extends the standard Mixture-of-Experts (MoE) transformer by routing tokens to experts specialized by modality. Image tokens route to vision experts while text tokens route to language experts, with some shared experts handling cross-modal interactions. This allows the model to scale parameters and capacity for each modality independently without proportionally increasing per-token FLOPs. It reduces interference between modalities during training and inference, improving both modality-specific quality and cross-modal tasks.

Key strategies include: (1) LoRA or QLoRA—adapting only low-rank matrices appended to attention layers; (2) freezing the LLM backbone and training only the visual projector or Q-Former; (3) replay-based methods that mix old task data with new; (4) elastic weight consolidation (EWC) penalizing changes to weights important for prior tasks; and (5) prompt tuning or adapter layers that insert small trainable modules. For most production VLMs, LoRA on the LLM with a trainable projector strikes the best balance of efficiency and retention.

SigLIP replaces CLIP's softmax-normalized contrastive loss with a sigmoid binary cross-entropy loss applied independently to each image-text pair. Each pair is treated as a binary classification (match or non-match) without requiring global normalization across the batch. This eliminates the need for very large batches to get good negatives (CLIP requires 32K+ batch sizes), enables training on TPUs or GPUs with smaller batch sizes, and achieves better performance at the same or lower compute budget. SigLIP-based backbones underlie several 2024-era VLMs.

OCR capability in VLMs is evaluated on benchmarks including OCRBench (which tests text recognition in natural images, document understanding, and handwriting), TextVQA (answering questions about text in images), DocVQA (PDF/document understanding), and ChartQA (chart comprehension requiring reading values). Metrics are typically exact match or normalized Levenshtein distance. Strong OCR performance requires high-resolution input processing, since text characters span few pixels in 336×336 images; hence HD tiling approaches are critical.

Reward models (RMs) for VLMs are trained on human preference data comparing model responses given an image and prompt—similar to text RLHF but with visual context. The RM scores responses for helpfulness, factuality, and harmlessness including absence of hallucination. VLM alignment pipelines like LLaVA-RLHF and InstructBLIP refinements use PPO or DPO with the RM signal. Multimodal RMs must handle the visual grounding of claims, making them more complex than text-only RMs. Preference datasets like POVID and RLHF-V provide curated multimodal comparison data.

Qwen-VL uses a visual encoder with a position-aware adapter that compresses each image to 256 visual tokens and prepends a special <img> tag plus an absolute image index to disambiguate which image subsequent text refers to. Multiple images can be interleaved with text in the same sequence, up to the LLM's context limit. Qwen-VL-Chat adds instruction tuning for referring to specific images by index, enabling multi-image dialogue, comparison tasks, and visual storytelling with up to several images per turn.

Grounded captioning requires the model to generate captions where each noun phrase or entity is linked to a specific bounding box or region in the image. This demands simultaneous language generation and spatial localization. Standard captioning only requires a fluent description without region attribution. Grounded captioning is evaluated on datasets like Flickr30K Entities and Visual Genome. Models need joint training on detection and language heads or structured output formats to generate coordinate tokens alongside text tokens.

A 1-minute video at 1 fps yields 60 frames; at 256 tokens/frame that's 15,360 visual tokens—far exceeding typical LLM context windows and making attention prohibitively expensive. Solutions include: (1) aggressive frame subsampling; (2) temporal pooling across frame tokens; (3) query-based resampling (Q-Former across frames); (4) sparse attention mechanisms; (5) hierarchical encoding where a coarse global encoder selects key frames for detailed encoding. Models like Video-LLaVA and MiniGPT4-Video apply these to fit within 4K–32K token budgets.

InternVL2 uses a dynamic resolution strategy: images are resized to the nearest tile configuration from a predefined set (e.g., 1×1 to 6×6 tiles of 448×448 patches). A thumbnail global view is always included. The number of tiles scales with image resolution, so simple images use fewer tiles (less compute) while document-heavy images use more. Each tile is independently encoded by InternViT-6B, then tile tokens are concatenated and fed to the LLM. This approach balances quality and cost across input resolutions.

CoCa trains a single model with two losses: a contrastive loss on unimodal image and text CLS embeddings (like CLIP) and a generative captioning loss (cross-entropy on caption tokens). The image encoder produces both a pooled embedding for contrastive learning and full spatial features for cross-attention with the text decoder. This unification means one model supports both retrieval (via contrastive embeddings) and generation (via the decoder), achieving state-of-the-art on both tasks without task-specific architecture changes.

Approaches include: (1) training or fine-tuning the LLM backbone on multilingual text so the language model natively handles multiple languages; (2) using a multilingual tokenizer (BPE with multilingual vocabulary) to avoid out-of-vocabulary issues; (3) multilingual visual instruction tuning datasets (e.g., MMBench-CN, multilingual LLaVA data); (4) translation-augmented training where captions are machine-translated; (5) language-agnostic visual encoders since CLIP encoders are somewhat language-agnostic. Models like Qwen-VL and InternVL2 achieve strong multilingual performance through multilingual LLM backbones.

Visual Prompt Tuning prepends a set of learnable continuous tokens (visual prompts) to the input sequence of a frozen vision transformer. Only these prompt tokens are updated during training while all model weights remain fixed. VPT is extremely parameter-efficient—typically fewer than 1% of model parameters—and achieves competitive accuracy on downstream vision tasks compared to full fine-tuning. Full fine-tuning updates all weights, requiring much more memory and compute and risking catastrophic forgetting. VPT is analogous to prefix-tuning in NLP.

Key considerations: (1) GPU memory—a 70B fp16 model needs ~140 GB VRAM, requiring multi-GPU tensor parallelism or quantization to int4/int8; (2) KV-cache size scales with visual token count per request, requiring careful batch size management; (3) PCIe vs. NVLink bandwidth between GPUs affects tensor parallel efficiency; (4) first-token latency driven by visual encoding and prefill; (5) throughput scaling via continuous batching (vLLM, TensorRT-LLM); (6) quantization with GPTQ or AWQ for memory reduction with minimal quality loss.

Visual Contrastive Decoding (VCD) generates two distributions at each decoding step: one conditioned on the image and text, another on text only (or on a distorted/noisy image). The final logits are the difference, amplifying predictions that are genuinely grounded in visual content over those the language model would generate from text priors alone. This reduces hallucination caused by overreliance on language statistics without additional training. VCD is a training-free inference-time technique compatible with any autoregressive VLM.

LLaVA-1.5 showed that replacing the original LLaVA linear projector with a simple two-layer MLP projector substantially improved performance across multimodal benchmarks, despite adding minimal parameters. Combined with using a CLIP-ViT-L/336 encoder and a higher-quality instruction dataset with academic task data, these small changes yielded state-of-the-art results at the time. The paper highlighted that the projector design and data quality matter more than architectural complexity, challenging assumptions that more elaborate bridging modules (like Q-Former) are always necessary.

Scene graph generation predicts a graph where nodes represent detected objects with attributes and edges represent relationships between them (e.g., 'dog on mat', 'woman wearing hat'). It provides structured, queryable representations of image content beyond flat captions. Scene graphs power downstream tasks like visual question answering, image retrieval by relationship, and image-conditioned reasoning. Modern VLMs can implicitly do scene graph reasoning, but explicit scene graph models (like Graph RCNN) produce structured JSON-like outputs that enable symbolic reasoning and database queries.

PaliGemma (Google, 2024) combines a SigLIP-So400m vision encoder with a Gemma 2B or 9B language model, projecting visual tokens via a linear layer into the Gemma embedding space. It is notably compact and designed for fine-tuning rather than general instruction following out-of-the-box. PaliGemma achieves strong results when fine-tuned on specialized tasks like remote sensing, medical imaging, and document understanding, outperforming larger models on narrow tasks. Its fully open weights and permissive license make it popular for research fine-tuning.

Multimodal in-context learning (ICL) provides a few image-text demonstration examples in the context window before the test query, guiding the model's output format and reasoning style without weight updates. Models like Flamingo pioneered this capability. Limitations include: (1) context window saturation as images use many tokens; (2) sensitivity to demonstration order and selection; (3) performance gap vs. fine-tuned models on specialized tasks; (4) inability to handle tasks requiring precise spatial understanding from in-context examples. Despite limitations, it enables rapid task adaptation without training.

Unified models handle both image understanding (captioning, VQA) and image generation within a single architecture. Show-o (2024) uses a unified transformer where text tokens are processed autoregressively and image tokens via discrete diffusion, supporting text-to-image, image-to-text, and editing tasks. Janus (DeepSeek) decouples visual encoding paths for understanding vs. generation within one model. Chameleon (Meta) uses a fully tokenized approach with a shared VQ-VAE vocabulary for both. Unification reduces inference infrastructure complexity and enables tasks like image editing with text instructions.

Attention sinks are a phenomenon where early tokens (especially the initial position tokens) accumulate disproportionate attention weight regardless of content, acting as 'dump' tokens that stabilize softmax normalization. In long-context VLMs, this wastes KV-cache slots. StreamingLLM proposes retaining a small window of recent tokens plus the initial sink tokens in the KV-cache, enabling infinite-length inference with fixed memory. For multimodal use cases like streaming video, this allows perpetual processing by evicting old frame tokens while keeping sinks and a recent window.

Web-scraped image-text pairs (CommonCrawl, LAION) are extremely noisy—mismatched alt-text, watermarks, NSFW content, duplicate images. Curation heuristics include: CLIP similarity threshold (keeping pairs with cosine similarity > 0.28–0.3); perceptual hash deduplication; language detection for target language; NSFW classifiers; aesthetic quality scores (via LAION-Aesthetics); face blur for privacy; and watermark/text overlay classifiers. DataComp showed that heavy filtering of small high-quality subsets outperforms training on large unfiltered corpora, motivating quality-over-quantity curation.

3D scene understanding requires reasoning about depth, spatial layout, object pose, and volumetric structure—information not directly available in 2D images. Inputs may be point clouds (LiDAR), multi-view images, or RGBD frames. Models like 3D-LLM and PointLLM encode point cloud features via specialized encoders (PointNet++, 3D transformer) and feed them into an LLM for spatial QA, navigation instructions, and manipulation planning. Unlike 2D VLMs, 3D models must handle unordered, sparse, variable-density inputs and 3D coordinate outputs.

Mantis (2024) addresses multi-image understanding by training a VLM specifically on diverse multi-image instruction datasets including image comparison, sequential reasoning, and multi-panel analysis. It uses an interleaved image-text format with standard LLaVA-style architecture but fine-tunes on Mantis-Instruct—a curated dataset of multi-image tasks sourced from NLVR2, Spot-the-Diff, Birds-to-Words, and synthetic data. Mantis substantially outperforms single-image VLMs adapted for multi-image use, demonstrating that task-specific multi-image training data is the key bottleneck.

Spatial reasoning evaluation benchmarks include: CVBench (close vs. far, left vs. right localization), What's Up (absolute spatial relationships), and SpatialBench. Tasks test whether the model correctly identifies object positions relative to each other and the camera. Metrics are accuracy on multiple-choice or binary questions. State-of-the-art VLMs still significantly underperform humans on precise spatial tasks. Fine-tuning on spatially annotated data (Visual Genome relationships, SpatialSense) and using high-resolution inputs improve performance.

Dense video captioning localizes and describes all events within a video, producing (timestamp_start, timestamp_end, caption) tuples for every distinct activity. Standard video captioning generates a single global description. Dense captioning requires temporal event detection (identifying event boundaries) in addition to language generation, making it significantly harder. Vid2Seq (2023) achieves this by prepending special time tokens to text tokens in a sequence-to-sequence transformer, enabling joint event localization and description generation without separate detection and captioning stages.

Synthetic multimodal instruction data is generated by: (1) prompting GPT-4V or Claude with images to create detailed QA, conversation, and reasoning chains—used in LLaVA-150K; (2) using structured metadata (COCO captions, bounding boxes, object labels) as context for GPT-4 text-only to generate visual questions without image API access—LLaVA original approach; (3) programmatically generating diagrams, charts, and math figures with known answers. Synthetic data fills coverage gaps in human-annotated datasets for rare skills like chart reading, spatial reasoning, and code-in-image tasks.

Token-level image generation quantizes images into discrete tokens via a VQ-VAE (e.g., VQGAN, LlamaGen) and trains an autoregressive LLM to predict these tokens—treating image generation like language modeling. This allows unified text and image generation in one model. Pixel-level approaches generate continuous pixel values directly, typically using diffusion models conditioned on LLM embeddings (e.g., DALL-E 3). Token-level is architecturally simpler for unification; pixel-level diffusion typically produces higher visual fidelity. Show-o hybridizes both for different granularities.

OPERA identifies that VLM hallucinations often coincide with tokens where attention over-concentrates on a small set of summary tokens (column sinks) rather than image tokens. OPERA modifies beam search by adding a penalty term that discourages over-reliance on such summary positions. It also implements a rollback-and-retrospection mechanism that reallocates attention when over-trust is detected, prompting the model to re-examine visual information. OPERA is training-free, applicable at inference time, and reduces hallucination on CHAIR and POPE without additional training.

BLIP-Diffusion uses a BLIP-2 Q-Former to extract subject-specific visual embeddings from reference images, then conditions a latent diffusion model on these embeddings to generate new images of the same subject in different contexts or styles. This enables zero-shot subject-driven generation without test-time optimization (unlike DreamBooth). The Q-Former acts as a bridge between the vision encoder and the diffusion model's cross-attention conditioning. It supports tasks like subject stylization, background replacement, and multi-subject composition guided by text prompts.

Mathematical reasoning tasks—geometry proofs, equation solving from diagrams, graph reading—often involve fine details like subscripts, operators, tick marks, and precise graph values that are invisible at low resolution (224×224 or 336×336). Studies show significant performance gains on MathVista and GeoQA when switching from standard to high-resolution inputs with tiling. InternVL2 and LLaVA-HD gain 10–20 percentage points on math benchmarks through HD processing. Additionally, large text in images (math equations) benefits from character-level fidelity that is lost when compressed to few visual tokens.

Variable-length image inputs produce variable visual token counts per request. In continuous batching (used by vLLM and TGI), requests are dynamically added to running batches. For VLMs: (1) the image is encoded offline before joining the batch queue; (2) visual tokens are prepended to the sequence; (3) paged attention manages the KV-cache with fixed-size pages accommodating variable token counts; (4) requests are padded within a batch to the longest sequence or bucketed by token count to minimize padding waste. Speculative decoding is harder with visual prefixes but possible with draft models that share the visual encoder.

Cambrian-1 (NYU, 2024) demonstrates that using multiple visual encoders—combining a semantic-focused encoder (CLIP, SigLIP) with a spatial/detail-focused encoder (DINOv2, ConvNeXt)—significantly improves performance across multimodal benchmarks. It introduces a Spatial Vision Aggregator (SVA) module that selects and merges features from multiple encoders via cross-attention with LLM queries. Ablations show different encoders excel at different tasks: CLIP for semantic reasoning, DINOv2 for spatial tasks. Cambrian-1 also releases a curated instruction-tuning dataset (Cambrian-7M).

POPE (Polling-based Object Probing Evaluation) constructs yes/no binary questions of the form 'Is there a [object] in the image?' where negative questions sample non-existent objects by three strategies: (1) random—randomly sampled objects from MSCOCO; (2) popular—frequently occurring MSCOCO objects most likely to be confused; (3) adversarial—objects that co-occur most frequently with objects actually in the image, maximally exploiting co-occurrence biases. Accuracy and F1 are reported. The adversarial split is hardest as it exploits the model's tendency to guess based on plausible co-occurrences.

ViT position embeddings encode the spatial location of image patches, enabling the transformer to preserve spatial structure. Standard ViT uses learnable 2D absolute position embeddings trained at a fixed resolution. For VLMs needing variable resolutions, interpolation of position embeddings (bicubic for patch grids) is used to generalize to higher resolutions. Some models use 2D RoPE (rotary position embeddings) or NaViT (patches sampled without resizing) to handle arbitrary aspect ratios. Dynamic resolution models reset or re-learn position embeddings per tile to avoid resolution-train/test mismatch.

Compositional understanding tests whether VLMs understand structured relationships like attribute binding ('red cube on left of blue sphere') rather than just recognizing individual objects. ARO (Attribution, Relation, Order) benchmark and Winoground test compositional sensitivity. VLMs often fail by focusing on noun identity while ignoring spatial/attribute relations. Improvements come from compositional data augmentation (creating hard negatives by swapping attributes or relations), fine-grained contrastive tuning (NegCLIP), and chain-of-thought prompting that explicitly reasons about each component before producing a final answer.

In language models, BPE segments text into subword tokens. In multimodal systems, image features are projected to the same embedding dimension as text token embeddings and concatenated in sequence—they bypass the text tokenizer entirely. The LLM processes a unified sequence of text BPE embeddings and projected visual embeddings. For models using discrete image tokens (Chameleon, LlamaGen), image VQ codes are added to the BPE vocabulary as special tokens, allowing the LLM's embedding table to learn both text and image token representations jointly.

Florence-2 (Microsoft, 2024) is a unified VLM trained with a sequence-to-sequence objective on 126M annotated image-task pairs using FLD-5B, a large Florence dataset. Region/box outputs are represented as quantized coordinate tokens appended to the vocabulary (e.g., <loc_350> for x=350/1000 of image width). The model learns to generate these location tokens interleaved with text, enabling object detection, grounding, segmentation prompts, and captioning with location annotations in one model. Post-processing parses location tokens to pixel coordinates.