Lecture 6 – Large Multimodal Models (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
Fusion combines modalities into one joint representation, whereas alignment keeps modality representations separate and relates their elements.
Fusion versus alignment
- Fusion: Map multiple inputs into one joint representation that captures cross-modal interactions. The operator may be additive, multiplicative, or dynamically weighted.
- Alignment: Keep modality representations separate while learning which samples or elements correspond, such as a word and an image region.
Alignment may itself be the output, or it may be a latent step that contextualizes features for a downstream predictor.
Contrastive learning aligns positive cross-modal pairs against negative pairs and emphasizes information shared by the modalities.
Contrastive alignment
Given representations \(z_A\) and \(z_B\), increase similarity for true paired samples and decrease it for randomly combined negatives. Positives come from the joint distribution \(p(x_A,x_B)\); negatives approximate \(p(x_A)p(x_B)\), connecting the objective to shared information between modalities.
### Task relevance matters
Alignment is most useful when the shared region contains what the label requires. Insufficient overlap loses signal, while aligning a much larger shared region can preserve nuisance variation that is irrelevant to the task.
Representation fission supplements a shared aligned representation with modality-specific representations for unique information.
Factor shared and unique information
Representation fission learns at least three components for two modalities:
1. A shared component trained from image-text positive and negative pairs.
2. A vision-specific component trained by contrasting augmented views of the same image against other images.
3. A language-specific component trained from meaning-preserving text variants against unrelated captions.
The factorization retains shared objects and people while preserving modality-only evidence such as visual texture, depth, syntax, and grammar.
Global alignment can jointly learn representations and infer unknown pairings when individually paired multimodal data is unavailable.
Alignment without known pairs
Ordinary contrastive learning assumes supervision at the same resolution as the desired alignment. With separate image and text collections, the positive pairs are latent.
Global alignment alternates or jointly optimizes two coupled quantities: the representation parameters that determine similarity and an assignment matrix indicating candidate pairs. A hard formulation searches for a maximum-weight bipartite matching; relaxed correspondence variables make linear-programming-style solutions possible. At convergence, the method aims to recover both the pairing and the aligned representation space.
Optimal transport generalizes hard one-to-one alignment to soft distributions of correspondence weights.
Soft rather than one-to-one correspondence
A hard matching selects one partner for each element. A soft alignment assigns a distribution of weights from an element in modality \(A\) to several candidates in modality \(B\). Optimal transport optimizes this correspondence matrix under mass constraints and a similarity or cost objective.
Use it when relationships are diffuse, many-to-many, or available only at the distribution level rather than as explicit paired samples.
Continuous modalities become easier to align after semantic segmentation or learned discretization into tokens.
Two routes from continuous signals to alignable units
- Segment boundaries: Use action segmentation for video or change-point detection for time series to isolate semantically meaningful intervals while ignoring fluctuations that are merely noise.
- Discretize representations: Cluster continuous speech features and predict cluster IDs during masked pretraining, avoiding difficult raw-signal regression. Vector-quantized autoencoders similarly map image, video, or audio features to a learned codebook.
The resulting speech or visual tokens can be aligned with text tokens and later decoded back to the original modality.
The Platonic representation hypothesis proposes that independently trained models converge toward compatible representations of a shared reality.
Emergent alignment hypothesis
The Platonic representation hypothesis says that capable networks may converge toward a shared statistical model of reality even when their modalities, data, architectures, and objectives differ and no alignment loss connects them.
Reported probes compare hidden states from independently trained open language models and DINO vision encoders on image-caption data. Alignment rises as language performance and vision-model scale improve. This is evidence consistent with a shared-world-model interpretation, but the lecture stresses that it remains a hypothesis rather than a settled result.
Claims of emergent alignment depend strongly on the similarity metric and on whether global or local geometry is measured.
What does similar mean?
The global analysis compares relational geometry: build a kernel or covariance-like matrix among all image embeddings, build another among all text embeddings, and measure how alike those two structures are. It does not require every apple image to sit next to the word apple.
Follow-up calibration removes confounders and can erase the reported global trend, yet local neighborhood structure still appears aligned. The practical lesson is methodological: conclusions about emergent alignment can reverse when the similarity measure, scale, or calibration changes.
Large multimodal models extend language-model interaction to multimodal understanding, reasoning, generation, and multi-turn use.
Target capabilities
Large multimodal models should classify audiovisual cues, answer open questions, explain with prior visual or auditory evidence, generate future scenes, and reason about counterfactual changes across multiple turns.
### Three-part blueprint
1. Learn strong multimodal representations that encode fusion and alignment.
2. Adapt a pretrained language model so text generation conditions on those representations.
3. Add image, audio, video, or other output generators.
The long-term system is interactive across both multimodal input and multimodal output, not a text-only model with an image attachment.
Transformers provide a common sequence-based architecture for contextualized multimodal representations.
Put modalities into a sequence interface
Text becomes tokens, a static image becomes patches, video becomes frames, and audio or sensors become time-indexed elements. Transformers can then use attention to model within-stream structure and cross-stream correspondences.
The alignment matrix is usually an internal mechanism, not the final product. It contextualizes representations used for classification, detection, question answering, and dialogue. Sarcasm is a motivating case because words alone may be ambiguous while vocal emphasis, facial behavior, and learned scene priors disambiguate intent.
Cross-modal attention projects two sequences into a common dimension, learns an alignment matrix, and uses it to contextualize one stream with the other.
Cross-modal attention mechanics
Project text and nonverbal sequences into a shared width \(d\), then compute
\(A = \\operatorname{softmax}((X_1W_Q)(X_2W_K)^T / \\sqrt{d})\).
The \(3 \\times 4\) example assigns each of three words a probability distribution over four nonverbal events. Scaling by \(\\sqrt{d}\) stabilizes magnitudes; row-wise softmax makes weights nonnegative and sum to one.
Next compute \(H_1=A(X_2W_V)\). Each word receives a weighted mixture of nonverbal values, so privilege can encode vocal emphasis and eye rolling. A sarcasm label trains the whole latent alignment and contextualization pathway end to end.
Positional encoding, token granularity, and directional cross-attention determine what multimodal transformers can align.
Practical representation choices
- Time: Positional encodings let attention distinguish an eye roll that follows a word from one that precedes it.
- Granularity: Fine visual tokens may capture subtle facial behavior but cost more, especially across many video frames. Learned speech or facial-landmark features can supply semantic tokens instead of raw samples.
- Interpretability: Word-to-patch weights often highlight the named object or region.
- Direction: Vision-to-language and language-to-vision attention are separate transformations and need not learn the same map; models commonly combine both with unimodal self-attention.
Multimodal transformer pretraining mixes local masking objectives with global image-text matching and may align features before fusion.
Complementary pretraining signals
- Masked language modeling: Recover hidden caption tokens using visible words and image regions.
- Masked image modeling: Recover hidden visual content using remaining regions and the caption.
- Image-text matching: Classify whole image-caption pairs as matched or mismatched.
An align-before-fuse variant first applies CLIP-like contrastive learning to the unimodal encoders, then passes already aligned features into the multimodal transformer. These objectives combine local and global correspondence, although the lecture notes that architecture choices remain highly empirical.
A lightweight adapter can map frozen modality features into a frozen language model’s token space and unlock multimodal prompting.
Adapter pattern
1. Keep a strong language model frozen.
2. Encode an image with a pretrained vision backbone.
3. Learn a lightweight projection from visual-feature width to the language model’s token-embedding width.
4. Prepend the projected visual tokens to the text context and train next-token prediction on captions.
Only the bridge may need updating, making training efficient. Once grounded, the model can answer questions and use multimodal in-context examples, including binding an unfamiliar word such as dax to the matching image rather than merely repeating memorized labels.
Early systems such as Flamingo and MiniGPT-4 illustrate how a small trainable bridge can connect strong frozen vision and language backbones.
Frozen backbones, trainable bridge
Flamingo demonstrates grounded multi-turn visual dialogue and can distinguish an object from misleading text placed on it. MiniGPT-4 makes the adapter recipe explicit: a frozen vision transformer and Q-Former encode the image, one linear layer converts those features, and a frozen Vicuna model generates language.
### Two training stages
- Alignment pretraining: Learn the bridge from abundant image-caption pairs.
- Instruction tuning: Train on image, user-instruction, and expert-response triples so the system follows useful tasks rather than only describing images.
Adapter-based multimodal models scale through abundant paired pretraining data, smaller instruction datasets, and modality-specific encoders.
Scaling and extending the recipe
Modality-specific encoders and adapter tokens can connect point clouds, additional languages, and other inputs to a language model. The lecture cites image-caption corpora at the scale of billions of pairs for broad alignment, followed by smaller curated image-task-answer datasets for behaviors such as mathematical reasoning or medical-image interpretation.
### Lecture synthesis
Multimodal transformers align sequence elements to build contextualized features. Lightweight adapters inject those features into a mostly frozen language model. Caption pretraining establishes grounding, instruction tuning establishes usefulness, and a third stage is still needed to move from multimodal input with text output to genuinely multimodal generation.
Enjoy Reading This Article?
Here are some more articles you might like to read next: