Lecture 5 – Multimodal Alignment (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
The session links the alignment topic to project mentoring, compute resources, and the ongoing fusion homework.
The lecture opens by connecting theory to execution: each project will receive mentoring, teams have limited model or compute credits, and Homework 2 supplies hands-on fusion practice.
The practical implication is to budget experiments deliberately—establish inexpensive baselines first, then spend compute on hypotheses that the baseline evidence justifies.
Early and late fusion trade representational expressiveness against simplicity and interpretability.
Late fusion: encode each modality first, then combine semantic features or predictions. It is lightweight and easier to inspect, but may discard fine-grained cross-modal interactions.
Early fusion: combine raw or low-level inputs so representation learning and interaction modeling happen jointly. It can express richer dependencies, at the cost of more parameters, optimization difficulty, and opacity.
These endpoints define a spectrum rather than a binary choice.
Additive, multiplicative, tensor, and low-rank formulations provide progressively richer fusion interactions.
Fusion complexity can be increased in controlled steps:
1. Additive terms model each modality’s independent contribution.
2. Multiplicative terms model conditional interactions, so one modality changes another’s effective weight.
3. Outer products collect bias, unimodal, bimodal, and higher-order terms in one tensor.
4. Low-rank factorization represents the large weight tensor as a sum of a few separable factors.
The rank becomes a tunable efficiency–fidelity trade-off; full rank recovers the unrestricted tensor, while modest rank is often sufficient in practice.
Gated fusion makes modality weights depend on the current example and can incorporate domain knowledge.
A fixed coefficient assumes a modality is equally reliable for every example. Dynamic fusion replaces that coefficient with a learned gate such as \(g_A(x_A,x_B)\), producing example-dependent contributions.
Gates need not be symmetric or purely learned from scratch. Prior knowledge can designate a stable primary modality—often language—and concentrate adaptation on modalities whose usefulness varies with context.
EMAP tests whether an apparently complex fusion model actually uses non-additive cross-modal interactions.
Diagnostic question: does a complex architecture exploit interactions that an additive model cannot represent?
EMAP approximates \(f(x_A,x_B)\) with marginal expectations of the form \(f_A(x_A)+f_B(x_B)\), then measures the prediction or performance gap.
- A tiny gap means most learned behavior is additive.
- If both models perform well, the task may not contain important non-additive structure.
- If both perform poorly, the structure may exist but neither model has captured it.
Architecture complexity alone is therefore not evidence of interaction learning.
Simple additive, early-fusion, late-fusion, and concatenation baselines are essential controls before interpreting small gains from complex models.
Treat simple fusion methods as scientific controls, not perfunctory baselines.
- Compare against concatenation, additive fusion, and both early and late fusion.
- Report variability or confidence intervals where possible.
- Decide whether an absolute gain is operationally meaningful for the domain.
A complex model that barely beats its additive projection has not demonstrated that its elaborate interaction mechanism was necessary.
Stage-wise fusion fits unimodal, pairwise, and higher-order residuals sequentially so added complexity cannot degrade the fitted training objective.
Stage-wise fusion performs an interaction-order decomposition:
1. Fit and combine unimodal predictors.
2. Fit bimodal interactions to the residual \(y-\hat y_{uni}\).
3. Fit trimodal interactions to \(y-\hat y_{uni}-\hat y_{bi}\).
Because every new stage targets only unexplained error, it should improve or preserve the fitted objective. Stop when the residual is acceptable; this prevents an unnecessarily large multimodal model from making performance worse.
The best fusion method should capture exactly the interactions supported by the data while respecting compute and interpretability constraints.
The design goal is all necessary interactions, nothing more.
Measure task performance against compute, parameter count, and interpretability as fusion moves from simple late combinations toward large early-fusion models. There are few universal theoretical guarantees for non-convex deep networks, so empirical curves and diagnostics matter.
A promising formal direction is to estimate how much information is unique to each modality versus synergistic across modalities, then choose model capacity that matches that structure.
Multimodal alignment models cross-modal connections in discrete, continuous, and contextualized settings.
Alignment has three related roles:
- Discrete alignment: match identifiable units across modalities.
- Continuous alignment: discover correspondences when boundaries and temporal resolution are unknown.
- Contextualized representation learning: use alignment internally so each feature is interpreted in light of relevant features from the other modality.
This lecture focuses on alignment as the direct objective; later transformer material uses it as a representation-building mechanism.
Discrete alignment is often one-to-many or relational rather than a simple one-to-one lookup.
Object nouns often admit clear region-level matches, but actions expose the limits of one-to-one alignment. Reading is supported by a relation among a person, gaze direction, posture, and a target such as a newspaper.
Statistical co-occurrence can discover that the text token newspaper repeatedly accompanies visual newspapers, while semantics explains the shared concept. A complete alignment model must capture point correspondences and relations among several elements.
Alignment preserves modality-specific embeddings and coordinates them with a similarity function, unlike fusion’s single joint representation.
Fusion: \((x_A,x_B)\rightarrow z\), one combined representation.
Alignment: \(x_A\rightarrow z_A\) and \(x_B\rightarrow z_B\), with \(G(z_A,z_B)\) enforcing consistency.
The separate encoders can respect each modality’s structure, while a differentiable similarity or distance objective coordinates them. This distinction matters because aligned embeddings remain independently usable for retrieval and transfer.
Similarity may align individual points, entire distributions, or the relational geometry within representation spaces.
Choose the alignment target to match the scientific goal:
- Pointwise: cosine or kernel similarity makes paired instances close.
- Distributional: correlation aligns global variation across modalities.
- Relational: match pairwise or higher-order geometry, preserving structures such as apple–orange proximity and their distance from unrelated concepts.
Relational alignment can require only set-level correspondence, whereas pointwise alignment needs explicit instance pairs. The choice of \(G\) therefore determines both the learned invariant and the supervision required.
The granularity of pairing supervision must match the granularity of the alignment loss.
Supervision is not a single yes-or-no property. It is required at the resolution enforced by the loss.
Point-level objectives need labeled pairs; set- or geometry-level objectives may only need corresponding collections. Once \(G\) is defined, its loss can backpropagate through both encoders, optionally starting from pretrained modality models.
Contrastive learning brings positive cross-modal pairs together and separates mismatched negative pairs.
Given a batch of paired examples:
- matched image–caption pairs are positives;
- nonmatching combinations are negatives;
- optimize the encoders so positives have high similarity and negatives low similarity.
This explicitly learns shared cross-modal structure while leaving the two embeddings separate.
Aligned spaces enable cross-modal arithmetic, and CLIP scales contrastive learning with an in-batch similarity matrix.
Alignment makes concepts interoperable across modalities: vector arithmetic can remove a text concept from an image embedding and add another before retrieval.
CLIP operationalizes this at scale:
1. Encode \(N\) images and \(N\) captions.
2. Form the \(N\times N\) cosine-similarity matrix.
3. Treat diagonal cells as matched pairs and off-diagonal cells as in-batch negatives.
4. Maximize each positive relative to the row or column’s alternatives with an InfoNCE-style objective.
CLIP-style aligned encoders support retrieval and open-vocabulary classification without a fixed output head.
A shared embedding space turns similarity into a general interface:
- Cross-modal retrieval: rank captions for an image or images for text.
- Open-vocabulary classification: write arbitrary class prompts and select the closest text embedding.
- Pairwise comparison: replace a single-item score with a differentiable relational score when the task requires it.
The flexibility of the label set is a major advantage, but errors in counting or composition show that aligned semantics are not equivalent to full reasoning.
InfoNCE can be interpreted as distinguishing the true joint distribution from randomly paired marginal samples, thereby maximizing a lower bound on mutual information.
Statistical view of contrastive learning:
- Positive pairs are sampled from the joint distribution \(p(A,B)\).
- Random negatives approximate the product of marginals \(p(A)p(B)\).
- The critic learns to separate these sources, implicitly estimating their density ratio.
- InfoNCE maximizes a lower bound on \(I(A;B)\), with a bound involving the contrastive loss and \(\log N\).
Thus contrastive alignment can recover, at best, the information genuinely shared by the modalities.
Alignment helps downstream tasks only when the shared information it learns is neither insufficient nor excessively irrelevant.
Contrastive alignment is ideal when the shared region \(I(X_1;X_2)\) matches the target-relevant information.
- Too little overlap: the aligned representation lacks signal needed for \(Y\).
- Just-right overlap: shared information is sufficient and compact.
- Too much overlap: nuisance correlations dominate, reducing task specificity.
This explains why increasing mutual information is not automatically beneficial; the objective should preserve the overlap that matters for the downstream task.
Fusion and alignment can be ordered or combined in different ways, while data augmentation can reshape which information is treated as shared.
There is no universal align-then-fuse or fuse-then-align rule.
Alignment is a strong training signal when captions and images share exactly the semantics needed for the task; fusion becomes necessary when prediction also depends on modality-unique or synergistic information.
Data augmentation can edit the overlap itself: transformed views encourage the representation to preserve invariants such as object identity. That strategy relies on a domain assumption—if the transformation destroys target-relevant content, the learned invariance is harmful rather than helpful.
Enjoy Reading This Article?
Here are some more articles you might like to read next: