Lecture 4 – Multimodal Fusion (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
Multimodal fusion learns joint representations whose design depends on the heterogeneity and abstraction level of the input modalities.
The design problem
- Goal: Learn a joint representation that uses interactions among modalities.
- Why fusion is difficult: Raw image, language, audio, and other signals can be structurally heterogeneous.
- Core trade-off: Encoding each modality first produces more homogeneous semantic features and simplifies fusion, but fixes more of the representation in advance. Fusing closer to raw data demands a richer fusion mechanism yet leaves more interaction patterns available to learn.
Redundancy, uniqueness, and synergy describe three distinct ways information can be distributed across modalities.
Three interaction regimes
- Redundancy: Multiple modalities repeat the same task-relevant evidence; positive words plus a smile reinforce positive sentiment.
- Uniqueness: One modality carries evidence absent from another, so the model must identify the informative stream and suppress irrelevant input.
- Synergy: The label becomes apparent only after combining modalities; positive words paired with anger can signal sarcasm.
These regimes call for different fusion behavior rather than one universally optimal operator.
Unimodal encoders can reduce heterogeneity before feature-level fusion and may be trained jointly, pretrained, or frozen.
Encode before fusing
A vision stream might use a CNN or vision transformer, while language might use word embeddings, BERT, or another language model. These encoders perform much of the work of reconciling heterogeneous inputs by mapping them to semantically meaningful features.
- Joint training: Update the unimodal encoders and fusion network end to end for the target objective.
- Transfer setting: Use pretrained encoders and optionally freeze them, then learn only the fusion and prediction layers.
Early and late fusion trade expressive cross-modal modeling against interpretability and parameter efficiency.
Where fusion happens
- Early fusion: Concatenate raw inputs or features before prediction. It is expressive because the predictor can learn many cross-feature interactions, but it tends to be larger and more opaque.
- Late fusion: Produce one prediction per modality and combine those outputs, such as by voting or aggregating softmax scores. It is easier to inspect and uses smaller unimodal predictors, but cannot model rich interactions before the decision.
- Intermediate fusion: Place one or more fusion operations between these endpoints to balance expressiveness, size, and interpretability.
A univariate fusion model separates bias, unimodal additive effects, cross-modal interaction, and residual error.
A minimal interaction model
For scalar modalities, the lecture uses
\(Y = W_0 + W_1X_A + W_2X_B + W_3X_AX_B + \\epsilon\).
- \(W_0\) is the baseline prediction.
- \(W_1X_A\) and \(W_2X_B\) are additive unimodal effects.
- \(W_3X_AX_B\) is the cross-modal interaction.
- \(\\epsilon\) is residual error.
In the worked example, \(X_A\) is the percentage of time a reviewer smiles, \(X_B\) indicates professional-critic status, and \(Y\) is the book score, making every coefficient directly interpretable.
Purely additive fusion changes modality contributions independently and therefore produces parallel response functions in the worked example.
What additive fusion can express
With \(Y = W_0 + W_1X_A + W_2X_B\), changing \(X_B\) only adds a constant offset; it cannot change how \(X_A\) relates to \(Y\). The fitted critic and non-critic lines are therefore parallel.
- Smiling has approximately the same positive slope, \(1.19\), in both groups.
- Setting the critic indicator to one shifts the expected score downward by about \(1.69\).
Additive fusion captures independent contributions, not conditional relationships between modalities.
A multiplicative term lets one modality change the effect of another, capturing a genuine interaction rather than a fixed shift.
Multiplication creates conditional effects
Adding \(W_3X_AX_B\) makes the slope on \(X_A\) depend on \(X_B\). When \(X_B=0\), the slope is \(W_1\); when \(X_B=1\), it becomes \(W_1+W_3\). The second modality can therefore rotate the response line instead of merely shifting it.
In the example, critics who barely smile give especially low scores, while increasing smiles raise their scores more sharply. Additive terms are first-order effects; multiplicative terms encode second-order interactions, and practical fusion models can contain both.
Bilinear fusion generalizes scalar multiplication by using an outer product to represent every pairwise interaction between feature dimensions.
From scalars to vectors
An element-wise product keeps only aligned interactions: two four-dimensional vectors yield four products. Bilinear fusion instead computes an outer product, producing a \(4 \\times 4\) matrix with all 16 pairwise feature interactions.
Consequence: Bilinear fusion is more expressive because feature \(i\) in one modality may interact with any feature \(j\) in the other; matching coordinate positions are not assumed to share meaning.
Appending a constant one before outer products packages bias, unimodal, bimodal, and higher-order interactions into one tensor-fusion representation.
Tensor fusion includes every interaction order
Append a constant \(1\) to each modality vector before taking outer products. For two modalities, the resulting augmented matrix contains four components: the full bimodal interaction block, the \(X_A\) unimodal terms, the \(X_B\) unimodal terms, and the constant bias.
With three modalities, the same construction contains trimodal, all three bimodal, all three unimodal, and bias terms in one tensor. The lecture applies this to spoken language, facial video, and vocal audio for sentiment and emotion recognition.
Low-rank factorization preserves much of tensor fusion’s expressiveness while avoiding explicit high-dimensional interaction tensors and huge projection matrices.
Controlling the dimensional explosion
An augmented \(5 \\times 5 \\times 5\) tensor already has 125 entries, and projecting it to a 10-dimensional output requires a \(125 \\times 10\) weight array. Both representation size and downstream parameters grow rapidly.
### Low-rank remedy
Approximate the large projection weights as a sum of a small number of outer-product factors, then rearrange the computation so each original modality is multiplied by its factor directly. The full interaction tensor never needs to be materialized. This works well when learned weights have low effective rank and connects conceptually to parameter-efficient methods such as LoRA.
Dynamic fusion replaces dataset-wide static weights with gates or attention functions that adapt modality weights to each input.
Sample-dependent fusion
Static coefficients use the same weighting for every example. Dynamic fusion instead learns gates such as \(g_A(X_A,X_B)\) and \(g_B(X_A,X_B)\), then uses them to modulate the modality features for the current sample.
A gate may be soft or hard, and its input may be the modality it controls, another modality, or the complete multimodal observation. Attention is one implementation of this general idea: the fusion rule adapts to which evidence is useful now.
Shifting-based fusion treats a strong primary modality asymmetrically and lets auxiliary modalities make context-sensitive adjustments.
Preserve a strong anchor modality
Shifting-based fusion is deliberately asymmetric: keep a powerful primary representation, often language, largely intact and let auxiliary visual or auditory evidence make a small contextual adjustment.
The word wow is lexically informative but polarity-ambiguous. A positive or negative facial expression can shift its language embedding toward the intended interpretation. This design exploits nonverbal evidence without discarding the structure already learned by a strong language model.
Enjoy Reading This Article?
Here are some more articles you might like to read next: