Lecture 7 – Multimodal Generation (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
Course project work should establish a credible data and baseline foundation before proposing a new method.
Establish the empirical foundation first
The course workflow mirrors a sound research sequence:
1. fine-tune an existing vision-language model on the project data;
2. move toward custom encoders only when the modality demands them;
3. process the data, reproduce relevant state-of-the-art baselines, and analyze results before claiming a new contribution.
Use one primary dataset and at most one backup. A useful benchmark sits between two bad extremes: it is not so saturated that improvement is implausible, and not so weak that an unrelated bottleneck dominates. The intended method should address the next unsolved delta on that dataset.
Cross-attention aligns elements across modality sequences to produce contextualized multimodal representations.
Alignment creates cross-modal context
Given language elements and nonverbal elements, a multimodal transformer learns a token-by-token alignment matrix. After normalization, each row can be interpreted as a distribution over which elements in the other modality matter.
If a word representation aligns with 0.7 vocal emphasis and 0.3 eye rolling, the updated word feature combines those signals. A superficially positive word can thereby acquire sarcastic meaning. Cross-attention is thus doing two jobs: discovering semantic correspondence and using that correspondence to contextualize each modality’s representation for downstream prediction.
The depth at which modalities are fused trades expressive flexibility against efficiency and possible information loss.
Early versus late cross-modal fusion
- Early fusion: attend over relatively raw token, patch, or signal embeddings. It preserves flexibility and lets the joint model discover features, but increases dimensionality and parameter cost.
- Late fusion: first extract semantically meaningful unimodal features, then run a smaller cross-attention module. It is efficient and can exploit domain knowledge, but discarded information cannot be recovered downstream.
The best choice follows encoder maturity. Vision and audio often have strong pretrained raw-data embeddings; specialized sensors and time series may still benefit from engineered variability or signal-processing features.
Small adapters can connect frozen unimodal encoders to frozen language models through alignment followed by instruction tuning.
Parameter-efficient multimodal adaptation
Keep the strong unimodal encoder and language model mostly frozen, and train a small adapter—sometimes only a linear projection—from visual feature dimension to language-token dimension.
Training has two distinct stages:
1. Alignment: image-caption pairs teach the adapter to express visual features in the language model’s representation space.
2. Instruction tuning: image, instruction, and expert-answer triplets teach the system to answer questions, advise, and solve tasks instead of merely captioning.
The surprising effectiveness of a simple adapter shows that pretrained representations can often be connected cheaply, though the result is strongest for modality content that language describes well.
Language-centered multimodal systems remain constrained by what data and sensors make expressible, motivating embodied and self-evolving AI.
Beyond the language bottleneck
A linear adapter is less convincing when the target concerns texture, depth, force, or touch—properties that are hard to verbalize and rarely paired with rich captions. Two explanations remain open: insufficient paired data, or an architectural bottleneck that language alignment alone cannot remove.
Moravec’s paradox sharpens the issue. Models excel at bar exams, standardized tests, and formal problems because people have curated textbooks and exercises for them. Robots still struggle with grabbing keys or pressing a clicker because effortless sensorimotor skills rely on embodiment, enormous interaction histories, and evolution rather than documented examples.
Possible routes forward include better tactile and physical sensors, richer embodied data, and self-evolving AI that repeatedly proposes its own tasks and learning objectives instead of only imitating a fixed human-curated dataset.
Multimodal output can be completed either by retrieving an existing item or generating a new one.
Two ways to close the multimodal output loop
- Retrieval searches a fixed collection and returns an existing image or other artifact. The result is real and bounded by known corpus content, but novelty is limited by coverage.
- Generation synthesizes modality elements such as pixels directly. It can produce combinations absent from any database, but may hallucinate or render poor-quality details.
Attaching either mechanism to multimodal perception and language reasoning creates an agent that can consume and emit multiple modalities in a synchronized interaction.
CLIP-style contrastive embeddings support context-aware text-to-image retrieval as a multimodal decoder.
Retrieval through a shared embedding space
CLIP learns image and caption encoders with a contrastive objective: matched pairs move closer and mismatched pairs move apart. At inference time, nearest-neighbor search retrieves the corpus image whose embedding best matches the current context.
A multimodal assistant can feed an LLM interleaved image-caption history, predict the next textual continuation, and use the resulting semantic embedding as the retrieval query. This supports dialogue that narrows an animal from squirrel to beaver, finds houses similar to a photographed design, or shows what a recipe phrase such as stiff peaks looks like.
Redundant image-caption context can be useful, but corpus coverage, safety, copyright, and attribution constrain retrieval.
Why keep both image and caption?
The two representations can overlap semantically without being interchangeable for a task. A caption may say that egg whites have stiff peaks, yet a learner still needs the corresponding appearance. Retaining both representations lets the model reason linguistically and retrieve a usable visual reference.
Retrieval remains bounded by database support and inherits the corpus’s risks. Web-scale image-caption datasets can contain sensitive, illegal, or copyrighted material, creating curation and attribution problems. One emerging compromise is to generate a new image while also returning nearest training examples as approximate provenance, though proximity is not a complete copyright solution.
Generative image decoders enable highly specific, iterative visual interaction that retrieval cannot guarantee.
Generation supports bespoke visual iteration
A diffusion-style decoder can turn text or image context into a newly synthesized image rather than choosing among candidates. Users can iteratively refine details—cookie spacing and arrangement, a sunflower tattoo’s location, or a birthday cake’s name, age, and design—that are unlikely to co-occur exactly in a retrieval corpus.
The full frontier-model pattern is therefore:
1. encode sequential multimodal inputs;
2. adapt their features into a language model for reasoning and dialogue; and
3. attach a retrieval or generative decoder on the output side.
A text-to-image system can be decomposed into image encoding, text-to-image-space mapping, and image decoding.
Three-stage text-to-image schematic
1. Encode images: use CLIP or another visual encoder to obtain features. CLIP is especially useful because contrastive pretraining already places visual and textual semantics nearby.
2. Map text into image space: encode the caption, then predict the image representation associated with that caption, often autoregressively.
3. Decode pixels: turn the predicted visual representation into a complete image.
This factorization isolates representation learning, cross-modal translation, and rendering, allowing each component to improve independently.
Discretizing image embeddings turns difficult high-dimensional regression into autoregressive visual-token prediction.
Images become a discrete vocabulary
Continuous visual embeddings are clustered into a codebook—often around 8192 entries—and each embedding is replaced by its cluster ID. A visual token is therefore not one pixel or one human-named concept; it identifies a learned cluster center shared by many nearby continuous features.
With paired image-caption data, text conditions an autoregressive sequence model that predicts visual token v₁, then v₂ | v₁, and so on. A decoder reconstructs pixels from the completed token sequence. Quantization makes prediction tractable but can blur detail because clustering and reconstruction are lossy.
Modular pretraining improves text-to-image systems, and quantization avoids regression-to-the-mean behavior.
Reuse strong components and classify codes
Different data sources can pretrain different modules:
- image-text pairs train the aligned image encoder;
- text-only corpora train the language encoder; and
- image-only data train the generative decoder.
Only the bridge from text features to visual representations then requires paired fine-tuning data. Historically, replacing jointly trained VAE pieces with pretrained CLIP encoders, improved LLMs, and diffusion decoders produced major system gains.
Quantization also changes the optimization problem. Regressing a roughly 700-dimensional continuous target with MSE or MAE tends toward averages; predicting one of thousands of codebook IDs uses cross-entropy and preserves multimodal alternatives more naturally.
Generative modeling learns a tractable probability distribution over data, often conditionally, for likelihood evaluation, sampling, and representation learning.
From discriminative labels to a data distribution
A classifier estimates P(Y | X). A generative model instead chooses a parameterization for P(X) and seeks three useful capabilities:
1. fit: maximize likelihood on real observations;
2. evaluate: assign high density to realistic data and low density to noise; and
3. sample: efficiently draw new observations from high-density regions.
Modeling P(X) also learns variation, clusters, and hierarchy, so generation doubles as unsupervised representation learning. Conditional generation estimates P(X | C) for a class, caption, or another modality. Editing or style transfer can be viewed as generating X₂ conditioned on source X₁ and instruction C.
Latent-variable models explain high-dimensional observations through smaller hidden factors, with Gaussian mixtures as a simple case.
Explain observations through hidden factors
Latent variables Z encode unobserved causes of high-dimensional data X, such as pose, hair length, color, age, or glasses. The generative direction is Z → X, or probabilistically P(X | Z).
A Gaussian mixture is the simplest example:
- categorical Z ∈ {1,…,K} selects a component;
- each component has its own mean and variance for P(X | Z); and
- sampling first chooses Z, then samples X from that Gaussian.
Choose latent dimensionality and component count with domain knowledge, visualization such as PCA, and held-out likelihood. Distribution choice must match the data: Gaussian assumptions are natural for continuous features but not directly for discrete text tokens.
Expectation-maximization fits a Gaussian mixture by alternating latent assignments and parameter updates.
Alternating inference and fitting
Expectation-maximization repeats two coupled updates:
1. E-step: with current Gaussian means and variances, compute each observation’s component-membership probabilities.
2. M-step: with those responsibilities fixed, update every component’s mean and variance from its weighted observations.
The next E-step uses the improved parameters, and the cycle continues to convergence. Because Gaussian density is nonzero away from its center, ambiguous boundary points can have probability under several components rather than a perfectly hard membership.
Variational autoencoders extend mixture models with neural encoder-decoder distributions and per-example latent parameters.
Neuralize both directions of a latent model
A VAE contains two distributions:
- the inference encoder qφ(Z | X), which maps an observation to latent Gaussian parameters; and
- the generative decoder pθ(X | Z), which maps a latent sample back to a distribution over observations.
Unlike a Gaussian mixture’s few dataset-wide component parameters, the encoder predicts a different mean and variance for every input. Neural networks make those mappings nonlinear and expressive, creating many local latent Gaussians. The reconstruction term trains decoded X̂ to match X, using a suitable loss such as cross-entropy or mean-squared error, while a second term keeps latent distributions compatible with the chosen prior.
The reparameterization trick makes stochastic latent sampling compatible with backpropagation through a VAE.
Move randomness outside the differentiable path
The encoder outputs per-example vectors μ(X) and σ(X). Directly sampling from N(μ, σ²) would put a stochastic operation in the gradient path. Reparameterization rewrites the sample as:ε ~ N(0, I)Z = μ + σ ⊙ ε
Randomness now lives in ε, while Z remains differentiable with respect to μ and σ. Reconstruction gradients can therefore update both decoder and encoder. Latent regularization keeps the encoded distributions near the assumed Gaussian prior, preserving a sampleable space. This is the key bridge from simple Gaussian mixtures to trainable neural latent-variable models and sets up the later progression to diffusion and flow matching.
Enjoy Reading This Article?
Here are some more articles you might like to read next: