Lecture 8 – Modern Generative AI (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
The lecture completes the pre-midterm generative-AI unit while tying compute access and assessment logistics to course work.
This is the final new-content lecture before the midterm. It completes the path from latent-variable models to modern diffusion and flow-based generation, then asks how the same machinery transfers across modalities.
Operationally, students should use the published forms for compute credits and approved accommodations; the upcoming review introduces no additional examinable content.
Modern multimodal systems connect modality adapters, language-model interaction, and generative output into one input-to-output loop.
The course’s foundation-model stack has three layers:
1. Multimodal encoding and alignment contextualize elements across input streams.
2. Adapters plus a language model support flexible questioning, answers, and dialogue.
3. Retrieval or generation heads turn output context into images or other modalities.
Together these layers support multimodal input and output rather than a text-only response to rich input.
Image retrieval offers grounded reliability, whereas image generation offers open-ended coverage with hallucination risk.
Retrieval path: embed the generated description and select a nearby image from an aligned database. It is grounded in known assets but cannot go beyond that inventory.
Generation path: condition a learned image distribution on the description and sample a new result. It is flexible in content and resolution but can hallucinate.
A robust multimodal assistant can expose both candidates and let task constraints—or the user—decide which trade-off is appropriate.
Text-to-image systems become tractable by aligning text and visual embeddings, discretizing visual targets, and decoding them with a high-quality generator.
A common text-to-image pipeline is:
1. Encode images into semantic latent features aligned with text.
2. Quantize continuous features into a finite vocabulary of visual tokens.
3. Predict visual-token IDs from caption tokens, often autoregressively.
4. Decode the predicted latent structure into high-resolution pixels.
Quantization replaces unstable continuous-vector regression with a classification problem. The same idea turns masked speech prediction into cluster-ID prediction rather than raw-waveform reconstruction.
Generative modeling learns a computable but expressive distribution over data for scoring, sampling, and conditional generation.
A generative model balances two requirements: expressiveness for complex high-dimensional data and tractability for training and sampling.
Latent-variable factorization introduces \(Z\), hidden factors of variation that generate \(X\). Training raises probability on real data; deployment may score new inputs, sample new data, or sample conditionally.
Foundation-model pretraining illustrates a broader payoff: modeling the raw-data distribution can implicitly learn structure and features that transfer to many prediction tasks.
A Gaussian mixture model illustrates latent-variable generation and expectation-maximization in a simple computable setting.
For a mixture of three Gaussians:
- \(Z\in\{1,2,3\}\) selects a component.
- \(p(X\mid Z)\) is a Gaussian with component-specific mean and variance.
- The E-step infers component responsibilities for each observation.
- The M-step updates component parameters from those responsibilities.
Sampling follows the hierarchy \(Z\sim p(Z)\), then \(X\sim p(X\mid Z)\). Fixing a chosen \(Z\) provides controllable sampling.
Semantic segmentation and tokenization choices determine what continuous-data models can learn.
Before choosing an architecture, define the units it will predict.
Fixed-rate windows are convenient but can ignore semantic duration; per-time-step models preserve resolution at enormous sequence cost; learned boundaries try to match meaningful units.
Language succeeds partly because subword tokenization balances reuse and meaning. Audio and video need analogous choices, and a poor segmentation can cap performance regardless of the downstream generator.
Variational autoencoders replace a small categorical mixture with continuous latents and neural conditional distributions.
A VAE upgrades the mixture model on two fronts:
- Latent space: a high-dimensional continuous \(Z\) replaces a small categorical choice.
- Conditional generator: a neural network parameterizes \(p_\theta(X\mid Z)\), so the output distribution varies nonlinearly with each latent code.
The prior can remain simple—usually \(\mathcal N(0,I)\)—because expressive encoders and decoders can absorb shifts, scales, and nonlinear structure.
The VAE encoder outputs a distribution and uses reparameterization so sampled latents remain trainable by backpropagation.
The VAE inference path is:
1. \(X\xrightarrow{q_\phi}(\mu(X),\sigma(X))\).
2. Sample \(\epsilon\sim\mathcal N(0,I)\).
3. Reparameterize \(Z=\mu+\sigma\odot\epsilon\).
4. Decode \(Z\xrightarrow{p_\theta}\hat X\).
Reparameterization isolates randomness in \(\epsilon\), allowing gradients to update the encoder’s mean and variance predictions.
VAE training balances reconstruction fidelity with a KL prior that smooths the aggregate latent space.
The VAE objective has two competing terms:
- Reconstruction: retain enough information in \(Z\) to recover \(X\).
- Prior matching: minimize a KL divergence that organizes inferred codes around \(\mathcal N(0,I)\).
Reconstruction alone can leave holes between memorized examples. Prior regularization fills and smooths the latent space so nearby codes decode continuously and random prior samples remain plausible.
A VAE bottleneck encourages compact factors of variation, while latent dimension and disentanglement remain empirical design choices.
A good latent code compresses the data without collapsing its important variation.
- Choose \(\dim Z\) as a capacity hyperparameter, not from the number of labels.
- The bottleneck forces reuse of dimensions across many examples.
- The Gaussian prior promotes a continuous, sampleable geometry.
- Latent traversals test whether changing one coordinate smoothly changes factors such as hairstyle, handwriting, or pose.
Disentanglement is an aspiration of the learned representation, not information supplied with the raw examples.
Diffusion encodes data by a fixed multi-step noising process and generates it through a learned reverse denoising process.
Diffusion replaces one large encode/decode jump with a chain:
- Forward: \(q(X_t\mid X_{t-1})\) adds prescribed Gaussian noise until \(X_T\approx\mathcal N(0,I)\).
- Reverse: \(p_\theta(X_{t-1}\mid X_t)\) learns to remove a small amount of noise per step.
To generate, sample \(X_T\) from noise and repeatedly apply the learned reverse transitions until reaching \(X_0\).
Diffusion training resembles a multi-level VAE with reconstruction losses at many noise levels and a terminal prior.
A diffusion objective can be understood as repeated local autoencoding:
1. Corrupt \(X_{t-1}\) into \(X_t\) with the fixed schedule.
2. Train the reverse network to reconstruct the cleaner state.
3. Repeat over many randomly selected noise levels.
4. Match the terminal state to a standard Gaussian prior.
This is analogous to a stack of small VAEs, but without a dimensional bottleneck in the original formulation.
Equivalent diffusion targets differ in optimization difficulty even when they imply the same reverse process.
Common parameterizations ask the network to predict:
- \(X_0\), the clean sample;
- \(\epsilon\), the injected noise;
- \(X_{t-1}\), the next cleaner state;
- or a velocity-like combination.
Although these are algebraically related, optimization is not identical. Natural data occupy structured low-dimensional manifolds; pure noise does not, which can make a semantically meaningful clean target easier to learn.
Gradual denoising and a carefully shaped noise schedule turn generation into a sequence of easier local problems.
Diffusion’s practical advantage is decomposition: each reverse transition solves a smaller denoising task than a one-shot decoder.
The noise schedule allocates fine steps near clean structure and coarser perturbations where the sample is already noisy. In the limit of infinitesimal steps, the trajectory becomes a continuous-time differential equation, providing the bridge to score-based and flow formulations.
Flow matching and latent diffusion improve efficiency while retaining the core continuous transformation from noise to data.
Two modern refinements target diffusion’s cost:
- Flow matching learns a continuous-time vector field that transports noise toward data and can support faster, simpler trajectories.
- Latent diffusion uses a VAE encoder to compress pixels, denoises in the smaller latent space, then applies the VAE decoder.
Both preserve flexible conditioning: text, frozen semantic embeddings, segmentation maps, or boxes can steer the generated result.
Diffusion language models generate and revise token sequences in parallel rather than committing strictly left to right.
Text diffusion changes the generation geometry:
- Begin with a sequence of masked or corrupted positions.
- Iteratively predict and revise tokens anywhere in the sequence.
- Stop when the sequence is fully denoised.
This can correct early mistakes and parallelize decoding, unlike strict left-to-right generation. The central modeling choice is the corruption process: categorical masking or substitution fits discrete tokens, while continuous embedding noise remains an active alternative.
Multimodal diffusion can corrupt and reconstruct text and image streams jointly with modality-appropriate noise.
A joint multimodal diffusion model uses one trajectory but different corruption operators:
- Text: categorical noise such as masks, drops, or substitutions.
- Images: continuous Gaussian noise in pixel or latent space.
The model learns to reverse both processes together, restoring words and visual content from noise. This unifies multimodal understanding, reasoning, and generation under a denoising objective rather than a purely autoregressive sequence objective.
Enjoy Reading This Article?
Here are some more articles you might like to read next: