Lecture 9 – Multimodal Reasoning (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
A project baseline is an existing general method applied competently to the chosen data, not a novel contribution merely because the dataset is new.
Research checkpoint
The project midterm should establish a credible empirical starting point:
- finalize the task, dataset, and candidate method;
- run suitable prior state-of-the-art baselines;
- report numerical results;
- perform error analysis;
- use the observed failures to motivate the next method.
A method remains a baseline even if nobody has previously run it on this particular dataset. Applying an existing LLM, fusion architecture, or agentic pipeline to a new dataset is useful experimental work, but application alone is not methodological novelty. A contribution begins when the team identifies a consequential limitation and proposes a defensible improvement. Dataset, architecture, topic, and even team details may change as evidence accumulates; the report should explain the current decisions and what the baseline results teach.
The second half of the course moves from foundations into reasoning, agents, applications, cross-modal transfer, and self-improving systems.
Second-half roadmap
The foundations block—fusion, alignment, representation learning, transformers, and generation—supplies the components for more capable systems.
The next stages are:
1. Reasoning: multistep inference and reinforcement learning, followed by explainable and prescriptive reasoning with uncertainty estimates.
2. Interactive agents: sequences of executable actions on websites, robots, manufacturing systems, and other environments.
3. Applications: manufacturing, design, cities, and transportation, all of which combine sensor, visual, graph, and tabular data.
4. Advanced transfer: use capabilities learned from a resource-rich modality such as language in robotics, smell sensing, or another modality with little direct training.
5. Self-evolving agents: systems that propose tasks, supervision, and reward functions and then improve recursively.
Reasoning is the bridge from passive multimodal prediction to trustworthy action.
Reasoning combines knowledge across multiple inferential steps while exploiting the structure of a problem.
Working definition
Reasoning combines knowledge through multiple inferential steps while exploiting the structure of the problem.
Three ingredients distinguish it from one-step perception:
- intermediate evidence: each step derives or retrieves information;
- composition: later steps depend on and combine earlier results;
- problem structure: the computation follows a sequence, tree, graph, or state-changing interaction.
A proof composes lemmas, a recipe executes ordered actions, and a robot plans over the geometry and dynamics of an environment. In the multimodal setting, representations and alignments provide local evidence from text, images, sensors, or actions. Reasoning organizes those pieces into higher-order inferences that solve tasks a direct input-to-label mapping cannot reliably handle.
Compositional generalization exposes reasoning failures that cannot be explained by missing object recognition alone.
Compositionality test
A model may know every individual concept and still fail to combine them in an unfamiliar relation. Models readily handled plants surrounding a light bulb, a common visual pattern, but struggled when the same entities were rearranged as plants inside a light bulb. Likewise, generating an astronaut riding a horse was easier than reversing the relation to a horse riding an astronaut.
These failures isolate relational composition: object recognition is not enough if the model defaults to the statistically familiar arrangement. Similar brittleness appeared in character counting and simple arithmetic expressed in a less familiar language.
Chain-of-thought prompting was an important turning point. A cue such as think step by step encourages the model to externalize intermediate computations. That extra sequential structure can convert an incorrect direct answer into a correct one without changing model parameters.
Reasoning prompts should reflect whether a task has chain, tree, graph, or search structure.
Match topology to task
- Chain of thought: appropriate when each result feeds a single next computation, as in short algebra or arithmetic.
- Self-consistency: samples several chains and aggregates them rather than trusting one path.
- Tree of thoughts: branches over candidate states, detects dead ends, backtracks, and continues search; Sudoku and crosswords are natural examples.
- Graph of thoughts: allows richer dependencies and reuse among intermediate states.
These techniques inject classic algorithmic structure at the output level. A model that merely generates plausible next tokens can be prompted to enumerate alternatives, evaluate partial solutions, and revisit earlier choices. The important choice is not the fashionable name of the prompting method but whether its topology matches the dependency structure of the task.
Socratic multimodal systems use language as a common reasoning medium across specialized perception and action models.
Socratic Models pattern
Language acts as an interchange format among pretrained specialists:
- vision-language and audio-language models describe perceptual input;
- an LLM composes descriptions into a plan or answer;
- a vision-language-action component maps the language plan to executable controls.
The demonstrations span three levels. Image dialogue requires perception plus open-ended language. Robot block manipulation requires decomposing a goal into ordered moves and executing each one. Long-video reasoning maintains a compact textual world-state log of objects, locations, and actions, enabling questions such as where an item was left or why an event occurred without storing and searching every raw frame.
The architecture illustrates modular multimodal reasoning: specialized encoders retain modality competence, while language provides a shared bottleneck for memory, planning, and composition.
Modern reasoning systems combine bottom-up learning with deliberately injected symbolic structure.
Data–structure spectrum
One extreme expects reasoning to emerge bottom-up from enough demonstrations. The other explicitly supplies symbols, theorem provers, search procedures, planners, or world models. Current frontier systems are hybrids: neural models learn broad representations from data, while chain-of-thought injects sequential dependence and tree-of-thought injects search and backtracking.
Three tensions shape the design:
1. Discrete versus differentiable: symbolic actions are naturally discrete, while gradient training needs differentiable computation or a suitable estimator.
2. Discrete versus continuous concepts: discrete units aid interpretation; continuous relaxations fit neural optimization.
3. Learned versus expert knowledge: the model can infer regularities from data, but some constraints or domain rules may need explicit injection.
The chosen balance changes interpretability, robustness, efficiency, and the number of examples required.
A reasoning system can be analyzed along the dimensions of structure, concepts, inference rules, and knowledge.
Four analysis dimensions
- Structure: single-step perception, temporal chains, hierarchies, trees, graphs, or interactive loops where an action changes the next state.
- Concepts: the units manipulated at each step—words, latent vectors, attention maps, image regions, or bounding boxes.
- Inference: the relations that combine concepts, including implication, conjunction, and disjunction.
- Knowledge: facts, constraints, and procedures supplied by data or human experts.
These dimensions are partly independent. A system can use language concepts in a tree search, visual regions in a sequential chain, or latent vectors in an interactive controller. The lecture narrows its focus to language-mediated reasoning because pretrained LLMs provide a powerful base, then asks how training can make multimodal reasoning accurate and robust rather than merely verbose.
Prompting, supervised fine-tuning, and reinforcement learning require progressively different forms of reasoning supervision.
Supervision ladder
1. Direct prompting: add an instruction or a few worked examples. Parameters stay fixed, and reasoning comes from capabilities already present in the model.
2. Supervised fine-tuning: collect complete triples of input, intermediate reasoning trace, and answer; train autoregressively on every reasoning token and the final output.
3. Reinforcement learning: when reasoning traces are absent, sample candidate traces and optimize a reward or verifier that can judge their outcomes.
Dataset size helps choose among the first two: zero examples suggests direct prompting, a handful supports in-context learning, and a larger trace set supports parameter-efficient or full fine-tuning. RL is attractive when verification is easier than construction. A theorem checker can validate a proof, and a test suite can score code, even when writing the correct proof or program is expensive.
Reinforcement learning optimizes a policy for cumulative return through state-action-reward interaction.
RL formalism
An environment supplies states s, actions a, a transition rule, reward r, an initial state, a horizon, and often a discount γ. A policy πθ(a|s) is a distribution over actions conditioned on the current state. The objective is expected cumulative return, not isolated next-step accuracy:G_t = Σ_{k≥0} γ^k r_{t+k+1}
The discount expresses how much future reward matters; the horizon prevents interaction from continuing indefinitely. An action with a short-term cost can still be optimal if it raises long-term return.
For an LLM, the accumulated prompt and generated prefix form the state, the next token is the action, and the response distribution is the policy. A useful reward should judge the completed response or interaction, creating a delayed and often sparse learning signal.
Imitation learning is useful initialization but fails under distribution shift because errors move the policy away from demonstrated states.
Why demonstrations are insufficient
Imitation learning converts expert trajectories into supervised pairs (state, expert action). A driving model can learn steering and acceleration from dash-camera data and human controls.
The weakness is covariate shift. Training states come from the expert’s safe trajectory, but deployment states are generated by the learned policy. A small steering error can move the car toward a sidewalk—precisely a state absent from clean demonstrations. With no recovery examples, the next prediction may be worse, and errors compound.
Imitation is therefore valuable for initialization but cannot by itself guarantee recovery in safety-critical settings. The analogy for LLMs is instruction fine-tuning: humans demonstrate desirable answers, and the model imitates them. Reinforcement learning adds a later stage that samples from the model’s own policy and can reward useful behavior beyond the exact demonstrations.
REINFORCE increases the likelihood of sampled action sequences with high return and decreases it for sequences with low return.
REINFORCE intuition
In Pong, the state is the visual board, actions are paddle up or down, and the sparse terminal reward is +1 for scoring or −1 for conceding. A neural policy outputs the probability of each action.
Without expert labels, the policy samples complete episodes. REINFORCE applies an update proportional to:G_t ∇θ log πθ(a_t|s_t)
A positive return raises the probability of every sampled action in the winning trajectory; a negative return lowers probabilities along a losing trajectory. This is a strong credit-assignment assumption because a long episode may contain both good and bad moves, yet all receive the same terminal signal. Nevertheless, over many sampled episodes the stochastic estimator points toward higher expected return, providing a simple bridge from trial-and-error outcomes to differentiable policy updates.
Effective policy optimization balances exploration with exploitation and learns from reward relative to a baseline.
Two stabilizing ideas
Exploration versus exploitation: an untrained policy must try varied actions to discover rewarding behavior. As it improves, training should increasingly exploit what the policy already believes is good. An epsilon-greedy scheme makes this explicit by choosing a random action with probability ε and otherwise following the policy, typically reducing ε over time.
Reward versus advantage: absolute rewards may all be positive or all negative and therefore poorly calibrated. Compare the sampled return with a rolling or expected baseline:A_t = G_t − b(s_t)
A positive advantage means the outcome was better than expected and its actions should become more likely; a negative advantage means worse than expected. Centering around a baseline reduces variance and makes the update depend on relative quality rather than an arbitrary reward origin.
LLM reinforcement-learning post-training replaces scarce human labels with a learned reward model that can score many sampled responses.
Three-stage LLM training
1. Pre-training: learn broad language regularities from large unlabelled corpora.
2. Instruction fine-tuning: imitate curated human prompts and expert completions.
3. RL post-training: sample several responses from the current policy, score them, and optimize toward higher reward.
Humans can label only a small fraction of all possible responses. A reward model turns those annotations into a reusable scorer by learning a mapping from prompt-response text to predicted human preference. Pairwise ranking is often easier and more reliable than inventing absolute scores; the comparisons can be converted into a reward-learning objective.
Unlike pure imitation, RL can explore alternative completions that no annotator wrote. If a novel route earns high predicted or verifiable reward, policy optimization can reinforce it. The gain depends critically on whether the reward generalizes faithfully.
PPO- and GRPO-style methods stabilize policy updates and replace raw rewards with relative advantages.
From policy gradient to PPO and GRPO
For each prompt q, the policy samples a group of responses. A reward model or verifier scores them, and raw rewards become advantages by comparison with a baseline. Those advantages scale the response log-probability update.
A frozen reference policy adds a KL-based proximity constraint so post-training does not erase a strong pretrained model: the new policy may improve, but should not move arbitrarily far in one step. This is the stabilizing intuition behind proximal policy optimization.
Classical actor-critic methods estimate the expected future return with a learned value function and use it as the baseline. The lecture characterizes GRPO as using the relative rewards of multiple samples generated for the same prompt to construct its baseline. The apparent novelty is therefore built from long-established ingredients: sampling, policy gradients, variance-reducing baselines, and constrained updates.
A clinical multimodal reasoning model combines modality-specific encoders with an LLM and learns from inputs plus diagnoses without annotated reasoning traces.
Clinical reasoning case study
The target system must synthesize clinical notes and conversations, radiology or pathology images, ECG and other time series, and patient history. Its output should go beyond a disease label: explain supporting evidence, rule out alternatives, suggest diagnostic or treatment steps, and visually localize relevant anatomy.
The backbone follows a standard multimodal LLM pattern. Text enters through the tokenizer; images and sensor streams pass through modality-specific encoders; projection adapters map their features into the LLM representation space. Time-series encoders required additional fine-tuning before integration.
The key data constraint is supervision: the curated collection contains rich multimodal inputs and known final diagnoses, but not expert-written intermediate reasoning traces. That makes the case well suited to reward-based learning, provided the team can define signals that distinguish sound, grounded reasoning from plausible narration.
Reward design teaches the clinical model both answer correctness and multimodal grounding.
Reward decomposition
The dataset supplies no ideal rationale, so training combines proxy signals:
- Answer accuracy: reward 1 when the final diagnosis matches the label and 0 otherwise.
- Semantic visual alignment: compare the model’s predicted bounding box with the annotated region using intersection over union, encouraging textual claims to point to relevant image evidence.
- Reasoning length: reward an explanation up to a useful threshold—described as at least roughly 100 tokens—without continuing to reward unbounded verbosity.
The implementation weights these components rather than treating them as interchangeable. Together they encode a richer goal than answer accuracy alone: the model should be correct, visually grounded, and sufficiently explanatory. The case also exposes reward design as the practical bottleneck of RL reasoning; the policy will optimize what is scored, not an unstated clinical ideal.
The trained clinical model produces grounded explanations and cross-modal predictions, illustrating the lecture’s full reasoning pipeline.
What the trained model demonstrates
One example localizes a suspected tumor and explains that a round, well-circumscribed lesion differs from surrounding tissue. Another combines X-ray evidence, mechanical-ventilation duration, ECG, laboratory values, and history to reason about a prolonged ICU stay. The outputs therefore mix text reasoning, visual localization, and time-series interpretation rather than treating modalities independently.
The lecture closes with a practical decision rule:
- use prompting when pretrained capability is sufficient;
- use supervised fine-tuning when trustworthy intermediate traces exist;
- use reinforcement learning when inputs and outcomes are known but reasoning traces are missing and useful rewards can be computed.
Policy-optimization algorithms are reusable machinery. The problem-specific intellectual work is choosing evidence, structure, and rewards that make correct, grounded, concise reasoning more valuable than shortcuts or fluent but unsupported explanations.
Enjoy Reading This Article?
Here are some more articles you might like to read next: