Lecture 10 – Multimodal Interaction (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
The remaining assignments shift the course from baseline analysis toward reasoning, agents, and a final research contribution.
The project sequence is evidence driven:
- Midterm report: problem motivation, processed dataset, strong baselines, preliminary results, and failure analysis.
- Homework 4: reward design and reinforcement learning for multimodal reasoning.
- Homework 5: interactive agents.
- Final project: poster plus final report.
The error analysis is the bridge: it should explain why the proposed method is necessary.
Reasoning synthesizes structured information over multiple inferential steps and can be induced by prompting, supervised traces, or reinforcement learning.
Reasoning transforms an input into a decision through an explicit or latent sequence of intermediate states.
Three training regimes differ mainly in supervision:
1. Prompting: ask a pretrained model to decompose the problem.
2. SFT: imitate human- or corpus-provided reasoning traces.
3. RL: optimize a final reward and let the model infer useful intermediate actions.
The same pattern applies to equations, code planning, and decomposing a household instruction into robot actions.
Policy gradients increase the probability of actions on successful trajectories and decrease it on unsuccessful ones.
For a sampled trajectory \(\tau\), a basic policy-gradient update scales \(\sum_t\log\pi(a_t\mid s_t)\) by its return.
- Positive advantage → raise the likelihood of the chosen actions.
- Negative advantage → lower it.
Use \(A=R-b\), where \(b\) may be an exponential moving average, because absolute rewards can all be positive or all negative. The baseline makes the update reflect relative quality.
GRPO-style language-model training combines grouped samples, reward or verifier scores, advantages, and a reference-model constraint.
Language-model RL maps familiar components into sequence modeling:
- Policy: the current LLM.
- State: prompt plus dialogue or reasoning history.
- Actions: output tokens.
- Group: several sampled completions.
- Reward: correctness checks, a learned preference model, or a composite evaluator.
- Advantage: reward relative to the group or running baseline.
- KL regularizer: keep the updated model near a frozen reference.
Outcome-based RL can elicit multimodal medical explanations, but reward shaping must specify the desired correctness, grounding, and form.
A binary final answer does not uniquely determine a useful explanation. Composite rewards can encourage several properties at once:
- Clinical accuracy receives the strongest weight.
- Visual grounding rewards correspondence between textual claims and image regions.
- Length or format discourages trivial or excessively long rationales.
The resulting behavior may reflect newly reinforced strategies or capabilities already latent in the base model; current evidence does not cleanly separate those explanations.
Generated reasoning must be evaluated for faithfulness rather than trusted because it sounds plausible.
Reasoning text is an output to audit, not proof of the model’s causal process or correctness.
Use domain experts to label factual and clinically useful claims, measure faithfulness separately from final-answer accuracy, and surface uncertainty. Reported performance around 80–90% still means consequential failures remain; a fluent rationale can be wrong even when the prediction is right, and vice versa.
An AI agent turns multi-step reasoning into a closed action–observation feedback loop.
An agent repeatedly executes observe → reason or plan → act → receive feedback → revise.
Grounding forms a progression:
- text and documents;
- digital interfaces and software;
- embodied robots with sensors and actuators;
- broadly world-grounded systems working across homes, schools, workplaces, or hospitals.
The core loop is shared, but the cost and risk of an incorrect action rise with richer grounding.
Agent structure ranges from a prompted base model through engineered workflows to self-optimizing and self-evolving systems.
Structural autonomy can be separated into four levels:
1. Base model: prompt or image in, action out.
2. Designed workflow: humans supply tools, APIs, and memory paths.
3. Workflow optimization: the agent chooses which available resources to orchestrate.
4. Self-evolution: the system creates subtasks, agents, and memory structures.
Each step increases adaptive capacity and also the chance that execution exceeds the user’s intended scope.
Human-in-the-loop agents often outperform all-or-nothing autonomy by making work small, transparent, verifiable, and learnable from feedback.
The practical autonomy question is not simply whether an agent can finish the whole task.
A strong human–agent interface:
- proposes a bounded next step;
- makes the change easy to inspect;
- requests confirmation at consequential points;
- learns from accepts, rejects, and corrections.
This pattern can be easier to train and safer to use than returning a large result only after a long opaque run.
WebArena established realistic, executable benchmarks for agents that complete tasks on cloned websites.
WebArena turns web use into a reproducible control problem:
- Observation: current page representation.
- Action space: clicks, text entry, scrolling, navigation, and tabs.
- Goal: an executable final site state.
- Environment: realistic but isolated clones of common services.
The large human–agent success gap shows that everyday tasks can be easy for people yet difficult for models when they require many grounded steps.
VisualWebArena adds screenshots and accessibility structure because raw HTML is verbose and loses crucial spatial information.
Website observations have complementary strengths:
- HTML: exact structure but huge and noisy.
- Accessibility tree: cleaner semantic controls and labels.
- Screenshot: compact spatial, color, typography, and embedded-image evidence.
VisualWebArena requires agents to combine these sources with task context. The benchmark measures both long-horizon difficulty and visual complexity such as the number of interactive objects.
Agent evaluation should verify the environment’s final state, and broader benchmarks expose unresolved questions about persistence and proactivity.
Do not score an agent solely on its self-reported success. Use state-based verifiers—exact matches, URLs, database state, or task-specific predicates—to test whether requirements were actually satisfied.
Cross-platform suites such as OS-Atlas broaden the interface distribution. Persistent monitoring adds a separate systems problem: define a bounded watch list, cadence, cost budget, and notification rule. A deterministic bot may be the better tool when the task is merely repeated search.
Set-of-Marks converts a webpage into numbered, actionable visual elements and modestly improves multimodal agents.
Set-of-Marks gives the model a compact action-oriented representation:
1. Detect interactive regions on the screenshot.
2. Assign each region a visible identifier and location.
3. Let the agent choose actions over those identifiers.
This improves grounding over raw captions or pixels alone, but benchmark success remained roughly in the teens while humans exceeded 80%. Better perception helps, yet it does not solve planning and recovery.
Long-horizon planning and multimodal perception are the two dominant failure modes for interface agents.
Agent reliability is constrained by two coupled systems:
- Reasoning and planning: decompose goals, track progress, replan, and escape dead ends.
- Multimodal perception: identify the correct entities and controls in cluttered, changing environments.
A perfect planner acting on a misread page still fails; a perfect encoder without a recoverable plan also fails. Designing interfaces for both people and agents is therefore an emerging HCI problem.
Hierarchical planning connects semantic subgoals to low-level executable actions in a recurrent agent loop.
A practical web-agent loop separates abstraction levels:
1. Convert the instruction into semantic subgoals.
2. Combine the next subgoal with the current observation.
3. Emit a concrete click, keystroke, scroll, or navigation command.
4. Observe the result and update the plan.
This prevents the model from attempting a complex task in one unstructured leap and makes progress easier to inspect.
A small human clarification can steer an otherwise capable plan, and uncertainty estimation can decide when to request it.
Human intervention is most valuable at a decision bottleneck, not after every action.
A single demonstrated next step can unlock the remainder of a plan. Estimate when help is needed by checking output stability across repeated samples or, when available, inspecting predictive probabilities.
High disagreement suggests the agent should clarify or defer. This black-box method is broadly applicable but trades additional compute and latency for safer control.
Inference-time search improves success by exploring multiple trajectories, but needs value and ranking functions to spend samples efficiently.
Planning commits to a path; search deliberately explores alternatives when the correct path is uncertain.
Best-of-\(N\) raises coverage by sampling several trajectories, but naïve search wastes compute. Two learned components make it useful:
- a value function estimates which partial paths deserve expansion;
- a ranking or verification function chooses among completed paths.
The design objective is not merely more samples, but more success per sampled trajectory.
Bounded, updateable memory lets long-running agents preserve salient state without unbounded context growth.
Long context is not the same as effective memory. A useful memory controller decides what to store, update, retrieve, and forget.
A recurrent fixed-length state compresses prior observations into a bounded context at each step, avoiding indefinite accumulation. External Markdown or database-like memory can extend this pattern to tool-rich and multi-agent workflows, but still needs policies for relevance, consistency, and deletion.
Vision-language-action models extend the same agent principles to continuous robot control through specialized action decoders.
A VLA system maps vision + language → action.
- A visual encoder represents the robot’s current scene.
- An adapter conditions a pretrained language model on that representation and the instruction.
- An action decoder produces continuous actuator commands.
Diffusion-based decoders are attractive because they naturally model continuous, multimodal action distributions. Much of the language backbone can remain frozen while robot-specific perception and control modules are trained on vision–instruction–action demonstrations.
Enjoy Reading This Article?
Here are some more articles you might like to read next: