Lecture 2 – Multimodal Research Tasks (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
The lecture connects multimodal research tasks and datasets directly to the course project workflow.
Purpose. This lecture is a map from research practice to a feasible multimodal course project.
What it supplies: a research workflow, a taxonomy of current tasks and datasets, and guidance for matching those opportunities to students’ domain backgrounds.
Course support: slides, recordings, Canvas materials, and a shared preference form help students find collaborators with compatible interests.
A strong AI project follows the scientific process from observation and literature review through falsifiable tests and reported conclusions.
Scientific workflow: observation → literature review → research question and hypothesis → implementation and experiments → analysis and iteration → conclusion.
The assignments mirror this sequence:
- Proposal: establish the idea, prior work, and limitations.
- Midterm: formulate questions and run initial baselines.
- Final: introduce new methods or experiments and defend evidence-based conclusions.
A negative result can still be useful when it cleanly falsifies a hypothesis.
Bottom-up discovery and top-down vision offer complementary ways to generate research ideas.
Bottom-up discovery asks: where does today’s best method fail, and how can that failure be fixed? It offers a clear baseline and measurable progress, although the contribution may be incremental.
Top-down design asks: what larger capability should exist, and which tractable steps move toward it? It encourages ambitious ideas but carries more execution and relevance risk.
The most productive project often combines them: use a broad vision to choose a direction, then use concrete baseline failures to define the next experiment.
Research questions become experimentally useful when their hypotheses are specific and falsifiable.
A useful research question commits to a testable contrast.
- Prefer Does X improve Y under condition Z? over How does X compare?
- State the expected answer before running the experiment.
- Define the comparator, dataset regime, metric, and possible disconfirming result.
For cross-modal alignment, asking whether stronger alignment reliably predicts downstream performance is directly testable. For DRPO, the sharper claim is whether it outperforms a named alternative on heterogeneous multimodal distributions.
Modern multimodal research can be organized around new modalities, core integration, reasoning, agents, social intelligence, embodiment, and human impact.
Seven project families: new sensing domains; fusion and alignment; reasoning; interactive agents; social intelligence; embodied systems; and ethics/safety.
Despite very different inputs, the common abstraction is stable: each modality contains elements arranged in space, time, or another structure; a model must represent their differences and connections, then use the joint representation for prediction, generation, action, or transfer.
New sensing modalities create valuable applications but impose severe data scarcity and pairing challenges.
Moving beyond image–text data expands what AI can sense, but the research problem changes.
- Modalities differ in privacy, invasiveness, sampling rate, noise, and cost.
- Data are scarce, and paired observations across modalities are scarcer still.
- Domain knowledge becomes central to meaningful preprocessing and evaluation.
Clinical resources such as CLIMB aggregate 1D signals, 2D imaging, 3D or temporal scans, graphs, and paired reports so researchers can study these trade-offs across diseases and regions.
MIMIC and electronic-smell datasets illustrate how heterogeneous signals can support health and chemical-sensing research.
Healthcare example: MIMIC combines free text, tabular hospital records, physiological time series, and imaging with clinically meaningful outcomes.
Olfaction example: arrays of gas sensors plus environmental measurements can identify foods, mixtures, or trace allergens.
A practical definition of modality is relational rather than categorical: examine how heterogeneous the channels are, how much information they share, and how they interact. Homogeneous time-series channels may be treated as one multichannel modality or several closely related modalities depending on the research question.
OpenTouch pairs egocentric vision, tactile pressure, and 3D hand pose to study physical interaction.
OpenTouch records the same manipulation from three complementary views:
- Egocentric vision shows the object and scene.
- Tactile sensing measures where and how strongly pressure is applied.
- Hand pose reconstructs finger and hand configuration from IMUs.
This pairing enables projects in robot grasping, visuotactile world models, and haptic communication—not merely object classification.
Manufacturing is a rich multimodal domain where models must transfer beyond familiar web audio and video.
Manufacturing turns multimodal perception into operational decisions: infer process state, detect anomalies, predict maintenance, and recommend or execute repairs.
The research opportunity is not simply to run an existing audiovisual model. It is to isolate why transfer fails—limited labels, unusual high-frequency audio, sensor drift, temporal dependencies, or a different relationship among channels—and then test a targeted remedy.
Foundational multimodal research must bridge discrete tokens with continuous signals and test generality across domains.
Two frontier questions connect today’s token-centric models to real sensing:
1. How can continuous, high-frequency streams be discretized without discarding semantic structure?
2. Can alignment and fusion operate natively in continuous space?
Omnimodal models aim to reuse one architecture and parameter set across many modality combinations instead of orchestrating separate encoders. Claims of such generality require multi-domain evidence; suites such as MultiBench and MultiBench+ enable controlled comparisons of fusion methods across affect, health, robotics, finance, and multimedia.
Multimodal reasoning extends perception by composing fine-grained evidence across several steps and modalities.
Perception: identify what is present.
Reasoning: decompose a question, combine evidence across steps or modalities, and derive an answer.
Captioning sits between the two: it demands a coherent summary but admits many correct phrasings, making automatic evaluation difficult. VQA narrows the target to a specific question, improving evaluability. The deeper research issue remains whether explicit symbolic structure is necessary or whether reliable compositionality can emerge from scale.
Video QA, procedural visual reasoning, and Winoground expose compositional failures hidden by ordinary benchmarks.
Harder benchmarks deliberately separate recognition from composition.
- TVQA/MovieQA: reason over events, characters, and relationships through time.
- NLVR/NLVR2: test counting, attributes, and spatial relations under controllable complexity.
- Winoground: swap relational word order while preserving the same objects, exposing reliance on familiar co-occurrence.
A benchmark-specific rule can raise a score without transferring. Evidence of reasoning should therefore include held-out structures, tasks, or domains—not only success on one procedural template.
PuzzleWorld and ScienceQA test open-ended exploration, external knowledge, symbol manipulation, and diagram-based reasoning.
Reasoning performance depends on both training stages and the inference procedure: pretraining supplies broad knowledge, instruction tuning shapes behavior, and test-time reasoning supports decomposition and revision.
PuzzleWorld stresses underspecified, exploratory problems and provides expert reasoning traces for intermediate supervision.
ScienceQA tests whether a model can combine text with scientific figures, force diagrams, geometry, and mathematical reasoning.
Together they move evaluation beyond short, pattern-matched answers.
Interactive agents ground language and vision in long action sequences across websites and operating systems.
A multimodal agent repeatedly interprets an instruction, observes an interface, chooses an action, and updates its plan.
Why vision matters: raw HTML is verbose and misses layout, color, embedded images, and other spatial cues. Screenshots and accessibility trees provide complementary representations.
Benchmark progression: WebArena → VisualWebArena → cross-platform environments such as OS-Atlas.
Success must account for both long-horizon reliability and adversarial safety; one mistaken action can compromise accounts or data.
Socially intelligent AI requires multimodal perception, interaction, common sense, and theory of mind.
Social intelligence layers several capabilities:
- perceive emotion, intent, and speech from language, face, voice, and physiology;
- track relationships and emotional dynamics across multiple people;
- apply social common sense and nested theory of mind;
- respond at the right moment and with appropriate affect.
Mime-based tasks stress the long tail: infer an imaginary heavy object, connect it causally to a later fall, and interpret mocking behavior without explicit speech or visible props.
Embodied multimodal AI couples noisy real-world perception to safe physical action.
Embodied agents close the loop between perceive → predict consequences → act → observe.
Representative data sources include:
- language-guided indoor navigation in Room-to-Room and Room-Across-Room;
- egocentric demonstrations from EPIC-KITCHENS and Ego4D;
- robot-arm trajectories with vision, proprioception, force, and actions for OpenVLA-style models.
Efficiency and robustness are essential because deployment hardware is constrained and physical errors can harm people or property.
Human-centered evaluation and multimodal safety must accompany improvements in model capability.
Capability metrics alone cannot establish that a multimodal system improves people’s lives.
Human-centered evaluation asks which tasks benefit from assistance, where people should retain control, and how collaboration changes speed, quality, trust, or workload.
Safety evaluation must test bias and sensitive-attribute leakage in every modality and in their interaction, because combining signals can amplify rather than cancel unfairness.
The immediate project milestone is to select, inspect, visualize, and label a meaningful multimodal dataset.
Next step: choose a dataset whose modalities and task match your expertise and curiosity.
Before proposing a sophisticated model, make the data research-ready: inspect distributions, process each modality, visualize cross-modal relationships, define labels, and document limitations.
The shared dataset catalogue and project-preference form support discovery and team formation; the first homework and TA tutorial turn that choice into a reproducible dataset pipeline.
Enjoy Reading This Article?
Here are some more articles you might like to read next: