Lecture 1 – Course Introduction (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
Multimodal AI is motivated by real-world intelligence that must combine diverse sensory and symbolic data.
Core thesis
Intelligence in the real world is inherently multimodal: no single channel contains everything a system needs to understand or act.
- People: words must be interpreted alongside voice, facial expression, and body language.
- Health: X-rays are only one source among histology, physiological sensors, behavior, and social context.
- Physical systems: robots and manufacturing systems may need vision, sound, touch, material sensing, and even chemical or smell sensing.
- Generation: the long-term goal is flexible conversion among text, images, video, audio, and other modalities.
- Agents: actions on websites, software, or robots must be grounded in what the agent perceives.
The recurring design problem is therefore to ingest heterogeneous evidence, connect it, and use the combined state to predict, generate, or act.
The course teaches modality-general design principles and multimodal learning rather than specializing in one data type.
Two learning goals
1. Learn a reusable method for applying AI to a new modality: identify its structure, choose suitable model biases, collect and label data, evaluate reliability, and plan deployment.
2. Learn how to connect multiple modalities: decide what paired data is needed and how representations should be fused, coordinated, or otherwise related.
The interdisciplinary teaching team and student examples make the intended scope concrete. Relevant modalities include text, images, moving images, audio, touch, molecular structures, biosignals such as EEG and EDA, event streams, tabular data, and symbolic representations. The course nickname, How to AI Almost Anything, reflects the aim of learning transferable building blocks rather than overfitting the curriculum to one familiar modality. Recent-paper reading and a substantial project develop both critical analysis and creative research practice.
The semester progresses from multimodal foundations to foundation models and then to domain applications.
Curriculum map
- Module 1 — Foundations: multimodal tasks and datasets; principles and inductive biases for individual modalities; fusion; representation learning and evaluation.
- Module 2 — Multimodal foundation models: transformers; pre-training and fine-tuning; transfer to downstream tasks; multimodal generation; multistep reasoning; interactive agents.
- Module 3 — Applications: design, manufacturing, cities, transportation, emerging topics, and student research presentations.
The twice-weekly lectures are paired with five roughly two-week homework and reading cycles, a midterm, and a major group project. The sequence deliberately moves from data and representation choices to increasingly capable models, and finally to real-world deployment contexts.
The five homeworks form a cumulative path from data preparation to multimodal agents.
Assessment and capability ladder
The grade is split across homeworks 35%, readings 15%, midterm 15%, and research project 35%.
1. Data: choose a domain-relevant dataset, then process, visualize, and label it. Multimodal can mean pairing an unusual modality with language; the source dataset need not already contain several modalities.
2. Supervised models: design modality-appropriate architectures and simple fusion baselines from scratch.
3. Multimodal language model: adapt an LLM so it can answer questions grounded in the new modality.
4. Reasoning: move from one-step perception to multistep question answering, including reinforcement-learning methods.
5. Agency: let the system take actions in an environment rather than merely describe its state.
Because the stages share one data setting, failures observed in the baseline ladder naturally become candidate research questions.
Reading assignments and the project train students to move from reproducing baselines to making a research contribution.
From reading to research
Each reading cycle asks students to summarize two required recent papers, discover related work beyond the assigned list, and synthesize an answer to open-ended prompts. The point is to reason about trade-offs, not recover one sanctioned answer.
The project then turns that habit into a research loop:
1. map the application domain and its literature;
2. implement strong classification, language-model, reasoning, and agentic baselines;
3. locate failures through error analysis;
4. formulate a method that addresses a consequential limitation;
5. iterate with experiments and ablations;
6. report both successes and failures.
Teams of two or three share responsibility and receive a dedicated mentor. Milestones are staged across preference matching, a short proposal, a baseline-heavy midterm report, a final presentation, and a revised final report.
Course policies and coding tutorials provide structure and practical support for the assignments.
Practical scaffolding
Lectures are recorded for public release, though students are expected to attend live when possible. The public site carries the syllabus, slides, and videos; Canvas handles enrollment and submissions. Individual late days apply to homeworks and readings, while team wild cards may extend written project milestones but cannot move the common presentation date.
Five implementation tutorials mirror the homework ladder:
- dataset processing, visualization, Python, PyTorch, and tensors;
- fusion architectures in code;
- multimodal LLM pre-training and fine-tuning;
- multimodal reasoning;
- multimodal agents.
The tutorials translate lecture ideas into guided notebooks and code skeletons. They are reinforcement and implementation support, not a separate stream of examinable theory.
The behavioral origins of multimodal AI showed that perception depends on interactions among signals.
Behavioral era
Early multimodal research in the 1970s and 1980s asked how humans combine language, gesture, facial movement, and sound during communication. David McNeill’s work helped establish gesture as integral to meaning rather than a decorative afterthought.
The audiovisual speech demonstration makes the key point experimentally: the audio track stays fixed, but changing the speaker’s lip movement changes the syllable many viewers believe they hear. Visual evidence therefore alters acoustic interpretation.
This result is more than a curiosity. It rules out a pipeline that treats speech as an isolated audio-recognition problem and motivates models in which evidence from one modality conditions perception in another. Cross-modal interaction was central before modern deep networks existed.
Multimodal AI evolved through computational, interaction, and deep-learning eras with increasingly capable tasks.
Capability eras
- Computational era: systems began interpreting speech plus pointing, tracking bodies and gestures, and searching digitized books, images, and videos.
- Interaction era: research shifted from one-way perception to multi-party and human-machine interaction, including meeting analysis and affect recognition.
- Deep-learning era: large datasets, GPUs, and neural networks produced stronger joint models. A canonical image-captioning architecture paired a CNN image encoder with an LSTM text decoder.
- Foundation-model era: still larger data, compute, and model scale broadened the number of tasks one model could perform.
The associated task history progressed from audiovisual speech recognition and multimedia retrieval to affect and event recognition, then to image captioning. Captioning mattered because the model generated an open-ended linguistic description rather than selecting only a fixed class, helping launch modern vision-language research.
Recent multimodal research expanded from question answering and navigation to generation, world models, and agents.
Modern task trajectory
Open-ended captions are difficult to score, so visual and video question answering introduced narrower prompts with checkable answers. The research frontier then expanded through multimodal dialogue, semantic video retrieval, event recognition, and embodied navigation in which an agent follows language instructions using visual perception and reinforcement learning.
After 2022, generative models drove rapid progress in text-to-image and text-to-video systems. By 2025, video world models could create realistic sequences and respond to user actions, blurring the line between passive generation and interactive simulation. Human-like virtual agents offer applications in health, therapy, and companionship, but also create social and safety risks.
The modality frontier is widening too: touch, smell, taste, biological structure, clinical data, and physical sensors are becoming first-class modeling targets rather than peripheral inputs.
A modality can be understood as a way a physical phenomenon is sensed, represented, and progressively abstracted.
Definition: modality
A modality is a way a physical phenomenon is expressed or perceived. A sensor converts that phenomenon into a digital representation that an AI system can process.
Modalities occupy a raw-to-abstract continuum:
- microphone waveform → recognized words → sentiment intensity;
- camera pixels → detected objects → object categories.
A modality is therefore not defined only by a device or file type; it can also denote a meaningful representational level produced by processing. This view matters because model architecture should match the level and structure being modeled.
Multimodal problems add three defining properties: heterogeneity, because modalities differ; connections, because they share some information; and interactions, because combining them can change what is knowable for a task.
Heterogeneity and connection are spectra rather than binary properties of modality pairs.
Two spectra
Heterogeneity measures how different modalities are in structure and information. Two camera views are relatively homogeneous; two languages differ in grammar and vocabulary but remain translatable; language versus vision contrasts sequential and spatial structure; language versus an arbitrary sensor can be more heterogeneous still.
Connection measures how much information the modalities share. An image and its caption may agree about a cup, a laptop, and their spatial relation, while the image alone preserves unmentioned geometry, furniture, color, and size. Shared and unique information coexist.
This prevents two common mistakes: treating modalities as either identical or unrelated, and forcing all information into a shared embedding. A useful model should exploit the overlap needed for the task while preserving modality-specific evidence that could matter later.
Multimodal interactions can contribute redundant, unique, or synergistic information to a task.
Task-dependent interaction types
Let the target be a task variable Y and the inputs be two modalities.
- Redundant information: either modality supports the same conclusion. Positive words plus a smile jointly strengthen evidence of positive sentiment.
- Unique information: one modality contributes task-relevant evidence absent from the other. A specific favorable statement may be informative even when the face stays neutral.
- Synergistic information: the target signal exists only in the relationship. Positive words paired with anger can imply sarcasm, although neither channel expresses sarcasm alone.
This yields the lecture’s compact definition: multimodal learning studies heterogeneous, connected, and interacting data. Heterogeneity explains why joint modeling is hard; connection and interaction explain why it can outperform isolated unimodal models.
Multimodal representation learning can use fusion, coordination, or fission to organize information.
Challenge 1: representation
Start with one basic element from each modality, such as a word, image region, facial expression, sensor sample, or molecule.
- Fusion: map several inputs into one joint representation that combines their information.
- Coordination: keep one representation per modality, but constrain their relationship with a similarity or matching objective. This supports retrieval because either side can be encoded independently and compared.
- Fission: learn a factorized or disentangled set of components, including shared factors and modality-specific factors.
The transcript briefly repeats the word fusion while describing the third option, but the earlier enumeration and the factorized description identify it as fission. The architectural choice determines which commonalities are shared and which heterogeneous properties remain distinct, so representation is a first-order decision in nearly every multimodal system.
Alignment connects elements across modalities at discrete, continuous, and contextual levels.
Challenge 2: alignment
Representation considered one element from each modality; alignment asks how many elements correspond across streams.
1. Discrete alignment: match already segmented units, such as a caption token to an image region. Its convenience depends on the strong assumption that meaningful discrete boundaries are known.
2. Continuous alignment: determine segmentation, temporal granularity, and correspondence when a signal such as a high-frequency sensor stream has no obvious token boundaries.
3. Contextualized representation: use alignment itself to improve downstream features. Attention in language models is an example: relationships among context elements shape the representation used for prediction.
Multimodal alignment generalizes this contextualization across inputs with different sampling rates, structures, and semantic units.
Reasoning and generation extend multimodal systems beyond direct perception.
Challenges 3 and 4
Reasoning combines evidence and knowledge over several inferential steps. It can be implicit in stacked neural layers or explicit in a chain, tree, search process, attention map, or neurosymbolic program. Three design questions recur: how intermediate concepts are represented, what topology the reasoning process uses, and how domain knowledge is injected without losing interpretability.
Generation produces new raw modality content while preserving coherence and alignment with its conditions. The lecture separates three cases:
- summarization: compress a long video or document into salient information;
- translation: express the same content in another modality, such as text to image or video;
- creation: expand a small seed into additional synchronized content, such as video, audio, and dialogue from an initial prompt or frame.
Together, these capabilities move a system beyond recognizing what is present.
Transfer addresses uneven data resources, while quantification explains and improves multimodal learning.
Challenges 5 and 6
Transfer handles resource imbalance. A target modality may have little data or few labels because collection is expensive, noisy, or private. Knowledge from a resource-rich modality—such as an LLM, textbooks, or public medical data—can support prediction in the scarce target.
Transfer includes:
- cross-modal adaptation from a rich source to a different target;
- co-learning, where an auxiliary modality is available during training but absent at test time;
- model induction, where only black-box prompts and outputs are accessible rather than model internals.
Quantification is the explanatory counterpart to challenges that build models. It uses empirical and theoretical analysis to characterize heterogeneity, modality interaction, and learning dynamics, asking why a method works, when it fails, and how to make training more stable and appropriately balanced.
Six recurring challenges organize the technical study of multimodal learning.
Six-challenge checklist
1. Representation: should inputs be fused, separately coordinated, or factorized into shared and unique components?
2. Alignment: which discrete or continuous elements correspond across modalities?
3. Reasoning: how should evidence be composed over multiple interpretable steps?
4. Generation: how can a model summarize, translate, or create coherent modality content?
5. Transfer: how can a resource-rich modality improve a scarce or unlabeled target?
6. Quantification: why does the learning process work, when does it fail, and how can it be stabilized?
These are not unrelated tasks. They all arise from one productive tension: modalities are heterogeneous enough to require different treatment, yet connected and interactive enough that modeling them together can reveal redundant, unique, and synergistic information.
The course targets grounded multimodal models, synchronized generation, physical sensing, health, and interactive agents.
Where the course is heading
The instructor highlights five frontiers: LLM-style question answering grounded in non-language modalities; efficient generation of several synchronized modalities; AI for physical sensing; multimodal health systems; and agents that perceive and act in digital or embodied environments.
The course will revisit these frontiers through the three defining properties and six technical challenges, moving from principles to recent methods and application domains. The immediate next step is a lecture on multimodal datasets and tasks. Students should use it to select a project domain, identify the modalities and task they care about, form teams of two or three, and begin the literature review that will support a concrete proposal.
Enjoy Reading This Article?
Here are some more articles you might like to read next: