Lecture 3 – Data and Heterogeneity (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
The course work turns data understanding into an applied workflow before the lecture generalizes those ideas.
From assignment to modeling principle
The opening connects the course deliverables to the lecture’s central workflow:
- Interrogate the data first: find it, visualize it, label it, and understand its failure modes.
- Formulate a research claim: motivate the problem, position it against prior work, name the dataset, and define experiments and questions.
- Choose architecture from structure: use the properties of each modality to select or design unimodal models before attempting fusion or alignment.
The practical message is that a model is only one part of an iterative research system spanning data, objectives, experiments, and evaluation.
A modality can be viewed along a spectrum from raw sensor measurements to increasingly abstract task representations.
Representation sits on an abstraction ladder
A modality begins with a physical phenomenon and a sensor, but a learning system need not operate at the rawest level. It may consume a speech waveform or image directly, or work with extracted words, objects, sentiment, or semantic classes.
Common encodings illustrate how representation choices impose structure:
- Vision: an H × W × C tensor, or a flattened pixel sequence.
- Language: tokens, bag-of-words indicators, or an ordered token sequence.
- Audio: amplitude over time, or a time-frequency spectrogram produced by a Fourier transform.
Choose the abstraction and encoding whose retained information is useful for the downstream target.
Sensing, tabular, graph, set, and point-cloud data differ chiefly in their element structure and ordering assumptions.
Less familiar modalities reveal different symmetries
- Sensing: multichannel time series such as robot joint angles support state, object, next-action, or next-state prediction.
- Tables: fields can be heterogeneous, but merely reordering columns should not change the semantic record.
- Graphs: nodes and edges support node classification, anomaly detection, link prediction, and the use of structured knowledge in downstream reasoning.
- Sets: item order carries no meaning, so set-level outputs should ignore permutations.
- Point clouds: unordered XYZ coordinates represent 3D objects and support classification or generation.
These are not superficial file-format differences: their ordering and relational assumptions determine which transformations a model should respect.
A modality profile exposes five dimensions that determine how heterogeneous modalities should be modeled together.
The five-part modality profile
1. Elements and distribution: What are the elementary units, how are they represented, and are they discrete or continuous?
2. Granularity: At what rate are elements sampled, such as objects per image or words per minute?
3. Structure: How do elements compose—spatially, sequentially, hierarchically, or relationally?
4. Information: What signal is carried by each element and by the modality as a whole? Entropy, mutual information, or density can formalize this when useful.
5. Noise and missingness: Which corruptions are characteristic—blur, occlusion, darkness, typos, misspeaking, or absent measurements?
Add the target tasks to this profile, then compare profiles across modalities. The mismatches make multimodal heterogeneity concrete rather than treating every input as an interchangeable tensor.
The modality-profile framework can organize even highly abstract domains such as financial risk.
Worked reasoning pattern for an unfamiliar modality
For a space such as financial risk, ask:
- What are the elementary risk types, and which are frequent versus long-tailed?
- Do risks compose across time, geography, firms, or industries?
- Which measurements carry useful information, and which uncertainties dominate?
- Which categories are commonly confused, producing false positives or false negatives?
- If the input is a market time series, what intrinsic noise and label space should the model handle?
The profile is a scaffold, not a closed taxonomy. Its value is forcing explicit assumptions before modeling an unfamiliar domain.
Supervised, unsupervised, and self-supervised learning differ mainly in where useful targets come from.
Three sources of training signal
- Supervised learning: learn X → Y from human- or expert-provided labels. Examples include object categories, protein-binding outcomes, translation pairs, and instruction data for supervised LLM fine-tuning.
- Unsupervised learning: learn structure from unlabeled X, for example lower-dimensional embeddings or a generative model of images and proteins.
- Self-supervised learning: generate a target from the input with a program, then train predictively at scale. Examples include one image region predicting another, masked-token recovery, and next-token prediction.
Self-supervision blurs the old boundary: its data are unlabeled by people, but its automatically constructed objective behaves like supervision.
Reinforcement learning optimizes long-horizon behavior through repeated state, action, and reward interactions.
From one-step prediction to long-horizon interaction
Supervised and self-supervised objectives usually finish after one prediction. Reinforcement learning instead repeats a loop of state → action → new state → reward. A good action may not maximize the immediate reward; it may be valuable because it enables a better delayed outcome.
The lecture maps this pattern onto:
- a robot taking many navigation actions before completing a kitchen task;
- Go or Atari agents receiving informative reward only after long sequences; and
- an LLM producing several dialogue turns before a person gives an overall satisfaction signal.
The essential distinction is optimization over trajectories and delayed credit, not merely the presence of a scalar score.
Many modern learning paradigms extend the three basic regimes by changing inputs, outputs, transfer paths, or interaction.
Extensions as input-output diagrams
- Multimodal supervision: X₁ + X₂ → Y.
- Multitask learning: X → Y₁ + Y₂, exploiting related outputs.
- Transfer learning: pretrain X → Y₁, then adapt the same input representation for a lower-data Y₂.
- Cross-modal transfer: learn X₁ → Y, then transfer useful information to X₂ → Y when the second modality has fewer labels.
- Self-supervised pretraining: first learn X → X′, then reuse the representation for X → Y.
- LLM adaptation: replace a narrow fixed label with open-ended language outputs and explanations.
- Curriculum or continual learning: progress from easier targets to harder ones.
- Human-in-the-loop learning: use subjective judgments while distilling scalable automatic metrics.
This notation separates what changes—inputs, outputs, supervision, or interaction—from the underlying learning machinery.
Reliable learning balances model capacity with disciplined validation, data preparation, and a task-aligned metric.
Capacity and evaluation must be designed together
A hypothesis class defines the functions available to the learner. Too little capacity causes underfitting: the model cannot capture the input-output relation. Excessive effective capacity can cause overfitting: the model memorizes observed points but fails on new data.
Use the splits for distinct purposes:
- Training: fit parameters.
- Validation: select models and tune hyperparameters.
- Test: report the final result once; do not tune on it.
Before model training, decide the label budget, consider automatic or unlabeled targets, normalize and standardize inputs, find outliers and noisy samples, inspect raw data and low-dimensional clusters, and select a metric that genuinely rewards useful behavior. A poorly aligned metric cannot guide a good model.
Dataset size should be justified relative to prior benchmarks, task novelty, diversity, and the leverage supplied by pretrained models.
There is no context-free sample-count rule
For an established benchmark, the new dataset must compete with predecessors through greater scale, broader coverage, stronger diversity, or another defensible advantage. For a new domain, the creator has more latitude but must still show that the task matters and that the dataset supports credible experiments.
The historical baseline of thousands to tens of thousands of examples reflected training models from scratch. Pretrained models, fine-tuning, few-shot methods, and in-context learning can extract value from hundreds—or sometimes only a handful—of carefully curated cases. For this course, 50–100 high-quality labeled examples can be a reasonable starting point, with collection expanded iteratively when experiments expose gaps.
Model choice lies on a spectrum from domain-specific construction to general-purpose reuse, but all deep models can be viewed as differentiable representation and aggregation pipelines.
Two complementary views of model design
Reuse spectrum: move from a fully custom network and objective, through adapted pretrained components, to off-the-shelf zero- or few-shot inference. The appropriate point depends on semantic granularity, data and label availability, training and deployment resources, usability, and explainability.
Compositional view: nearly every deep architecture alternates:
1. differentiable representation functions such as linear layers, nonlinearities, convolutions, and self-attention; and
2. differentiable aggregation functions such as addition, maxima, concatenation, or cross-attention.
The hierarchy produces task-level features, the loss compares predictions with targets, and backpropagation updates the trainable modules. Architecture design is therefore the deliberate placement of reusable extraction and aggregation operations.
Set models should share an encoder across elements and use a permutation-invariant aggregation function.
Encode set symmetry directly
For set elements xᵢ, a set-level classifier should behave like ρ(Σᵢ φ(xᵢ)):
- the same encoder φ processes every element; and
- a commutative reducer such as summation removes order before a final predictor ρ.
Two tempting alternatives fail. Position-specific encoders make the representation depend on which element occupies each slot, while concatenation preserves an arbitrary order even if the encoders are shared. With five elements, learning around either mistake could require examples for all 5! = 120 permutations. The correct inductive bias converts those equivalent arrangements into training signal automatically.
Invariances specify transformations a task output should ignore, whereas equivariances specify transformations the output should track.
Invariance versus equivariance
Let T transform an input and f be the model:
- Invariant: f(Tx) = f(x). The output should ignore T, as a set classifier ignores permutations or an object classifier preserves a label after an allowed translation or recoloring.
- Equivariant: f(Tx) = T f(x). The output should change in the corresponding way, as a segmentation mask moves when the source image is translated or rotates when the image rotates.
The same physical transformation can demand different behavior for different tasks. The output semantics—not the modality name alone—decide whether invariance or equivariance is appropriate.
Sequence models combine time-shared parameters with order-sensitive information accumulation.
Sequence architecture follows two requirements
1. Share parameters over time. The tokenizer, embedding, or encoder applied at one position is reused at every other position, so shifting a sequence with padding does not create a wholly new computation.
2. Accumulate ordered context. A representation at time t depends on earlier states or an otherwise order-aware context, so swapping words can change meaning.
RNNs and LSTMs satisfy both through shared input, recurrent, and output matrices plus hidden-state recurrence. Autoregressive decoders mirror the pattern. Temporal convolutional networks replace recurrence with shared causal filters, while state-space models use another structured accumulation mechanism; the inductive requirements remain the same.
Attention replaces recurrent aggregation with learned, context-dependent weighted combinations while retaining shared parameters and positional sensitivity.
Attention is dynamic information aggregation
For a sequence matrix X, attention forms queries, keys, and values with shared projections. Pairwise query-key products produce a token-by-token score matrix. Scaling by √d prevents dot-product variance from growing with feature dimension, and row-wise softmax gives nonnegative weights that sum to one. Multiplying by the value matrix yields a contextualized representation for every token.
In compact form: Attention(Q,K,V) = softmax(QKᵀ / √d)V.
The weights change with the sentence, so aggregation is context-dependent rather than a fixed recurrence. Shared projection matrices preserve parameter sharing across positions; positional embeddings or related mechanisms restore sensitivity to order. Encoder-decoder attention uses the same idea to align output tokens with relevant input tokens, including non-diagonal alignments when languages order words differently.
CNNs and vision transformers encode spatial structure through local elements, parameter sharing, and structured aggregation.
Two implementations of spatial inductive bias
A dense pixel-to-output layer is both expensive—a 200 × 200 image already has 40,000 input dimensions—and sensitive to where a pattern happens to occur.
CNNs address this with sparse local receptive fields, one filter shared across spatial locations, and pooling that aggregates nearby responses. These choices improve efficiency and tolerance to small translations or scale changes.
Vision transformers split the image into K × K patches, apply shared patch encoding and attention projections, and aggregate evidence with learned patch-to-patch attention. Patch tokens and attention look different from convolutions and max pooling, yet both systems reuse parameters across spatial elements and combine local representations according to image structure.
Graph neural networks generalize the shared-parameter and structure-aware aggregation view across sets, spatial grids, and sequences.
Graphs expose the lecture’s general recipe
A graph neural network applies the same node encoder everywhere, making node relabeling or reordering irrelevant, and aggregates messages only along edges, making representations sensitive to actual relationships. Repeated message passing lets each node incorporate increasingly distant neighborhoods.
Special cases reveal the unification:
- no edges gives an unordered set;
- a regular spatial neighborhood gives image-like structure; and
- a chain gives temporal structure.
The reusable design loop is: identify elementary units, encode transformations that should leave outputs unchanged through parameter sharing, aggregate along relations that outputs should track, choose an objective, evaluate, and collect or clean more data as failures reveal what is missing.
Enjoy Reading This Article?
Here are some more articles you might like to read next: