Lecture 11 – Cross-Modal Transfer (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
Cross-modal transfer moves knowledge from a resource-rich modality or model to a resource-poor target through transfer learning, co-learning, or model induction.
Three transfer mechanisms
- Transfer learning: Start from a powerful resource-rich model and adapt its parameters or knowledge to a low-resource target.
- Co-learning: Use auxiliary modality \(B\) as an extra input or output target while training the model for primary modality \(A\), then deploy using only \(A\).
- Model induction: Keep the models architecturally separate and exchange predictions or behaviors at the wrapper level.
The setting is especially valuable when the target has few labels, noise, missing observations, or access constraints such as medical privacy.
Conventional fine-tuning, in-context learning, instruction tuning, multitask learning, and self-supervision are all mechanisms that can support cross-modal transfer.
Reusable starting points
A pretrained language, vision, or multimodal model can initialize a target system, then be adapted through fine-tuning, instruction tuning, or in-context learning.
The lecture distinguishes several axes:
- Task transfer: Reuse input \(X\) while moving from target \(Y_1\) to \(Y_2\).
- Cross-modal transfer: Move from input \(X_1\) to \(X_2\) for the same or related target.
- Multitask and multimodal learning: Share parameters across multiple outputs or inputs.
- Self-supervision: First predict held-out structure within \(X\), then adapt the learned representation to label \(Y\).
Large generalist models promote transfer by sharing parameters across many modalities and tasks through a common sequence interface.
Why generalist pretraining can transfer
Shared parameters create a route for information learned from one modality or task to influence another. Internet video may help a scarce robotics-control dataset; abundant tables outside medicine may improve tabular features for healthcare.
The unifying approximation is to express each modality as a sequence: words for language, samples for speech and sensors, patches or frames for vision, and time points for force or proprioception. A common transformer can then process all streams, with a modality identifier marking each token’s source.
A hierarchical multimodal multitask architecture can reuse the same unimodal and cross-modal transformer parameters across heterogeneous domains.
Shared hierarchical encoder
1. Add a modality embedding to each sequence so the network can distinguish language, vision, audio, tables, or sensors.
2. Run a shared unimodal transformer to model interactions within each stream.
3. Run shared cross-modal transformer blocks over every available modality pair; three modalities require three pairwise combinations.
4. Attach task-specific heads to the resulting unimodal and multimodal representations.
The striking claim is parameter reuse: the same blocks can learn speech-gesture interactions and force-proprioception interactions even though their downstream heads differ.
Pretraining on more multimodal tasks improves transfer most strongly when the target task has little data.
Evidence for breadth helping scarcity
HighMMT transfers a shared multimodal multitask backbone to a new combination of inputs and a new prediction head. In the reported experiments, target performance rises as more source datasets are included, and the gain correlates inversely with target-data volume: low-resource tasks benefit most.
Unified multitask training can match or exceed separately trained models, although small differences require confidence intervals and significance checks. Gato illustrates the related idea that visual classification experience can improve perception and action in embodied robotics.
It remains unsettled whether fully unified native multimodal models or modular language models with modality adapters will dominate.
Unification is a design bet, not a settled answer
- Adapter route: Pretrain a strong language model, then attach vision, audio, or speech interfaces.
- Native multimodal route: Train all modalities jointly from the beginning.
- Modular route: Keep specialized models and orchestrate them compositionally for reasoning.
A fully shared model assumes that serialization preserves enough modality structure and that a simple modality tag can express deep heterogeneity. Those assumptions can fail for spatial, relational, tabular, or physical signals, so native unification must be compared against specialization rather than treated as automatically superior.
Unified-model research ranges from shared architectures with separate weights to shared encoders and fully multimodal multitask systems, but real-world generalization remains difficult.
Degrees of sharing
Perceiver uses one architectural pattern across modalities but may learn different parameters. Later image-text encoders and PolyViT-style systems share both architecture and weights across more modalities. Fully multimodal multitask systems additionally combine inputs and support many outputs.
### Open challenges
Progress remains concentrated in digital image, text, and audio data. Real-world sensors are scarce; transformers do not consistently dominate classical table or time-series methods; unified models demand more compute and data; and interpretability remains poor. Architectural convergence can focus engineering and theory on one backbone, but may sacrifice useful specialization.
Fusion-based co-learning uses an auxiliary modality during training and substitutes a missing-value representation for it at test time.
Train multimodally, test unimodally
During training, fit \(\\hat{Y}=F(A,B)\) with any early, late, additive, or multiplicative fusion method. At deployment, evaluate \(F(A,b_0)\), where \(b_0\) is a missing-modality representation such as zeros or the training mean.
The correct baseline is an independently trained model \(G(A)\) because both systems receive only \(A\) at test time. If \(F(A,b_0)\) outperforms \(G(A)\), the advantage came from using \(B\) to shape learning rather than from privileged deployment information.
Fusion co-learning may help through extra information, implicit cross-modal prediction, and regularization, but careful ablations are required.
Why might it work?
- Information enrichment: Joint training encourages \(A\) to encode structure predictive of \(B\), so the model can partly reconstruct or imagine the absent view.
- Regularization: A hypothesis must explain two views, eliminating some complex functions that fit \(A\) alone but generalize poorly.
These explanations are plausible but not a complete theory.
### Use the right ablation
Compare full \(F(A,B)\), missing-view \(F(A,b_0)\), and unimodal \(G(A)\). The first gap measures the test-time contribution of \(B\); the second isolates the value of multimodal training. Explicit modality dropout can further reduce the train-test shift.
Classic image-text and audio-visual experiments show that multimodal training can improve performance after one modality is removed.
Early empirical evidence
A multimodal Boltzmann machine learned an image-text joint distribution, generated an internal text view when only an image was supplied, and outperformed image-only models. Audio-visual speech work similarly learned shared features from lip motion and sound, then evaluated with one stream masked.
These examples motivate a stronger training curriculum: expose the model to increasingly frequent missing views so it learns to reconstruct or compensate for them before deployment. Human sensory plasticity is an intuition for this idea, not experimental evidence about machine-learning mechanisms.
Alignment-based co-learning creates a shared geometric structure that enables zero-shot prediction in the primary modality.
Zero-shot transfer through shared geometry
Contrastive learning places image clusters near the embeddings of their category words. Visual similarity puts an unseen cat image near known dog images, while linguistic similarity puts the word cat near dog. The two neighborhoods meet in the aligned space, so nearest-neighbor comparison can assign the unseen label.
No cat image-label training is required, and inference still begins with only the image. Text contributes semantic geometry during training rather than privileged test-time evidence.
Scaled alignment methods improve unimodal representations and let robots operate when either vision or touch is unavailable.
Alignment benefits each view independently
CLIP-style training does more than compare images with text: its visual embeddings transfer to detection and segmentation, and its text embeddings can support language classification.
For robots, jointly predicting or aligning vision and touch creates a representation usable when the camera is occluded or tactile hardware fails. The surviving modality maps into the shared space and retains information learned from the missing one.
Translation-based co-learning uses the auxiliary modality as a training target, and cycle consistency discourages the model from ignoring that target.
Put the auxiliary modality on the output side
Train a primary encoder \(h(A)\) for two objectives: predict task label \(Y\) and translate into auxiliary modality \(B\). At test time, discard the translation head and compute \(Y\) from \(A\) alone.
For sentiment, text can be trained to predict likely facial expression as well as polarity. If the network ignores the difficult visual target, add cycle consistency: translate \(A \\rightarrow \\hat{B} \\rightarrow \\hat{A}\) and penalize disagreement between \(A\) and \(\\hat{A}\). The cycle forces the latent representation to preserve cross-modal structure.
Cross-modal prediction supplies compositional or dense supervision when the main task label is sparse or low resolution.
Match supervision density to the input
Translation can make relations explicit, such as verbalizing that a red cross lies below a square, and visual targets can enrich masked-language pretraining.
The medical example exposes a resolution mismatch: eight to ten hours of breathing measurements may receive only one Parkinson’s label. Predicting synchronized EEG provides a dense target at a comparable time scale and sampling rate. The breathing encoder must model moment-to-moment physiology, producing features that improve the sparse disease prediction even after the EEG head is removed.
Self-training induces improvement without modifying a model’s architecture by iteratively adding high-confidence pseudo-labels.
Self-training loop
1. Train a classifier on the trusted labeled subset.
2. Predict labels and confidence scores for an unlabeled pool.
3. Add only the highest-confidence pseudo-labeled examples to the training set.
4. Retrain, then repeat on the remaining pool.
Selecting points far from the current decision boundary can move that boundary and unlock new correct labels. Adding every pseudo-label at once usually preserves the original errors. Because the method only consumes predictions, it works at the model-wrapper level and resembles modern synthetic-data generation and filtering.
Co-training extends self-training to two redundant views whose separate classifiers exchange confident pseudo-labels.
Co-training across views
Train \(f_1\) on labeled \(X_1\) and \(f_2\) on labeled \(X_2\). Each classifier selects examples on which it is confident, supplies those pseudo-labels to retrain the other classifier, and repeats until the unlabeled pool is exhausted.
The classic guarantees rely on assumptions including informative, redundant views. Modern instances exchange labels between RGB and optical-flow video models, between BERT- and GPT-style models, or among debating language-model agents.
### Full taxonomy
Transfer shares parameters or starting knowledge; co-learning changes the representation objective using training-only modalities; induction keeps model internals fixed and exchanges predictions at the API level.
Enjoy Reading This Article?
Here are some more articles you might like to read next: