Lecture 12 – Self-Evolving AI (MIT How to AI Almost Anything/Multimodal AI, Spring 2026)
Strong project conclusions depend more on rigorous analysis than on whether the headline result is positive.
Analyze the result, not just the score
The final project should establish three things:
1. What happened: finish the experiments and compare against credible baselines.
2. Why it happened: diagnose both successes and failures rather than hiding negative results.
3. What is new: position the method against prior work and communicate the contribution clearly.
Grading emphasizes analytical depth, novelty, and report quality—not whether every experiment wins. A poster or four-to-five-slide presentation is therefore a compression exercise: identify the few visual claims that best support the argument, gather feedback, and use it to strengthen the final paper.
The course framework treats multimodality through heterogeneity, connected information, and interacting information.
Three lenses for multimodal data
Heterogeneity asks how modalities differ. A modality profile compares elemental units, their distribution and frequency, spatial or temporal structure, information content, characteristic noise, and suitable tasks.
Connections ask how much information is shared. Correspondence may be strong, weak, or nearly independent; it may link one word to one object or a relational word to several regions. Contrastive learning captures such connected information by pulling positive cross-modal pairs together and pushing negatives apart.
Interactions ask what fusion contributes:
- redundancy: both modalities carry the same useful signal;
- uniqueness: only one modality carries the needed signal for a case; and
- synergy: the target becomes inferable only jointly, as when positive words plus negative expression imply sarcasm.
Representation and alignment are foundational technical challenges for combining heterogeneous modalities.
Represent first, then discover correspondence
Representation strategies include:
- fusion: combine modalities into one joint representation;
- coordination: retain separate representations but constrain their similarity; and
- factorization: split information into latent components with different roles.
Alignment begins as a matching problem between discrete elements such as words and image regions. Continuous alignment is harder because element boundaries and granularity must also be inferred. The final step is aligned representation: use correspondence weights to contextualize each feature. Attention and multimodal transformers operationalize this weighted update and then support pretraining, fine-tuning, and instruction tuning.
Reasoning and generation extend multimodal models from representation toward multi-step inference and realistic output creation.
From representation to inference and creation
Reasoning replaces a single prediction with a sequence of intermediate inferences. Those steps may live implicitly in neural layers or explicitly in language, attention maps, chains, or branching search trees. Training signals can come from human reasoning traces through supervised fine-tuning or from verifiers and rewards through reinforcement learning.
Generation creates or translates data rather than only classifying it. VAEs, diffusion models, and flow matching progressively improved image and video synthesis. Modern generation is usually multimodal because text, latent variables, images, or other conditions tell the model which realistic output to create.
Cross-modal transfer and quantification complete the six-challenge view by moving information to scarce modalities and measuring multimodal phenomena.
Transfer information, then measure its effect
Cross-modal transfer helps a scarce target modality borrow from abundant text, natural images, or another related source. Mechanisms include shared-backbone multitask learning, adding the auxiliary modality during training, predicting the auxiliary modality and discarding its head at test time, or black-box co-training in which one classifier labels examples for another.
Quantification is the scientific layer over the other five challenges. It measures heterogeneity, connected and interacting information, optimization behavior, and scaling so architecture choices can be explained rather than merely observed. In the six-part framework, representation and alignment are foundational; reasoning, generation, and transfer serve task needs; quantification acts as the magnifying glass.
Native multimodal models train jointly from multiple raw modalities instead of treating a pretrained language model as the fixed center.
Make multimodality part of pretraining, not an attachment
A language-first system freezes an LLM, projects visual features into its token space, and translates desired outputs back through text prompts into an image generator. It excels when the visual content is captionable, but language becomes an information bottleneck.
A native multimodal model updates or trains the central network directly on several modalities:
- late fusion: encode vision first, then combine its learned features with text; or
- early fusion: feed near-raw image patches and text tokens into the shared model.
Joint objectives such as video next-frame prediction may capture physical regularities learned by infants and animals before language. The best design is still unsettled, but text-data saturation and performance plateaus motivate direct multimodal pretraining.
Scaling laws predict how validation loss changes with model size, dataset size, and training budget.
Scaling laws turn expensive experiments into forecasts
Train model families across parameter counts N, dataset sizes D, and compute durations, then fit the relationship between those resources and held-out loss. Small models may converge quickly but plateau high; larger models may begin worse yet reach lower loss when trained long enough.
A schematic law is:L(N,D) = E + A / N^α + B / D^βE is an irreducible floor, while the remaining terms shrink with more parameters and data. Empirical runs determine the coefficients and exponents. The law helps decide whether an extra dataset, a larger model, or longer training is the best use of a limited compute budget.
A multimodal scaling law augments independent modality losses with synergy benefits and optimization competition.
Multimodal scaling has gains and interference
The proposed extension decomposes loss into four conceptual pieces:
1. an average of the two independently modeled modality losses;
2. a synergy benefit subtracted when joint information lowers loss;
3. a capacity term that improves with total model size; and
4. an optimization competition penalty that is worse when modality datasets are too small.
Fit the coefficients from runs spanning model sizes and separate modality data sizes. Power-law structure is an empirical hypothesis, so compare it against alternatives such as logarithmic forms using both fit error and held-out prediction on unseen resource configurations.
Native multimodal scaling experiments favor mixture-of-experts capacity and shared rather than modality-specific experts.
Empirical findings from native multimodal scaling
For the studied interleaved text-and-speech setting:
- early fusion has a small advantage at small scale;
- early and late fusion become comparable at larger scale, and a dedicated modality encoder may not always be necessary;
- native multimodal loss follows power laws resembling unimodal LLM scaling, with exponents affected by data and cross-modal synergy;
- sparse mixture-of-experts models scale better than dense transformers; and
- modality-agnostic experts shared across tokens outperform experts rigidly assigned to text or speech.
These are measured findings for a specific setup, not universal architectural laws, but they challenge the intuitive idea that every modality needs its own permanent expert.
Sparse mixture-of-experts models provide enormous capacity while activating only a small routed subset per input.
Sparse activation separates capacity from inference cost
A mixture-of-experts model trains many expert blocks but uses a router to activate only a small subset for each token or query. Thus a model can store knowledge across hundreds of billions of total parameters while executing perhaps ten percent of them on one pass; the lecture cites a roughly 600-billion-parameter model activating about 37 billion.
The same routing principle can specialize by interaction type rather than by modality. Redundant examples may benefit from contrastive features, unique examples from selective attention, and synergistic examples from deeper joint fusion.
Information decomposition can route examples to interaction-specific experts, with especially large gains on synergistic tasks.
Route by what fusion must compute
Interaction labels can be estimated statistically:
- pointwise mutual information approximates per-example overlap and redundancy;
- conditional-entropy-style quantities capture information unique to one modality; and
- partial information decomposition separates total predictive information into redundant, unique, and synergistic parts.
The last component is hardest because synergy is information available only from the joint observation. Scalable approximations can label or weight training examples, which are then routed to dedicated expert heads. A synergy expert yields the largest gains on sarcasm, humor, figurative language, and cartoons because ordinary video-language models often sit near chance when the modalities must be combined nontrivially. Heads may share an architecture but use distinct parameters and more layers for the harder synergistic computation.
Adding modalities can reduce performance when datasets contain shortcuts or modalities learn and generalize at different rates.
More modalities do not guarantee more useful information
Fusion can underperform because the optimizer follows easy shortcuts:
- a VQA model says a green banana is yellow from its language prior;
- it answers two people because that label dominates training questions; or
- a captioner infers gender from a laptop while never attending to the person.
Counterfactual balancing pairs the same question with images supporting different answers, forcing visual grounding. Optimization must also account for different learning rates: a mature language branch may reduce loss quickly while a harder visual branch lags, causing the joint model to ignore vision. Comparing multimodal and unimodal learning curves can expose this imbalance, after which loss weights, gradients, learning rates, or update schedules can be rebalanced.
Error comparison and stage-wise fusion reveal whether another modality offers complementary signal and how to exploit it.
Diagnose complementarity before changing fusion
Train unimodal baselines A and B, then inspect their most confident mistakes:
- overlapping errors suggest redundancy or a shared missing signal;
- different errors suggest that the modalities may correct one another.
A stage-wise fusion strategy starts with the strongest modality, freezes or preserves what it already solves, and trains the next modality on the residual error set. Further modalities address the remaining failures. This makes each added branch responsible for measurable complementary value rather than allowing it to compete indiscriminately with a dominant branch.
Self-evolving agents extend learning from model parameters to memory policies, tool use, and the surrounding harness.
Evolve the system around the model
Long multimodal histories raise cost and forgetting. Mem1 treats memory management as an RL policy: retaining or discarding information is an action, and the agent maintains a bounded internal state summarizing the useful past.
Memory is part of a wider harness—the wrapper that provides tools, APIs, guardrails, reflection, and context management without changing the base model’s parameters. Modern coding agents derive substantial capability from this hand-engineered layer. Self-evolving agents ask whether the system can automatically improve or write its own harness; self-evolving multi-agent systems extend that adaptation to several specialized agents.
AlphaEvolve-style systems improve solutions through an iterative propose, verify, remember, and resample loop while keeping model weights fixed.
Harness-level evolution loop
1. A human specifies a problem and an objective evaluator, such as algorithm runtime.
2. Language models generate diverse prompts, heuristics, or candidate programs.
3. The evaluator verifies each candidate quantitatively.
4. A database records prompts, code, scores, and the trajectory of attempts.
5. New proposals synthesize what previously worked, and the cycle repeats.
AlphaEvolve-style search has improved constants in mathematical bounds and packing problems. Structured variants such as ShinkaEvolve represent mutations as a tree and prune weak branches. Crucially, the underlying language-model weights remain fixed: improvement lives in prompts, memory, search policy, and the harness.
Theta-evolution methods feed verified improvement back into model weights through reinforcement learning.
Move evolution from context into parameters
Harness evolution already produces an objective signal: how much each verified program, proof, or algorithm improves the target metric. Theta-evolution methods treat that improvement as reinforcement-learning reward at test time.
The familiar loop—generate candidates, verify them, store the trajectory, and propose the next attempt—now also backpropagates success into the base-model weights. The system therefore evolves on two levels: its external search context and harness adapt immediately, while its parameters learn reusable discovery behavior.
Coral generalizes self-evolution to a population of specialized agents coordinated through shared memory and lifecycle management.
Population-level evolution with CORAL
CORAL seeds multiple agents with different expertise around one specification and evaluator. Each agent can modify code, run evaluations, and read or write shared memory containing prior attempts, scores, notes, and acquired skills.
A heartbeat controller periodically measures progress, removes unproductive agents, and spawns replacements from promising directions. This population combines parallel exploration, specialized roles, shared cumulative knowledge, and selection pressure—extending self-evolution from one prompt trajectory to a coordinated multi-agent ecosystem.
Enjoy Reading This Article?
Here are some more articles you might like to read next: