Stanford CS329A Self-Improving AI Agents | Part 9 | Future Research Areas
The course connects self-improving agents through feedback, search, tools, memory, planning, and evaluation.
A unified course map. Self-improvement is presented as a loop rather than a single algorithm: agents generate candidate behavior, interact with tools or environments, obtain feedback from verifiers, search or revise when needed, and turn successful experience into better future behavior. Retrieval, memory, planning, and evaluation become necessary as goals extend across multiple steps.
An agent generalizes an LLM by directing a goal-oriented process and correcting it with environmental feedback.
What makes the system agentic. A language model supplies capabilities, while an agent coordinates those capabilities toward a goal: it plans, acts, checks progress, and repairs mistakes. The next frontier is to make more of this loop self-directed without losing reliable feedback. The lecture therefore organizes future work around diversity, verification, curriculum generation, and the cost of serving intelligence.
Multi-agent fine-tuning combats the diversity collapse of self-training with a single generator.
Diversity is training signal, not decoration. Self-training repeatedly amplifies whatever distribution the generator already prefers. Multiple specialized generators broaden the candidate reasoning paths before critique begins; critics then compare and repair those paths. This attacks a central failure mode of synthetic-data flywheels: convergence onto a narrow family of superficially successful answers.
Debate trajectories, majority filtering, and supervised updates preserve diversity across repeated fine-tuning rounds.
A reusable multi-agent data loop. Generate independently, share evidence, critique, revise, aggregate, filter, and fine-tune. The reported open-model experiments suggest that this loop can preserve response diversity longer than single-agent self-training and may transfer beyond the training benchmark. These are paper-specific results, not proof that debate or majority agreement guarantees correctness.
DeepSeekMath-V2 targets proofs where a correct final answer can hide invalid reasoning.
Verification must inspect the path. Final-answer rewards are too coarse when the artifact is a proof: a lucky endpoint can conceal missing or false steps. The proposed loop teaches a verifier from expert-identified defects, then uses its critiques to improve the generator. This moves feedback from outcome matching toward explicit analysis of the reasoning chain.
A meta-verifier audits verifier critiques, enabling an iterative generator–verifier improvement loop.
Verify the verifier. A learned judge is another fallible model, so DeepSeekMath-V2 adds a second-order check on the judge’s stated reasons. This can make proof feedback more trustworthy and scalable, but it does not remove the need for sound initial supervision or extend automatically to subjective and slow-to-evaluate domains. The numerical gains are lecture-reported benchmark results.
Self-proposed curricula aim to remove human prompt selection from the self-improvement loop.
Let the learner choose its next lesson. If experts must continually supply harder prompts, data collection becomes the bottleneck. A proposer–solver system instead seeks tasks at the model’s learning frontier, while an executable environment supplies objective checks. The opportunity is an adaptive curriculum; the danger is a closed loop that invents invalid, repetitive, or reward-hacking exercises.
The proposer is rewarded for valid, moderately difficult tasks that yield mixed solver outcomes.
Difficulty must track capability. Tasks that everyone solves add little signal, while tasks nobody solves offer no positive trajectory. Rewarding mixed success keeps examples near the current competence boundary. Executability makes this curriculum practical in code because proposed tasks can be checked before they enter training; analogous validation is much harder in open-ended domains.
A task buffer supports an evolving curriculum, and synthetic coding practice can transfer beyond code.
A synthetic-data flywheel with memory. The buffer preserves prior tasks and their difficulty, allowing proposer and solver to co-evolve rather than restart each round. Reported coding and math transfer suggests that learning useful problem-solving procedures can generalize beyond the data generator’s immediate domain. Those claims remain experiment-specific and rely on reliable executable checks.
General self-improvement is limited by diversity, verifier quality, and the speed and objectivity of feedback.
Verification defines the reachable frontier. Fast objective checks make large search and RL loops possible in math and code. Slow experiments, subjective criteria, and imperfect learned rewards weaken that loop. Surrogates and decomposition can reduce cost, but both trade direct evidence for approximation; reward hacking and out-of-distribution errors remain central research risks.
Cloud-only inference faces rapidly growing compute and energy demand even though many requests may not need frontier models.
Match model scale to task demand. Centralized frontier serving is expensive, while a large share of everyday requests may not require maximum capability. This creates an opportunity to route suitable work to compact local models. Several spoken scale figures in this section are internally unclear or unit-inconsistent, so they should be treated as lecture-reported directional evidence rather than audited measurements.
Intelligence per watt combines useful task accuracy with the power needed to produce it.
Evaluate capability and efficiency together. Raw benchmark accuracy misses whether a model is economical to run. Intelligence per watt measures how much successfully completed work is obtained per unit of power, making hardware, quantization, model size, and workload mix part of the evaluation. The study’s local-model threshold and benchmark suite are operational choices, not universal definitions.
Rapid model and hardware improvements make local inference an increasingly credible complement to cloud serving.
Local capability is moving faster than its efficiency gap. Compact models can now answer far more of the evaluated workload, while local accelerators still trail specialized data-center chips on the chosen efficiency metric. Because model and hardware gains compound, routing an appropriate subset of requests to laptops or phones could reduce cloud pressure. All percentages and ratios are study-specific lecture reports.
Open research spans the foundations of test-time scaling, synthetic-data distillation, continual learning, and specialized serving systems.
Close the loop in both learning and infrastructure. Today’s pipeline alternates online experience with asynchronous data collection and fine-tuning. A stronger agent would retain useful experience continuously, while the serving stack would efficiently support branching search, repeated sampling, and tool calls. Theory, learning algorithms, and systems design therefore need to advance together.
Hybrid local–cloud routing and energy-aware architectures are key efficiency frontiers.
Route work to the cheapest capable substrate. Hybrid systems should keep suitable requests on-device and escalate only when a larger remote model is justified. Achieving that requires calibrated difficulty estimates, smooth handoff, privacy-aware design, and kernels tuned to edge hardware. Intelligence per watt offers one objective, but latency, quality, and reliability must remain part of routing decisions.
Continual learning may combine external memory, extended context, and selective weight updates.
No substantive slide was displayed during this Q&A segment.
Memory and learning solve different problems. Databases and context are the simplest way to expose an agent to changing knowledge; weight updates are more appropriate when behavior itself must change. A practical continual learner will likely combine them, deciding what to retrieve, what to retain temporarily, and what experience is valuable and safe enough to consolidate into model parameters.
Self-created environments help only when they faithfully represent the real tasks an agent must master.
No substantive slide was displayed during this closing discussion.
A simulator is useful only to the extent that it transfers. Finite games are easy to encode; open-world work is not. The research challenge is therefore less about who writes the environment and more about whether its tasks, dynamics, and rewards predict real outcomes. The lecture ends with an invitation to build on the course’s enduring feedback-and-search foundations as the field changes.
Enjoy Reading This Article?
Here are some more articles you might like to read next: