Stanford CS329A Self-Improving AI Agents | Part 7 | Self-Improvement and Deep Research Agents
Search-based improvement requires both broad exploration and reliable selection.
Central idea. A capable model may assign some probability to a correct solution without producing it on the first attempt. Search improves the system only when it combines two functions:
1. Explore the model’s output space through diverse samples or queries.
2. Select and refine the candidates that are actually useful.
The lecture develops this pattern first for AlphaCode-style program search and then for Search-o1-style deep research.
AlphaCode transformed code completion into end-to-end competitive-programming problem solving.
Why the task is harder. AlphaCode is not merely an autocomplete system. Given a competitive-programming prompt, it must infer the intended computation, select a solution strategy, and implement the entire program under a formal input-output contract.
The lecture presents its roughly mid-field Codeforces performance across ten contests as an early demonstration that AI code systems could generalize beyond narrow completions to longer, end-to-end problems.
The original AlphaCode trained a specialized code generator and fine-tuned it to favor meaningful token patterns.
Generator training. AlphaCode learned from two complementary sources: broad GitHub code for programming syntax and idioms, and CodeContests for the structure of competitive-programming problems and solutions.
During fine-tuning, GOLD-style weighting emphasizes high-likelihood tokens and downweights low-likelihood ones, with the goal of concentrating probability on more meaningful program patterns. The lecture also notes value conditioning and prediction as mechanisms for shaping which programs the generator prefers.
AlphaCode searched one million generated programs through execution filtering and behavioral clustering.
Massive-search pipeline. AlphaCode:
1. Samples one million programs per problem, half Python and half C++, with high-temperature and randomized prompt metadata.
2. Removes programs that fail the examples supplied in the problem.
3. Uses a learned test-input generator to create additional probes.
4. Clusters programs that look different syntactically but behave equivalently on those probes.
5. Selects a diverse subset for evaluation.
Clustering is essential because Codeforces permits only a small number of submissions; sending many variants of the same algorithm wastes that budget.
AlphaCode’s contest performance exposed both distribution-shift and candidate-selection bottlenecks.
Observed performance and variance. The lecture reports an approximately 54th-percentile aggregate result over ten Codeforces contests. More important than the exact table entry is why performance varied:
- Some contests were closer to the training distribution than others.
- Search coverage and final selection are different capabilities.
- A correct program can exist among the samples while the selection stage chooses several nearly correct alternatives.
This makes candidate ranking a first-class bottleneck rather than an implementation detail.
Pass@k measures raw coverage, while 10@k measures coverage constrained by a realistic selection budget.
Two evaluation questions. pass@k asks whether the correct program appears anywhere among k attempts. 10@k asks whether a system can find ten good submissions inside that same pool.
The reported trends are consistent:
- A 41B model has a better scaling slope than a 9B model.
- Increasing the sampling budget from thousands toward one million improves solve rate.
- Adding clustering improves the constrained ten-submission result by preserving diversity.
At the high end, the lecture contrasts pass@k above roughly 40% with 10@k around 30%, quantifying the cost of imperfect selection.
Brute-force sampling helps only while it discovers genuinely new solutions and remains impractical for difficult problems.
Limits of brute force. More samples are valuable only when they increase solution diversity. If ten times more generations reproduce the same approaches, coverage will not improve.
Other limitations were structural:
- Language-model loss is a weak proxy for program correctness because many different programs can solve a task.
- Dynamic-programming and constructive problems remained difficult.
- One million samples per task is expensive and slow.
- Under a time limit, candidates must be ranked as they arrive.
- Hard tasks often require multi-step correction rather than a single, one-shot program.
Training-data similarity checks nevertheless suggested that successful outputs contained novel code rather than simply copying stored solutions.
AlphaCode 2 combined a fine-tuned Gemini Pro family, higher-quality data, and a learned correctness scorer.
Three upgrades over AlphaCode. AlphaCode 2 changes the search stack rather than merely increasing its budget:
1. Stronger base model: fine-tune Gemini Pro rather than pretrain a code model from scratch.
2. Diverse generator family: vary data partitions, tags, difficulty levels, and hyperparameters so parallel samples explore different modes.
3. Learned ranking: train a reward-like scorer on high-quality, human-vetted comparisons to estimate correctness between 0 and 1.
CodeContests V2 supplies improved open training problems, while the separate scoring data teaches which candidates should be preferred.
AlphaCode 2’s execute-cluster-rerank pipeline matched AlphaCode with far fewer samples and nearly doubled its maximum solve rate.
AlphaCode 2 search funnel. Samples are generated only in C++ across a family of tuned models. Compilation and public-test execution remove about 95% of candidates. The remaining pool is behaviorally clustered; the largest ten clusters preserve broad approaches, and the scorer chooses the most promising program inside each cluster.
This more capable system reaches the original AlphaCode solve rate with roughly 100 rather than one million samples. At a one-million budget, the lecture reports about 43% versus 25% solve rate, attributing the gain to a stronger generator, broader diversity, and learned selection.
Feedback-guided refinement and train-time learning can exchange massive parallel search for more intelligent sequential work.
Reducing search cost. Two complementary levers can replace some brute-force sampling:
- Sequential self-refinement: execute a candidate, collect feedback, and repair it. This spends more steps on fewer trajectories and is only as good as the feedback source.
- Train-time improvement: use reinforcement learning or successful search traces to make the base solver more capable, reducing later test-time demand.
AlphaCode 2 is therefore usefully viewed as a system of specialized roles: several models explore and another model scores. Distillation could eventually compress this system into a stronger single reasoning model.
A useful scoring model needs independent but in-distribution data, and AlphaCode 2 remains costly and code-specific despite strong results.
Scorer design and remaining costs. Reward-model data should be close enough to the task to teach meaningful preferences but separate from held-out contest problems. Mixing data across stages may also be necessary to limit catastrophic forgetting.
The lecture reports AlphaCode 2 at roughly the 85th percentile, between Codeforces Expert and Candidate Master. That is a large one-year improvement, but it does not erase the drawbacks: the system is specialized, experimentation is expensive, and most generated programs never compile or pass the basic tests.
Search compute should be allocated according to task difficulty rather than uniformly across all problems.
Adaptive test-time compute. A practical controller can first estimate task difficulty, then choose a solver and budget:
- Easy: use a smaller or specialized model and stop after few samples.
- Medium: expand the parallel search or add one refinement cycle.
- Hard: route to stronger models, deeper search, and explicit feedback.
This separates two benefits that are often conflated: parallel sampling increases diversity, while sequential refinement tries to repair what the model already produced.
Embedding reasoning into the solver requires rationales, decomposition, intermediate checks, and backtracking.
No substantive slide was displayed during this discussion segment, so the note is presented without a representative frame.
From one-shot code to reasoning search. Training examples can expose the intended algorithm, intermediate plan, and rationale rather than only the final program. STaR-style data generation starts from a known answer and asks the model to reconstruct a valid reasoning path.
At inference time, complex tasks suggest a tree search: propose a step, check it, branch into continuations, and backtrack after failure. Decomposition helps when tasks share recurring solution patterns. When the needed pattern is truly out of distribution, however, automated checks may be insufficient and a human-in-the-loop scientific workflow remains appropriate.
Search-o1 targets knowledge gaps that appear as uncertainty during long-form reasoning.
Why reasoning needs iterative retrieval. A long reasoning chain may expose several independent knowledge gaps. If the model guesses at the first unfamiliar entity, that uncertainty contaminates every downstream step.
A single RAG call is adequate for a simple lookup such as current weather, but not for a multi-part scientific problem. Each stage can require a different source, and inserting long documents without analysis can overwhelm the context. Search-o1 treats uncertainty as a trigger for another retrieval action rather than something the model must reason through unaided.
Search-o1 interleaves query generation with reasoning and extracts only the relevant evidence from retrieved documents.
Search-o1 loop. During one reasoning session, the model repeatedly:
1. Detects an unresolved entity or fact.
2. Generates a targeted search/tool query.
3. Retrieves candidate documents.
4. Reasons inside each document to isolate relevant evidence.
5. Adds only that evidence to its working context and resumes the original chain.
The chemistry case illustrates the difference: unsupported guessing fails, dumping many documents still yields an incorrect count, but focused extraction integrates the missing structural fact and supports the intended answer.
Search-o1 extends Agentic RAG with document-level reasoning but assumes reliable state and context management.
Agentic RAG versus Search-o1. Both systems can invoke retrieval in the middle of reasoning. Agentic RAG generally returns the documents directly. Search-o1 first converts them into concise, query-specific notes—closer to how a researcher reads references and records only useful evidence.
That extra stage reduces noise, but the architecture must retain the current goal, earlier reasoning, and retrieved evidence in a state or memory buffer. Its benefit also depends on context-window quality: a model that cannot reason reliably over its accumulated context will still lose coherence.
Evidence extraction lets Search-o1 benefit from more documents on scientific and multi-hop tasks, subject to retrieval and domain-quality limits.
Empirical pattern. More raw context does not automatically help: direct reasoning and ordinary RAG can saturate as document count rises. Search-o1 instead improves because it filters every source before integration, enabling parallel information gathering without an equally large reasoning burden.
Reported results are strong on GPQA scientific questions and multi-hop benchmarks including HotpotQA, 2Wiki, MuSiQue, and Bamboogle. Interpret them carefully:
- Comparisons are to sampled domain experts on a specific benchmark, not to human expertise in general.
- Chemistry appears to retain more headroom than some other domains.
- Retrieval precision and recall are upstream assumptions.
- Mathematical and chemical notation may be damaged during extraction.
The mechanism’s clearest benefit is reduced uncertainty through repeated, evidence-focused searches.
Prompted search, reinforcement-learned search, and calibrated confidence are complementary routes to more reliable agents.
No substantive slide was displayed during this closing discussion.
Toward learned and calibrated search. Search-o1 orchestrates retrieval through prompting; Search-R1 aims to teach when and how to search through reinforcement learning. These approaches can coexist with explicit tool loops and later distillation.
Confidence cannot be read directly from a raw sum of output-token log probabilities. Even after more meaningful averaging or geometric aggregation, models are often overconfident—the lecture gives the intuition of behavior that is only about 50% correct while expressing roughly 80% confidence. Second-pass estimators and RL/RLHF calibration may reduce this mismatch, but confidently wrong answers remain common. Reliable agents therefore need external verification in addition to self-reported certainty.
Enjoy Reading This Article?
Here are some more articles you might like to read next: