Stanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification
Verification closes the gap between generating a correct candidate and reliably selecting it.
Generation is only half of test-time scaling. Pass@k can rise while delivered accuracy remains poor because the system still needs to recognize the correct trajectory. GSM8K, with roughly 8,500 multi-step grade-school problems, provides a compact testbed for studying this generator–verifier gap.
A learned verifier ranks many sampled solutions using correctness labels while leaving the generator responsible for proposing answers.
Basic verifier recipe. Fine-tune or prompt a generator, sample many complete solutions, label them from known answers, and train a separate scalar correctness head. At inference, score every candidate and select the maximum. The original setup also retained a language-modeling loss; with modern instruction-following generators, the initial task-specific generator fine-tuning may be less necessary.
Last-token outcome scoring becomes valuable only with enough verifier data and can outperform generator-only fine-tuning.
Outcome verification needs scale. Intermediate token scores may fluctuate, but the system ultimately learns one judgment for the full solution. Sparse labels do not teach a reliable decision boundary; sufficient verifier data lets selection improve beyond simply specializing the generator.
Verification gains depend on generator capacity and candidate count, and excessive best-of-N sampling can reduce accuracy.
Compute allocation is not monotonic. More samples help only while the verifier can rank the increasingly close candidates. The best division of parameters and FLOPs between generator, candidate count, and verifier remains empirical rather than a rule that always favors more search.
Extreme best-of-N selection magnifies tiny verifier errors, whereas a separate verifier can steer a general model without retraining it.
Why accuracy can fall at large N. The maximum of many noisy verifier estimates is increasingly likely to be an error. Parallel sampling reduces latency but not this statistical failure. External verification remains useful because it can specialize selection while leaving the underlying generator broadly capable.
Process reward models assign stepwise credit and catch invalid reasoning that happens to reach the right final answer.
Process supervision improves diagnosis. A final answer can be correct by accident. A PRM supplies denser credit, supports retry decisions, and favors reasoning that people can inspect. Its trajectory score penalizes any low-confidence step instead of allowing a lucky outcome to hide a broken derivation.
PRM800K uses targeted human annotation to make process verification data-efficient and robust under distribution shift.
Target the hard negatives. Human effort is most valuable on plausible-looking solutions with a subtle bad step, such as an otherwise sound equation ending with incorrect arithmetic. Iteratively selecting these cases teaches a PRM the distinctions that matter most at inference.
Process rewards add label-cost, threshold, and reward-hacking trade-offs, so they are often combined with outcome checks.
PRMs are not complete objectives. Step quality and final correctness answer different questions. Combining PRM and ORM signals can discourage both invalid local reasoning and globally failed trajectories. When a PRM becomes a training reward rather than only a selector, explicit tests for shortcutting and omitted reasoning are essential.
Math-Shepherd creates automatic process labels by estimating whether rollouts from each step can still reach the correct answer.
Rollouts approximate step potential. A step receives credit when continuation sampling shows that it can lead to success. Hard labels ask whether success is possible; soft labels estimate its frequency. The signal is scalable but imperfect: hard tasks can produce zero successful rollouts, and a wrong step may look viable if a later continuation repairs it.
Math-Shepherd uses its automatic PRM both for best-of-N selection and reinforcement-learning rewards.
A self-improvement loop without step labels. The model generates rollouts, final answers create automatic process targets, the PRM selects better candidates, and the same reward improves the generator through RL. Verification and generator training can be combined, although returns eventually plateau.
Rubrics, executable tools, and deliberate FLOP allocation can strengthen process verification.
Use the strongest available signal. Model judgments are useful when execution is unavailable, but deterministic tools can directly test arithmetic, code, or constraints. Evaluation should control training data and inference FLOPs; otherwise an apparent verifier advantage may really be a data or compute advantage.
Weaver combines multiple imperfect and diverse verifiers instead of depending on one new judge.
Diversity can narrow the verifier gap. Different judges make correlated but non-identical mistakes. Weaver treats their agreement and disagreement as supervision, provided low-quality members are filtered and redundant judges are not mistaken for independent evidence.
Weaver normalizes, filters, weights, and combines verifier scores to estimate candidate correctness.
Score → filter → weight → select. Filtering is essential because indiscriminately averaging weak judges can degrade results. Weaver borrows weak-supervision ideas to infer which agreement patterns are trustworthy, then ranks candidates using the combined probability rather than a raw vote.
Verification compute can scale through more candidates, larger models, or more verifiers, and Weaver improves the delivered answer rather than only oracle coverage.
Three test-time scaling axes. Systems can buy more samples, a stronger generator or verifier, or a broader verifier ensemble. Weaver’s gains are measured on the selected top answer, showing that weak-to-strong aggregation can turn small-model coverage into useful end-to-end accuracy.
Distilling Weaver compresses an expensive verifier ensemble into a small selector while retaining nearly all of its accuracy.
Train the ensemble once, deploy one judge. Distillation transfers the combined decision rule into a compact model, preserving most ranking quality without recurring multi-verifier expense. This turns verifier scaling from a permanent serving cost into an offline training cost.
Verification has progressed from outcome ranking to process supervision, automatic rollout labels, and weak-to-strong ensembles.
A layered verification stack. Outcomes check whether the destination is correct; process rewards inspect the path; tools supply executable evidence; ensembles reduce dependence on one fallible judge. These components can select answers or create rewards for the next round of model improvement.
Executable tests can verify coding agents, but optimizing pass@1 must preserve useful exploration and verifier diversity.
No substantive slide was displayed during this discussion segment, so the note is presented without a representative frame.
Reliability should not collapse diversity. Strong pass@1 is efficient, but exploration remains valuable for discovering rare solutions and creating training data. Coding offers direct executable feedback; other domains need diverse judges and careful bias checks. The open design problem is to improve selection without making the generator prematurely narrow.
Enjoy Reading This Article?
Here are some more articles you might like to read next: