Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling
Inference scaling spends additional compute on a deployed model without changing its parameters.
A third scaling axis. Pretraining buys general capability once, fine-tuning changes behavior for a target distribution, and inference scaling allocates computation per problem. The lecture focuses on the last regime: improving a fixed model through sampling, revision, verification, and composed model calls.
Repeated sampling plus verification can expose capabilities that a smaller model rarely produces on its first attempt.
Coverage versus delivered accuracy. A model may already assign nonzero probability to a correct solution while usually returning something else. More independent attempts raise the chance that success appears, but the system only benefits if a verifier can identify it. This creates both an opportunity for smaller models and a potentially large inference bill.
Aggregate coverage follows an empirical inference-time scaling law across model families and domains.
Predictable test-time returns. Like pretraining scaling laws, the fitted inference curve estimates how much sampling is needed to reach a desired success coverage. The transcript does not state the aggregate equation clearly enough to reproduce it exactly, so the reliable takeaway is the empirical regularity rather than a specific formula.
A long tail of very hard problems produces power-law aggregate gains under repeated sampling.
Mechanism behind the curve. The probability that every attempt fails is (1 - p_i)^k; subtracting from one gives per-problem coverage. A dataset-wide power law emerges when difficulty has a sufficiently long tail. Correlated generations would weaken the simple independence model, so diversity matters as much as raw sample count.
Test-time compute is most useful in domains with strong, scalable verification signals.
Verification determines usable scaling. Mathematics, code, and constrained transformations can expose executable checks that convert coverage into trustworthy answers. Even these are conditional: passing a finite test suite does not prove full correctness when inputs or behaviors are uncovered.
The generation–verification gap leaves much of pass@k coverage unrealized.
Rare correctness is difficult to recognize. Easy problems benefit from consensus, but hard-tail solutions are neither frequent nor obviously phrased. Generating one correct trace can therefore be substantially easier than proving which trace is correct. Closing this gap is the central systems problem for inference scaling.
Verifier design can combine revision, tools, rejection signals, and diverse weak judges, but each option introduces cost and coverage assumptions.
A portfolio of imperfect checks. When positive proof is hard, systems can still eliminate contradicted candidates, retrieve external evidence, simulate consequences, or aggregate judges with different failure modes. The strongest design depends on domain-specific signal quality and the additional compute required to obtain it.
Parallel sampling and sequential revision are the two fundamental ways to spend more inference compute.
Two compute axes. Parallelism expands breadth and raises coverage. Sequential work adds depth by correcting, extending, or backtracking within a trajectory. Practical systems can nest the two—revise several candidates independently, then compare their final results.
Outcome and process reward models guide best-of-n selection, beam search, and hybrid reasoning paths.
Guide search before the final answer. ORMs choose among completed candidates, while PRMs can prune an unpromising branch early. A hybrid may apply sequential revision inside parallel candidates and use either reward type at the end. No single allocation or reward recipe dominates every task.
The best mix of parallel exploration and sequential revision depends on problem difficulty.
Adapt the budget per problem. Easy items do not need wide search, while harder items may need exploration plus deeper revision. Difficulty estimation is itself part of the controller. Test-time scaling can beat additional pretraining tokens in some regimes, but it does not erase the value of frontier base capability on the hardest questions.
Pretraining and inference compute have different economics and cannot be compared as if each problem owned a full training run.
Cost accounting matters. An inference-heavy solution may be accessible yet expensive at scale. Comparisons should amortize training across expected usage and charge each request for sampling, verification, and latency. A specialist fine-tune is another design point, not a free substitute for a general foundation model.
Tree search combines exploration and revision by scoring partial trajectories and pruning weak branches.
Inference becomes resource control. Tree search turns token generation into an adjustable per-problem budget. The controller must decide how wide to explore, how deep to revise, and when added work no longer justifies its cost.
Archon formalizes test-time scaling as inference-architecture search under model-call and quality constraints.
Search for the workflow, not only the answer. Instead of prescribing repeated sampling or one fixed agent loop, Archon treats the orchestration graph as an optimization variable. Its output specifies which models and operations should be composed for the target tasks and budget.
Archon’s primitive operators generate, fuse, critique, rank, and verify candidate responses.
A reusable inference vocabulary. These operators separate proposing information from evaluating and integrating it. Once represented as graph nodes, they can be rearranged, assigned to heterogeneous models, and evaluated as complete systems.
Fusion can exceed oracle selection because it constructs a new answer from complementary candidates.
Selection preserves candidate limits; fusion can transcend them. Several imperfect answers may contain different correct pieces. A capable fuser can integrate those pieces into a response that none of the original candidates supplied. The shown benchmark is not clearly identified in the transcript, so this should be treated as a reported experiment rather than a universal guarantee.
Generated tests and layered operator graphs broaden the space of inference architectures.
Tests can become graph nodes. Generated checks add targeted evidence, although an LLM judging a test is weaker than deterministic execution. Layering multiple operators lets later stages consume critiques and filtered candidates, but evaluating each candidate architecture is itself costly.
Constrained Bayesian optimization finds deeper inference graphs more efficiently than greedy or random search.
Structure makes architecture search tractable. Constraints encode operator dependencies and eliminate nonsensical graphs. Bayesian optimization uses prior evaluations to choose informative configurations, making it more sample-efficient than blindly testing orchestration recipes.
An optimized open-model inference architecture reportedly exceeded contemporary proprietary systems on historical pass@1 benchmarks.
System performance is not base-model performance. The comparison demonstrates that many coordinated open-model calls can outperform one contemporary closed-model call on selected benchmarks. It is historical, benchmark-specific, and includes a substantial inference architecture, so it does not establish that any single open model is universally stronger.
Enjoy Reading This Article?
Here are some more articles you might like to read next: