Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL
Train-time scaling uses verified model-generated data to make smaller models substantially better at reasoning.
Why train-time scaling matters. The lecture reports roughly 5% AIME accuracy for the presumed 175B-parameter GPT-3.5, 51.7% for 7B DeepSeekMath, as much as 60% with additional techniques, and 50% for DAPO on Qwen-32B.
Interpretation: parameter count is not the only useful scaling axis; compute spent learning from filtered model outputs can materially improve reasoning.
Caveat: these are attributed, benchmark-specific figures from the lecture rather than controlled claims that apply across models or tasks.
Train-time scaling closes a self-improvement loop by training on outputs found and filtered during inference.
The central loop is sample → verify and filter → fine-tune → repeat.
- Pretraining supplies broad underlying capability.
- Test-time compute searches over multiple traces without changing parameters.
- Train-time compute makes successful behavior more likely on the next pass@1 attempt.
An o1 AIME plot is described as improving with both training and inference compute. The unresolved systems question is how to divide a finite compute budget between those two axes.
Reasoning traces allocate computation to analysis, decomposition, self-evaluation, correction, and alternative approaches.
Observed reasoning patterns: problem analysis, task decomposition, self-evaluation, backtracking, and parallel exploration of alternatives.
A batch-script example separates parsing, matrix construction, transposition, and output formatting; a chemistry example replaces an initially chosen pH formula after detecting an error.
The lecture reports greater-than-50% human preference win rates over GPT-4o for programming, data analysis, and mathematical calculation, but much smaller gains for writing and editing. Strong external checks appear central to this difference, and visible reasoning text is not itself proof of a reliable internal verifier.
Test-time and train-time scaling complement each other, but both require a capable base model and a trustworthy success signal.
Different roles: test-time compute searches over what the pretrained model can already express, whereas train-time compute increases the probability that successful behavior appears directly.
Both approaches depend on verification, and training additionally needs a nonzero success rate from which to hill-climb. The instructor cautions that a displayed train-versus-test compute curve is only one empirical instantiation, not a universal efficiency law. Fine-tuning harder tasks should not inherently erase easy skills, although malformed repetitive chains can produce overthinking.
STaR bootstraps explicit reasoning data from a small seed set and a larger collection of questions with known answers.
STaR implements a simple iterative recipe:
1. Few-shot prompt with a small rationale seed set.
2. Generate a rationale and answer for many training questions.
3. Keep directly correct attempts.
4. For an incorrect attempt, reveal the answer and request a backward rationalization.
5. Train on the resulting question-rationale-answer records without exposing the hint, then repeat.
This is best understood as an off-policy-RL-like bootstrap, not a full reinforcement-learning algorithm.
STaR assumes that correct answers imply useful rationales and that the base model can already solve part of the target distribution.
Three critical assumptions: the final answer is a reasonable rationale-quality proxy; answer-conditioned rationalization is valid; and the initial model solves a nonzero subset of the task.
If the entire task lies outside the base model’s capability, the loop has no hill-climbing signal. Original STaR does not inspect hinted intermediate steps, so humans or a process reward model would be needed to reject plausible but invalid rationales. The discussion also distinguishes the comparatively mature use of positive examples from the still-open problem of learning reliably from failures.
Vanilla STaR alternates direct generation, answer-conditioned rationalization, and fine-tuning over multiple outer iterations.
Concrete training flow: directly generated correct rationales and answer-hinted rationales for failures form a single fine-tuning set, and the refreshed model repeats the cycle.
The reported implementation used roughly 6B-parameter GPT-J, a short warm-up, constant learning rate, several outer iterations, and progressively more inner-loop work. Evaluation covered approximately 9,000 GSM8K examples, CommonsenseQA, and synthetic multi-digit arithmetic.
STaR can improve data efficiency, but gains plateau and rationale quality remains difficult to measure.
Observed benefits: filtering can produce a smaller, higher-quality training set, and human raters found the CommonsenseQA rationales broadly reasonable. One spoken comparison also cites about 72.5% accuracy using roughly 86% of the data, although its exact baseline is unclear.
Limits: repeated STaR iterations plateau; direct fine-tuning can match rationalization on tasks already within capability; prompt style shapes subsequent rationales; and final-answer filtering can preserve invalid intermediate steps. Rationale validation therefore still needs human review or a process-level evaluator.
STaR variants change the verifier or reasoning representation, but self-improvement remains bounded by the base model.
A stronger teacher can always generate traces for a smaller student, but that is distillation, whereas STaR asks a model to learn from its own outputs. V-STaR adds a verifier to the generator loop; Quiet-STaR is described as moving reasoning toward an internal representation rather than requiring every step in English.
The ceiling remains the starting model: it must be able to rationalize hinted answers, some answers are easier to reverse-engineer than others, and enough solvable examples are required to generalize. These bounds should remain explicit when presenting STaR as self-improvement.
DeepSeekMath first strengthens its mathematical base through code initialization and carefully curated web data.
Capability comes before RL. DeepSeekMath begins with a code-trained model, curates high-quality mathematical web pages across many domains, performs supervised training, and only then scales reinforcement learning.
The lecture reports that targeted Common Crawl and OpenWebMath curation yielded broader coverage and more useful tokens than indiscriminate arXiv training, while code initialization transferred reasoning and tool-use skills. This is a paper-specific empirical finding, but it reinforces the broader lesson that RL cannot manufacture missing domain knowledge from no signal.
GRPO reduces PPO memory requirements by replacing the learned critic with standardized rewards from a sampled group.
For rewards r_i in group G, the lecture gives the intuition A_i ≈ (r_i - mean(r_G)) / std(r_G).
This group-relative comparison removes the learned value model and reduces memory pressure. DeepSeekMath is reported to improve from 46.8% to 51.7% on MATH, becoming an early open 7B model above 50% without a critic.
The full GRPO objective is intentionally omitted in the lecture; the standardized-reward expression is an intuition, not sufficient implementation detail.
GRPO requires reward variance and was shown to improve output consistency rather than pass@K coverage.
Learning signal: groups need reward variance; uniformly impossible or trivial questions contribute little or no group-relative gradient. A well-chosen difficulty distribution supplies hill-climbing examples, while KL regularization can help preserve capabilities of the reference policy.
The reported gain is in majority@32, not pass@32: successful behavior becomes more frequent and consistent, but the set of problems solved at least once does not expand. This distinction prevents consistency gains from being mislabeled as new fundamental capability.
DAPO stabilizes large-scale reasoning RL by preventing exploration from collapsing under symmetric PPO clipping.
The lecture contrasts a roughly 30% naive-GRPO AIME result on Qwen-32B with a cited DeepSeek result near 47%, then attributes part of the gap to implementation details omitted from basic GRPO.
Clip-Higher: asymmetric probability clipping permits more growth for low-probability actions, preserving entropy and exploration while improving accuracy. Entropy is used here as a diagnostic proxy for how many alternative reasoning trajectories remain accessible.
DAPO dynamic sampling retains only question groups whose mixed outcomes provide a usable gradient.
Dynamic sampling uses the GRPO signal more efficiently:
1. Oversample answers for each question.
2. Remove all-correct and all-wrong groups.
3. Keep groups with 0 < correct_count < group_size.
4. Form the update from those informative groups.
The stated group size of 64 is illustrative. How much to oversample depends on benchmark difficulty and base-model competence, so it is not a universal hyperparameter.
Token-level loss and soft overlength penalties keep long reasoning trajectories from destabilizing DAPO training.
Length-aware stabilization: token-level loss avoids treating every full response identically, and a soft overlength penalty reduces the influence of traces that grow until truncation. Increasing context length is an alternative explored by other work.
The lecture reports a cumulative Qwen-32B AIME ablation: GRPO 30, overlong filtering 36, asymmetric clipping 38, soft overlong punishment 41, token-level loss 42, and dynamic sampling 50. These are pipeline-specific reported scores, not isolated causal estimates for every component or benchmark.
STaR, GRPO, and DAPO occupy different points on the tradeoff between data availability, infrastructure complexity, and task difficulty.
Method selection:
- Use STaR as a lightweight bootstrap when only a small rationale set and known answers are available.
- Use GRPO when the base model is already capable and memory-efficient online RL is needed.
- Add DAPO-style clipping, sampling, and length controls when long reasoning traces make training unstable.
Supervised fine-tuning is often faster with abundant high-quality demonstrations; RL can hill-climb from fewer examples when rewards are strong, but it is harder to implement. All three may improve majority@K, formatting, and coherence without necessarily improving pass@K or out-of-domain capability.
Current train-time scaling is limited by base capability, reward quality, and weak methods for learning from failures.
The lecture identifies several open problems:
- Why does majority@K improve while pass@K often does not?
- Are backtracking and self-correction genuinely learned behaviors or simply more frequent existing patterns?
- How can training extract useful information from failures instead of discarding them?
- How can RL remain robust to noisy or hackable rewards?
- Can verifier ensembles cover a single verifier’s blind spots?
- Can STaR rationalization be combined with DAPO stabilization?
Math and code are unusually favorable because answers, execution, and unit tests provide external feedback; many domains lack comparable signals.
Reinforcement learning improves exploration within learned knowledge, while data sufficiency and verifier coverage remain separate challenges.
The closing Q&A separates three constraints: knowledge in the base model, enough informative examples to hill-climb, and verifiers that measure the desired behavior. Multiple verifiers may improve evaluation coverage, but they do not create missing training knowledge.
The instructor gives informal, explicitly uncertain estimates that frontier training has moved from roughly 1% RL toward perhaps 5%, and discusses a Grok-4 claim of 50% RL without a proportional gain. These vendor proportions are unpublished or informal and should not be presented as audited facts.
Enjoy Reading This Article?
Here are some more articles you might like to read next: