Stanford CS329A Self-Improving AI Agents | Part 1 | Course Overview
The course frames self-improving AI agents through scaling trends before turning to practical course logistics.
Lecture roadmap. CS329A studies AI systems that can improve their behavior through stronger models, additional inference, feedback, and interaction with an environment. This lecture first builds the technical motivation and then explains how students will apply it in the course.
Pretraining scale enabled broad capabilities and practical zero-shot and few-shot learning, although pure size scaling began to face diminishing returns.
Pretraining scaling: increasing compute, dataset size, and parameter count historically reduced test loss and expanded model capability.
Usability consequence: zero-shot prompting specifies a task directly, while few-shot prompting adds worked examples. Large models made both modes practical across many tasks, reducing the need for task-specific training.
Chain-of-thought prompting exposed reasoning abilities that appeared abruptly at sufficient model scale.
Emergent reasoning: some capabilities do not improve smoothly with size; they become visibly useful only beyond a scale threshold.
Chain of thought: demonstrations that include intermediate steps can teach a model a procedure it can apply to new problems. The lecture presents this as an early sign that scaling changes not only loss but also the kinds of behavior a model can express.
ChatGPT’s leap over raw pretrained models came from a post-training stack that made model behavior higher quality, instruction-following, and aligned with human preferences.
A raw pretrained model mirrors patterns in its corpus but is not inherently optimized to understand a user’s goal, produce a sensible answer, or behave safely. The post-training pipeline adds:
- high-quality curated examples;
- instruction and response pairs covering diverse tasks;
- explicit solution processes where useful.
The breadth and quality of this data strongly shape the resulting behavior.
RLHF turns human comparisons into a learned reward signal that can optimize answers for correctness, usefulness, specificity, and safety.
Reinforcement learning from human feedback converts preference judgments into a scalable training signal:
1. Generate multiple candidate responses.
2. Have people compare or rate them.
3. Train a reward model from those judgments.
4. Optimize the language model toward higher-reward behavior.
Together with instruction tuning, this explains why a post-trained assistant can be substantially more useful than a similarly large raw pretrained model.
Inference scaling improves coverage by sampling many solutions from a fixed model and using a verifier to identify successful ones.
Core distinction: training-time scaling changes the model, while inference-time scaling spends more compute using the same parameters.
For a problem with success probability across diverse samples, pass@k or coverage can rise sharply with more attempts—even when pass@1 remains modest. Verification is essential because some hard problems may yield only a handful of correct answers among thousands of plausible generations.
Verified inference trajectories can become synthetic training data, closing a self-improvement loop despite substantial compute and verifier bottlenecks.
The self-improvement loop is:
1. Generate diverse candidate solutions at inference time.
2. Verify them with known answers, tests, tools, or learned evaluators.
3. Retain successful reasoning trajectories as synthetic data.
4. Fine-tune a stronger model on that data.
The limiting factors are verification quality and compute cost. Increasing temperature alone is not a substitute for this process because excessive randomness eventually produces incoherent output.
Test-time reasoning allocates extra computation to analysis, decomposition, feedback, correction, and backtracking without increasing parameter count.
Reasoning-time workflow: analyze the task, decompose it, attempt a solution, inspect feedback, correct errors, and backtrack when necessary. Calculators, code execution, tests, or evaluators can supply feedback. These learned procedures particularly improve mathematics, programming, and data analysis, though early reasoning models were not uniformly better for writing or text editing.
Learned reasoning improves first-attempt selection, whereas blind repeated sampling mainly increases the chance that one of many attempts is correct.
Pass@k versus pass@1: repeated sampling expands coverage, but a reasoning system must learn or apply a selection signal to make the first returned answer reliable.
Open questions include whether reasoning and final-response generation should use different models, how evaluator feedback should be organized, and how a system should decide that a difficult problem deserves more sequential or parallel attempts.
Agents extend conversational models into goal-directed loops that plan, act in an environment, use feedback, and decide when to stop.
An agentic loop couples goal interpretation → planning → tool action → observation → revision → termination. Tools provide access to external environments, while memory helps preserve task state. Current reliability often comes from deliberately engineered graphs—such as generator/evaluator or parallel research patterns—because open-ended action loops remain harder to control.
Reusable orchestration patterns combine LLM calls, routing, parallel work, tools, and verifiers into dependable workflows.
Common workflow building blocks include:
- prompt chains for staged transformations;
- routing for task-dependent paths;
- parallel fan-out and aggregation for independent research or proposals;
- orchestrator-worker structures for dynamic decomposition;
- evaluators and executable verifiers for feedback.
Domains with strong checks—especially code, mathematics, and rule-based tasks—support the most reliable self-correction.
Coding-agent gains are attributed mainly to stronger models and training, while reliable verification remains the central constraint on self-improvement.
Generator–verifier gap: producing candidates is often easier than judging them. Tests make code unusually favorable, while creative or subjective domains can bottleneck on human evaluation.
A capable end-to-end coding agent must also clarify intent, locate relevant files, choose operations, generate or run tests, and establish that the requested task is actually complete. Better models increasingly support this loop, but robust verifier construction is still an active research problem.
Agent workflows already support repetitive engineering, customer assistance, deep research, and scientific ideation.
Applications share a common pattern: decompose a broad objective, gather evidence or manipulate artifacts, verify intermediate results where possible, and synthesize a deliverable. Scientific brainstorming may benefit from generating many unconventional ideas, but novelty does not establish correctness; experiments and expert judgment remain necessary.
Chain of thought began as an observed emergent behavior, whereas modern reasoning models explicitly reinforce when and how to deliberate.
Historical progression: intermediate reasoning first appeared as a useful behavior discovered through prompting and scale; later systems deliberately train and reward it.
The practical target is not maximal reasoning on every query. A capable model should recognize when extended deliberation is valuable, use planning and correction mechanisms, and answer directly when the task is simple.
The course combines core and guest lectures with three homeworks and relies on Canvas for current materials, deadlines, and announcements.
Course structure: lectures and guest talks span post-training, agents, multimodality, robotics, and related frontier topics. Three homeworks provide guided depth, while Canvas carries the latest operational details. Students are expected to confirm prerequisites and plan around published deadlines.
The course project must pose a research question and evaluate a substantive agentic-system hypothesis rather than merely demonstrate an application.
A strong project states:
- a concrete hypothesis or research question;
- the system property or improvement being investigated;
- experiments and evaluation criteria;
- evidence explaining what worked and what did not.
Teams should use the papers and prior successful projects as starting points, then contribute original analysis rather than reproducing existing work.
Project milestones require early experiments and visible progress, and successful course work can develop into publishable research.
Milestone logic: the proposal commits to a question and experimental direction; the midterm demonstrates executed work and early evidence; the final report presents the complete research result; and the poster communicates it publicly. Starting experiments early is essential because each milestone is expected to advance beyond the previous one.
Assessment is split evenly between homework and the project, with a public poster session and fixed course policies supporting the final deliverables.
Completion and administration: the public poster session is the final project showcase. Grading is divided 50/50 between the three homeworks and the research project. Office-hour details and updates appear on Canvas; public questions help the whole class; Gradescope receives submissions. Students should plan around the published late-day policy and honor-code requirements.
Enjoy Reading This Article?
Here are some more articles you might like to read next: