Stanford CS329A Self-Improving AI Agents | Part 4 | Learning from Feedback with Tools/Code
Self-improving agents differ mainly in where their feedback signal comes from.
Three feedback loops. ReAct grounds behavior in tool and environment observations. RLEF turns test execution into both repair context and a training reward. Constitutional AI asks models to critique behavior against explicit principles. The shared mechanism is repeated action → feedback → improved next behavior.
ReAct connects language-model reasoning with grounded action in an external environment.
Reasoning needs contact with the world. A model’s parametric memory cannot reliably answer every current or obscure question. ReAct bridges verbal planning and environment interaction so each observation can alter the next thought or action rather than leaving reasoning isolated from evidence.
A prompted thought–action–observation loop forms a simple, interpretable language agent.
Minimal ReAct loop. Thought chooses what information is missing; action affects or queries the environment; observation supplies grounded context; the next thought adapts. Thoughts remain text, whereas actions have external effects. Restricting the valid action vocabulary can prevent malformed or unsupported tool calls.
Few-shot ReAct demonstrations teach a frozen model when to reason, search, reformulate, and finish.
Demonstrations define an agent protocol. The model learns not only which tool exists but when to alternate deliberation and action. Modern thinking models may produce similar traces because they were trained or distilled on tool-use patterns, but the foundational result showed that a frozen model could be orchestrated through context alone.
ReAct requires deliberate calibration, evidence handling, and control over sequential reasoning cost.
Operational caveats. Treat confidence claims as weak signals. Add retrieval policies, repeated evidence checks, or guardrails for conflicts. The basic loop waits for each observation before choosing the next action; parallel plans and adaptive deliberation budgets require an outer orchestration layer.
ReAct reduces hallucination and outperforms action-only baselines, but hybrid reasoning and fine-tuning still matter.
Evidence and limits. Retrieved observations reduce unsupported answers, while parametric reasoning remains useful when tools fail or knowledge is already present. Fine-tuned ReAct improves further. Large action spaces consume demonstration context, multi-step errors cascade, and repeated calls add latency, so the method is a foundation rather than a complete production agent.
Robust agents extend ReAct with reflection, backtracking, memory, decomposition, parallelism, and adaptive budgets.
No substantive slide was displayed during this discussion segment, so the note is presented without a representative frame.
From loop to cognitive architecture. Real systems need mechanisms for escaping failed trajectories, retaining useful experience, splitting tasks, and exploring alternatives concurrently. More reasoning is not automatically better; a controller should spend deliberation only when uncertainty and task value justify it.
RLEF turns executable test feedback into an iterative coding loop and a reinforcement-learning signal.
Two nested loops. At inference, the current policy writes code, runs tests, reads failures, and repairs. Around that interaction, RL updates the policy so later attempts become better. This converts one episode’s execution evidence into both immediate correction and longer-term self-improvement.
Public repair tests and hidden reward tests separate actionable feedback from final generalization measurement.
Fast feedback without direct target leakage. Public tests expose enough behavior to repair a candidate; private tests judge whether the repair generalizes. Execution, not a model guess, determines failure. Although generation is token-level, the discussed objective applies one turn-level outcome signal to the whole response.
RLEF training improves solve rate and teaches targeted repairs that transfer beyond its training benchmark.
Execution traces become reusable training signal. Merely showing faulty programs is insufficient; the model benefits from repeatedly responding to concrete failures. Sampling adds diversity, while the learned policy improves how later attempts use feedback. Reported transfer suggests more than memorizing one benchmark, but the small-program setting limits the claim.
Binary final rewards can train simple repair loops, while complex software needs better credit assignment and evaluation design.
Reward design scales with task complexity. A pass/fail bit works when episodes are short and tests are informative. Long-horizon coding needs intermediate evidence, resource penalties, and diagnostics that distinguish syntax, logic, performance, and integration failures.
Repository-scale coding agents must retrieve and represent relevant context before test feedback becomes useful.
Large-code loop. Determine what information is missing, retrieve only relevant files and relationships, form a patch from that compact representation, and use tests to validate it. Execution feedback cannot compensate for a wrong or incomplete view of the repository, so context acquisition is part of the agent policy.
Constitutional AI moves much of scalable preference collection from humans to model critique guided by a small set of principles.
Humans specify values; models apply them at scale. The constitution encodes desired constraints, while instruction-following models generate critiques, safer revisions, and preference pairs. Human work shifts from labeling every response to authoring principles and validating whether automated judgments remain aligned.
Constitutional training uses critique-and-revision SFT followed by reinforcement learning from AI feedback.
Two-stage post-training. First teach the model how to detect and rewrite principle violations. Then scale preferences with an AI judge and optimize the policy against that learned reward. Repeated revision improves harmlessness but can reduce helpfulness, so the objective must balance both.
Constitutions are relatively cheap to update, but forgetting old rules and validating AI feedback remain unresolved.
Editable behavior is not erased knowledge. New constitutional post-training can shift output probability without proving that an old capability or rule disappeared. The reported best Pareto frontier combines constitutional RL with reasoning, but self-critics remain overconfident; multiple critics, consensus, or external validation can improve reliability.
ReAct, RLEF, and Constitutional AI show that self-improvement depends on an informative and usable feedback channel.
Unified lesson. An agent improves only when feedback distinguishes better from worse next actions. The signal may be retrieved evidence, a deterministic test, or a normative critique, but it must enter a loop that changes the next trajectory or the policy itself.
Human-inspired workflows are useful agent building blocks, but general self-improvement requires searchable strategies and domain-specific safeguards.
No substantive slide was displayed during this closing discussion.
Toward general agents. Decomposition, memory, reflection, and parallel analysis are practical scaffolds while open-ended strategy search remains difficult. Definable environments make RL tractable; real professional domains need tailored tools, permissions, verification, and risk controls. Filtering and architectural constraints can reduce undesirable behavior, but cannot guarantee removal of everything learned during internet-scale pretraining.
Enjoy Reading This Article?
Here are some more articles you might like to read next: