AgentDevel: Reframing Self-Evolving LLM Agents as Release Engineering

Source: arXiv:2601.04620, version 1, accessed 2026-09-01.

Overview / Takeaway

AgentDevel reframes agent self-improvement as release engineering: maintain one canonical agent blueprint, generate one release candidate (RC) at a time, and promote it only after explicitly examining what it fixes and what it breaks. Its central contribution is not a new language model but a development loop that separates implementation-blind symptom reporting, executable diagnosis, RC synthesis, and flip-centered gating. The reported final agents improve substantially over their starting blueprints on four benchmarks, while the WebArena ablation shows that removing the gate can slightly increase final accuracy but causes many more regressions and four “bad releases.” The paper therefore makes a credible case for regression-aware evolution, but not a formal non-regression guarantee: accepted updates still contain pass-to-fail flips, gate thresholds and release budgets are unspecified, and several reporting discrepancies limit reproducibility.

1 Introduction

1. Self-evolving agents should be treated as maintained software, not unconstrained search objects. An agent is an executable artifact containing prompts, orchestration code, and tool wrappers. AgentDevel’s engineering question is therefore whether a proposed version is safe enough to release, not merely whether one candidate has the highest aggregate reward.

2. Aggregate scores hide the most consequential behavior changes. Two versions can have the same score even when one fixes many failures and introduces equally many regressions. The paper makes per-example transitions—especially previously passing examples that become failures—the central evidence for promotion.

3. The framework deliberately uses a single canonical release line. Each iteration begins from the currently promoted blueprint, creates exactly one RC, and either promotes or discards it. This contrasts with population/archive approaches that retain many variants, and makes version history, rollback, and attribution easier at the cost of reduced search diversity.

4. Diagnosis is separated from repair. An implementation-blind critic receives observable execution evidence but not the blueprint. It reports surface symptoms rather than hypothesizing causes or proposing changes. A separate stage converts those records into executable analysis and then into a blueprint diff.

5. The paper’s three claimed contributions form one release pipeline. They are: implementation-blind, open-taxonomy feedback; LLM-generated executable diagnostic scripts; and flip-centered gating with explicit stopping signals. Their value is compositional: blindness aims to reduce evaluator–implementation entanglement, scripts aggregate repeated symptoms, and the gate converts the resulting intent into a release decision.

6. Evaluation separates development data from a held-out final test. Iterative diagnosis, synthesis, gating, and stopping use only the TrainSet. The TestSet is run once after development. This avoids direct test-set selection, although repeated reuse of the same TrainSet still permits adaptive overfitting.

7. The release-engineering framing changes the optimization objective. AgentDevel does not simply maximize fixes. It weighs fixes against regressions and asks whether changes match their declared intent. The StableToolBench trace later shows an RC with 42 fixes being rejected because it also causes 28 regressions.

8. “Non-regression” is a priority, not a demonstrated guarantee. The abstract’s guarantee-oriented language should be read cautiously. The method defines a configurable gate, the full WebArena system records 18 pass-to-fail transitions, and no theorem or universal threshold ensures zero regression.

2 Method

The figure exposes the paper’s core control flow. Stage A records implementation-blind surface symptoms; Stage B generates and executes diagnostic code before proposing one RC; Stage C classifies each training example as stable pass, regression, fix, or persistent failure and uses those transitions to accept or discard the RC.

2.1 Setup & Overview (What We Build)

1. The mutable object is the complete agent blueprint. Let \(b\) denote prompts, agent code, and tool-invocation internals. Executing blueprint \(b\) on input \(x\) produces an answer and trace:

The trace can include actions, tool calls, observations, errors, and the final response. Because the mutation boundary includes code and wrappers, AgentDevel is broader than prompt optimization.

2. Development and final evaluation are disjoint by construction. The data are split as

Every iterative decision is made on \(\mathcal D_{\mathrm{train}}\); \(\mathcal D_{\mathrm{test}}\) is reserved for one final evaluation. The paper does not disclose \(N\), \(M\), the split construction, or per-benchmark iteration budgets.

3. A deterministic scorer is optional. When a benchmark exposes a rubric or executable checker \(R\), the system can compute

The scorer supplies hard evidence where available, while a critic handles cases requiring semantic interpretation.

4. The version graph is intentionally linear. AgentDevel maintains one promoted \(b_t\), one candidate \(b_t^{\mathrm{RC}}\), and a decision to promote or discard. This supports auditability and attribution but foregoes crossover, parallel exploration, and recovery of rejected innovations from an archive.

2.2 Running the Agent & Producing Quality Signals (What We Observe)

1. The critic is implementation-blind by interface. For each training example it sees the rubric \(R\), execution trace \(\tau_t(x)\), and optionally scorer output \(g_t(x)\), but not \(b_t\). It returns a critic pass label \(\tilde p_t(x)\), a symptom label \(\ell_t(x)\), and a symptom description \(d_t(x)\). It must describe observable failure rather than assign an internal cause or propose a repair.

2. Hard scoring overrides the critic’s pass decision. The final pass indicator is

This hierarchy limits judge subjectivity where executable verification exists. The paper assumes the critic is deterministic or run with a fixed seed, but does not report agreement, calibration, or perturbation tests.

3. The symptom taxonomy remains open and monotonically expands. Rather than forcing errors into fixed classes, the critic may introduce new labels, with \(\mathcal L_t\subseteq\mathcal L_{t+1}\). This can capture unforeseen failures, but the paper offers no merging, deprecation, or consistency mechanism for label proliferation.

4. Per-example records are the diagnosis stage’s only inputs. Each record combines output, trace, optional score, critic decision, final pass status, symptom label, and description. The set \(\mathcal R_t=\{r_t(x):x\in\mathcal D_{\mathrm{train}}\}\) creates a provenance boundary: repair does not begin until observable evidence has been recorded.

5. Blindness is informational, not statistical independence. Traces and tool errors can reveal aspects of implementation behavior. The design prevents direct inspection of prompts and code; it does not make the critic independent of the agent or eliminate all pathways for evaluator overfitting.

2.3 Executable Diagnosis & RC Synthesis (How We Propose Changes)

1. An LLM writes analysis code rather than only free-form commentary. AgentDevel generates a Python diagnostic script, executes it over \(\mathcal R_t\), and uses its concrete aggregation to produce \(D_t\). The script can count symptom labels, detect trace patterns, retrieve representative examples, and estimate prevalence.

2. Diagnostic programs are iterative artifacts. The previous script is supplied as a soft reference when generating the next one. Saving and version-controlling these programs makes analytical changes auditable, although the paper does not specify a script sandbox, permissions model, or tests against unsafe generated code.

3. Execution grounds the diagnosis but does not remove model dependence. For fixed code and records, aggregation is reproducible. However, both the diagnostic program and its narrative summary are LLM-generated; the paper’s contrast with “model output” is therefore best interpreted as model-generated code plus deterministic execution, not a model-free diagnosis.

4. The diagnosis becomes an engineering specification. \(D_t\) identifies dominant symptoms, triggers, affected surfaces, and representative cases. This creates a narrower interface between symptom discovery and code modification than an end-to-end “reflect and rewrite” prompt.

5. Exactly one RC is synthesized with declared intent. The proposal operator is

The RC may alter prompts, code, or tool wrappers and is accompanied by intent \(I_t\), identifying the symptoms it is meant to fix. One-at-a-time synthesis improves attribution but can become a local search bottleneck.

2.4 Flip-centered Gating, Promotion, and Stopping (How We Decide & When We Stop)

1. Candidate evaluation reuses the full TrainSet. Both \(b_t\) and \(b_t^{\mathrm{RC}}\) are evaluated on the same examples. This paired design enables exact behavioral differencing but makes development increasingly adaptive to one fixed set.

2. The two decisive transition sets are regressions and fixes. The formal expression makes the mechanism and its constraints explicit.

Stable pass and persistent failure are also visible, but \(\mathcal P2F_t\) is direct regression evidence and \(\mathcal F2P_t\) is direct repair evidence.

3. Normalized rates account for different opportunity sets. The formal expression makes the mechanism and its constraints explicit.

Raw counts and rates should be read together: a small P2F count can be substantial when few examples currently pass.

4. The gate is abstract and configurable. The formal expression makes the mechanism and its constraints explicit.

Its principles are to treat P2F as high risk, use F2P as evidence of benefit, and check whether fixes match \(I_t\). The paper does not publish universal thresholds or a concrete gate implementation sufficient for independent reproduction.

5. Promotion is atomic. If accepted, \(b_{t+1}\leftarrow b_t^{\mathrm{RC}}\); otherwise \(b_{t+1}\leftarrow b_t\). Rejected candidates do not enter an archive. This is closer to a protected main branch than evolutionary population search.

6. Stopping is based on diminishing safe progress. Suggested signals include consistently small F2P sets, rising P2F rates, repeated gate rejections, or gains concentrated on a shrinking subset. These are heuristics rather than a fixed stopping rule, and the held-out TestSet is explicitly excluded.

7. A rejected RC is informative but the retry policy is underspecified. The current blueprint remains unchanged, yet the algorithm does not state how the next diagnosis or RC is forced to differ. A deterministic engine with the same inputs could repeat a failed proposal unless prompts, seeds, or rejection evidence change.

3 Result

3.1 Experiment Settings

1. The implementation uses Claude Code with Claude Sonnet 4.5 throughout. Claude Code generates and executes diagnostic scripts, summarizes symptoms, and proposes blueprint diffs. The “Devel engine” and tooling are held fixed across iterations, reducing within-run model confounding but leaving framework-versus-model effects untested across model families.

2. The audit trail is a first-class artifact. The implementation saves and version-controls diagnostic scripts, RC diffs, critic outputs, and flip lists. This is the clearest operational consequence of the release-engineering framing.

3. Four domains test different agent surfaces. SWE-bench Lite and Verified test repository repair, WebArena tests browser interaction, and StableToolBench tests tool use. Development uses only TrainSet records; the TestSet is evaluated once at the final promoted version.

4. Important experimental details are absent. The paper does not report split sizes, exact blueprints, candidate budgets, gate thresholds, seeds, repeated-run variance, token/API cost, or wall-clock cost. The prior-work values marked with a dagger were reported by their original papers rather than rerun under the same setup and budget.

3.2 Main Results

† Prior-work number reported by its source paper, not rerun under AgentDevel’s exact model, split, tools, or budget.

1. Every final blueprint materially improves over its initial version. Absolute gains range from 11.0 to 19.5 percentage points. Relative to the initial score, improvements are 100% on both SWE-bench variants, approximately 108.8% on WebArena, and approximately 36.1% on StableToolBench.

2. The comparison to prior work is mixed rather than uniformly state of the art. AgentDevel exceeds the listed SWE-agent Lite result by 4.0 points and DFS on StableToolBench by 3.3 points, but trails the listed GPT-4o Verified result by 3.2 points and CER_hybrid on WebArena by 1.2 points. Because the daggered systems were not rerun, these gaps are contextual, not controlled head-to-head evidence.

3. The table demonstrates final capability, not release safety by itself. No per-release flips, uncertainty intervals, or final regression composition accompany three of the four headline rows. The next two sections provide the stronger evidence for the release-engineering thesis.

4. The WebArena headline conflicts with the ablation table. Main results report 35.50% for AgentDevel, whereas the “Full” WebArena ablation reports 34.2%. The ablation controls only state that its rows share the same blueprint, split, and budget; the paper does not explain whether the headline came from another run or configuration.

3.3 Case Study

The StableToolBench trace covers 11 RC decisions. “Hit rate” and the columns “FTP / P2P” are reported without formal definitions; P2P appears to measure pass preservation and FTP appears related to accumulated fail-to-pass progress, but those interpretations should not be treated as author-defined.

1. The gate rejects attractive but destructive updates. Iteration 3 has the largest F2P count in the table—42 fixes—but is rejected because it also creates 28 regressions, a 4.0% P2F rate, and only 0.41 hit rate. Aggregate net gain alone would obscure that release risk.

2. The decision is genuinely multi-criterion. Iteration 11’s 0.4% P2F rate is no higher than several accepted iterations, yet the RC is rejected with only two fixes and 0.67 hit rate. This demonstrates that no single regression threshold explains every decision.

3. Accepted releases accumulate far fewer regressions than rejected proposals. Across the eight accepted rows, the displayed transitions sum to 142 F2P and 25 P2F. Across only three rejected rows they sum to 53 F2P and 46 P2F. These are transition counts, not unique examples or final pass gains; an example may flip at multiple iterations.

4. Safe improvement becomes harder late in development. Accepted F2P counts fall from 38 to 5 while reported hit rate rises from 0.74 to 0.92. The trajectory is consistent with harvesting large obvious fixes early and demanding more precise interventions later.

5. The trace supports risk reduction, not zero regression. Every accepted RC in the table still has two to five P2F transitions. AgentDevel’s gate limits regression exposure and rejects particularly harmful packages; it does not prove a literal non-regression property.

3.4 Ablation Study

The WebArena ablation uses the same initial blueprint, split, and development budget across variants.

1. Removing the gate exposes the central performance–safety tradeoff. The ungated variant is 0.8 points better on the test metric and 2.5 points better on train pass, but creates 95 rather than 18 P2F transitions: 5.28 times the count and 4.77 times the rate. It also promotes four bad releases. The gate buys release stability, not the best raw score in this ablation.

2. Executable diagnosis materially improves repair yield. Removing it reduces test performance by 2.4 points, train pass by 4.5 points, and total fixes by 64, while the gate rejects 63% rather than 42% of candidates. Programmatic aggregation appears to produce more promotable RCs than unstructured diagnosis.

3. Giving the critic implementation access improves training fit but hurts held-out behavior. A non-blind critic reaches 83.5 train pass, five points above the full system, yet test performance drops 1.7 points. It also more than doubles the P2F rate from 3.1% to 6.7%. This is evidence—though only one benchmark/run—that separating surface evaluation from repair can reduce evaluator–implementation entanglement.

4. “Zero bad releases” depends on an undisclosed threshold. A bad release is defined as a promoted update whose regressions exceed a preset threshold, but that threshold is not reported. Zero bad releases therefore cannot be independently reproduced and does not mean zero pass-to-fail transitions.

5. The flip-accounting description is ambiguous. The paper says F2P/P2F are computed by comparing each promoted release to its evaluated RC, even though the formal method compares the current promoted version with the candidate before promotion. The intended paired comparison is clear conceptually, but the exact aggregation protocol should have been specified.

6. Counts lack denominators and uncertainty. The table does not report the number of RCs, number of examples, repeated runs, confidence intervals, or how examples that flip multiple times contribute. The rejection percentages therefore cannot be converted into proposal counts.

4 Conclusion

1. AgentDevel’s strongest idea is release discipline, not a new search algorithm. It turns traces into observable symptom records, records into executable diagnoses, diagnoses into one scoped RC, and paired outcomes into a promotion decision. This produces a coherent audit trail around agent self-modification.

2. The empirical results support all three stages, with qualifications. Final scores improve on four benchmarks; diagnosis increases fix yield; critic blindness improves held-out behavior relative to the non-blind ablation; and gating sharply reduces regressions and bad releases. Yet the ungated system achieves the best WebArena test score, so regression control and benchmark optimization are distinct objectives.

3. The framework is best understood as a policy template. The gate, stop rule, critic contract, and safety envelope require concrete project-specific definitions. Without published thresholds and operational controls, the paper does not yet provide a drop-in reproducible release system.

4. Its conceptual contribution is broadly reusable. Per-example behavioral diffs, declared change intent, atomic promotion, and held-out final evaluation apply beyond agents to prompts, tool policies, workflows, and code-generating systems.

Appendix A Limitations

1. Development incurs substantial overhead. Repeated full-TrainSet executions, critic calls, generated diagnostic scripts, RC synthesis, and paired re-evaluation consume more compute and wall-clock time than a single-pass agent. No token, API, or latency accounting is provided.

2. Quality signals can be biased or inconsistent. Programmatic checkers mitigate this where available, but semantic tasks still depend on an LLM critic. Fixed seeds do not establish validity, calibration, or immunity to systematic bias.

3. Scope is limited to a single agent and repository or environment. Multi-agent coordination, very large codebases, nonstationary environments, and continuously changing tools remain untested.

4. Gate and stopping rules are not universal. Acceptable regression risk varies by domain. A medical or financial agent may require a much stricter policy than a benchmark navigator; AgentDevel leaves this customization to implementers.

5. Repeated TrainSet reuse risks adaptive overfitting. Reserving the TestSet prevents direct leakage into decisions, but there is no development-validation split, reusable holdout, or correction for many adaptive RC evaluations.

6. Generated executable diagnostics create an unaddressed security surface. The paper does not describe sandboxing, network or filesystem restrictions, secret handling, dependency policies, timeouts, or human approval for scripts and blueprint diffs. These controls are essential if development touches real repositories or external tools.

7. A linear release line trades breadth for auditability. One RC per iteration makes causal review easier but can miss combinations of changes, get trapped in local fixes, and discard useful partial innovations. There is no branch, archive, or cherry-pick mechanism.

8. The experimental evidence is not statistically characterized. There are no multi-seed results, confidence intervals, evaluator-reliability measurements, or controlled reruns of prior baselines. Consequently, headline improvements and ablation gaps may include run-specific variation.

9. Some reported quantities are underdefined or inconsistent. Gate thresholds, hit rate, FTP/P2P, bad-release threshold, and flip aggregation are not fully defined. The 35.50 versus 34.2 WebArena discrepancy is not reconciled.

10. Human release review and rollback are discussed more as metaphors than evaluated mechanisms. The system versions artifacts and can discard RCs, but the paper does not test human approvals, production canaries, incident response, rollback latency, or post-deployment monitoring.

Appendix B Related works

1. Reflection-based agents are an explicit conceptual predecessor. Reflexion and Self-Refine improve behavior through textual feedback and repeated revision. AgentDevel contrasts their cognition-centered loops with artifact-centered versioning, executable diagnosis, and promotion gates.

2. Search-based optimization supplies a contrasting family. PromptBreeder and Tree of Thoughts explore multiple candidates. AgentDevel instead commits to one canonical line and evaluates one RC at a time, prioritizing auditability over population diversity.

3. Self-modifying systems motivate the release boundary. Darwin Gödel Machine and Live-SWE-Agent are named as population/archive-oriented or self-modifying systems. AgentDevel’s methodological response is to make every self-change an explicit candidate release with regression evidence.

4. LLM-as-judge work supplies part of the evaluation machinery. MT-Bench, Chatbot Arena, G-Eval, and AlpacaEval show how model-based evaluators can judge open-ended outputs. AgentDevel narrows their role with an implementation-blind contract and defers to hard scorers where possible.

5. Continuous integration and software release engineering are the deepest non-agent lineage. Regression tests, release candidates, protected promotion, audit trails, and stopping on unstable changes structure the entire framework. The paper adapts these practices to mutable LLM-agent blueprints.

6. SWE-agent is both a scaffold precedent and an experimental comparator. AgentDevel treats scaffolds like SWE-agent as maintainable artifacts and reports a 22% SWE-bench Lite resolved rate versus the cited 18% SWE-agent result, while caution is required because configurations were not matched.

Appendix C Pseudo-Code of AgentDevel

1. Initialization creates state for both the agent and its diagnostic process. Start with current blueprint \(b\), previous diagnostic script \(\pi=\varnothing\), and open taxonomy \(\mathcal L=\varnothing\).

2. Each loop observes before it modifies. Run \(A_b\) across the TrainSet, score outputs where possible, apply the blind critic, update \(\mathcal L\), and assemble records \(\mathcal R\). This ordering prevents the repair stage from substituting speculative causes for recorded symptoms.

3. The loop then generates evidence-producing code. Generate a diagnostic script using \(\mathcal R\) and soft reference \(\pi\), execute it, save the script as the next \(\pi\), and produce diagnosis \(D\).

4. One intended change package is proposed and tested. Synthesize \((b^{\mathrm{RC}},I)\) from \((b,D)\), evaluate the RC on the same TrainSet, and compute F2P/P2F transitions against the current blueprint.

5. The gate protects the canonical state. Promote \(b^{\mathrm{RC}}\) only if \(G\) accepts it; otherwise retain \(b\). The loop stops when the configured stop predicate detects insufficient or unsafe progress, after which the final \(b\) is evaluated once on the TestSet.

6. The pseudocode reveals two implementation questions. Its compact critic call omits the optional programmatic score described in the method, and it does not specify how rejection information changes the next proposal. A production implementation should make both data flows explicit.

Open Questions

1. Can release safety be specified as a measurable guarantee? What confidence bounds, severity weights, or risk budgets would convert a heuristic P2F gate into a defensible domain-specific guarantee?
2. How should adaptive evaluation be controlled? A three-way train/validation/test design, reusable holdout, or sequential-testing correction could reduce overfitting across many RCs.
3. What is the right exploration–auditability frontier? Can small branches or a bounded archive preserve useful rejected changes without losing the canonical release discipline?
4. How robust is critic blindness? Tests should measure whether traces leak implementation details and whether blindness helps across models, tasks, and adversarial failures.
5. How should generated diagnostic code be secured? Sandboxing, dependency allowlists, resource quotas, secret isolation, provenance checks, and human approval deserve direct evaluation.
6. How much does each stage cost? Report tokens, model calls, environment executions, wall-clock time, and cost per accepted improvement—not only final accuracy.
7. Does regression-aware development improve real deployment reliability? Production studies should include canaries, rollbacks, delayed failures, side effects, and distribution shift.
8. How should open symptom taxonomies be maintained? Automatic merging, hierarchical labels, stability metrics, and human reconciliation may prevent taxonomy drift.
9. Would independent engines improve trust? Using different models for criticism, diagnosis, synthesis, and gating could reduce correlated error, but may increase inconsistency and cost.
10. Can the unexplained metrics and result discrepancy be reconciled? Future reporting should define hit rate, FTP/P2P, gate thresholds, proposal counts, and explain the two WebArena final scores.