Symbolic Learning Enables Self-Evolving Agents

Source: https://arxiv.org/abs/2406.18532

Overview / Takeaway

Agent symbolic learning treats a language agent as a trainable symbolic network whose prompts, tools, nodes, and inter-node connections play roles analogous to neural-network weights, layers, and computation graphs. A forward execution produces a trajectory, an LLM turns that trajectory into a textual language loss, backward prompting assigns node-specific language gradients, and three symbolic optimizers rewrite the agent. Across HotPotQA, MATH, HumanEval, creative writing, and five software-development tasks, this holistic update procedure generally improves on fixed agents, isolated prompt optimization, and search-based optimization, with the largest reported gains on GPT-3.5 MATH and the open-ended tasks. The evidence is promising but proof-of-concept: the paper reports no run-to-run variance, statistical tests, token or dollar costs, broad safety evaluation, or controlled ablation of the loss, gradient, and optimizer components.

1 Introduction

1. Language agents join neural and symbolic computation An agent combines a connectionist LLM backbone with a symbolic pipeline of prompts and tools. Complex tasks are decomposed into nodes, each of which may carry prompts, API functions, knowledge bases, memories, and other tools; the pipeline specifies how these components interact over multiple steps.

2. Manual agent construction creates an engineering bottleneck Developers must choose the task decomposition, write node prompts, select or implement tools, and connect the nodes by hand. This engineering-centric process makes it difficult to optimize an entire agent against data, weakens robustness under new tasks or distributions, and repeats customization work that numerical neural-network training automates.

3. Earlier automation optimizes fragments rather than the complete agent DSPy and GPTSwarm search over prompt components or LLM pipelines, while Agent-pro and AgentOptimizer use LLMs to improve isolated policies, prompts, or functions. The central diagnosis is that independently optimizing a component can produce a local optimum that does not improve the end-to-end system; search methods also require a codable numeric objective and cannot naturally create new tools or nodes.

4. The proposed analogy makes the whole agent trainable The agent pipeline corresponds to a neural computation graph, a node corresponds to a layer, and prompts plus tools correspond to learnable weights. Agent execution becomes a forward pass; an execution record becomes a trajectory; textual evaluation becomes a language loss; backward textual critique becomes a language gradient; and prompt-driven rewriting becomes gradient-style optimization.

The analogy in Figure 1 identifies the exact correspondence that turns manually engineered agent components into a joint optimization target.

5. Holistic updates are intended to enable post-deployment evolution The framework jointly edits prompts, tools, and pipeline topology instead of tuning each component separately. Because its loss prompt can evaluate a trajectory without a ground-truth label, the agent can in principle learn from experience after deployment; updating the underlying LLM through collected-data fine-tuning is explicitly left for future work.

6. The contribution is positioned as a proof of concept The experiments test whether symbolic learning can improve both conventional benchmarks and tasks whose quality is difficult to encode as an equation. The released framework and prompts target a transition from engineering-centric agent building toward data-centric agent learning, rather than claiming a mature or universally reliable training algorithm.

2 Related Work

2.1 Language Models, Prompts, and Language Agents

1. Prompts expose capabilities without changing model parameters Autoregressive Transformer LLMs acquire broad language capabilities through scale, while prompts control how those capabilities are expressed. In-context learning, chain-of-thought, ReAct, self-refinement, self-consistency, and recurrent prompting demonstrate that substantial behavior can be induced in the input and control layer rather than through parameter updates.

2. Agents extend prompting into tool-using systems Language agents embed LLM calls inside multi-step pipelines and let them invoke external tools. This changes the optimization object from one prompt to a structured system containing prompts, tools, routing decisions, intermediate state, and multiple LLM-backed roles.

2.2 From Automated Prompt Engineering to Agent Optimization

1. Automated prompt engineering follows two broad strategies Prompt-based methods ask an LLM to critique and rewrite a prompt, whereas search-based methods explore combinations or mutations using algorithms such as genetic search. Agent symbolic learning adopts LLM-mediated rewriting but expands its reach beyond one prompt.

2. Search-based agent optimization depends on explicit fitness Variational methods, DSPy, and GPTSwarm can optimize stacked LLM components when success is numerically measurable and searchable. The paper argues that software development and creative writing violate this convenient assumption because end-to-end quality cannot be captured fully by a simple executable metric.

3. Existing methods leave structure and tool creation under-optimized Prior component-wise methods do not jointly assign responsibility across nodes, add pipeline nodes, or implement new tools. Agent symbolic learning addresses these gaps through backward credit assignment and distinct prompt, tool, and pipeline optimizers.

4. Backbone fine-tuning and cross-task transfer are complementary directions Synthetic trajectories can fine-tune an agent’s underlying LLM, while inter-task transfer can reuse what one agent learns on another task. The proposed framework changes the symbolic scaffold and explicitly leaves joint evolution of symbolic components and model parameters unresolved.

3 Agent Symbolic Learning

1. Algorithm 1 alternates execution, textual credit assignment, and rewriting For an input \(\mathcal{I}\) and agent pipeline \(\mathcal{A}\), the method initializes a trajectory \(\tau\), executes every node, records each tuple \((\mathcal{I}_n,\mathcal{O}_n,\mathcal{P}_n,\mathcal{T}_n)\), computes a language loss, and traverses the nodes in reverse to generate language gradients. It then updates every node’s prompts \(\mathcal{P}_n\) and tools \(\mathcal{T}_n\), followed by the pipeline \(\mathcal{A}\) itself.

2. Reverse propagation couples local edits to the final task outcome The gradient for node \(n\) conditions on both the global language loss and the gradient from node \(n+1\). This makes an early node’s update respond not only to its own output but also to what the downstream node required, which is the mechanism intended to avoid independently reasonable yet globally incompatible edits.

3.1 Problem Formulation

1. The pipeline is a potentially dynamic computation graph The agent pipeline \(\mathcal{A}=\{\mathcal{N}_1,\mathcal{N}_2,\ldots,\mathcal{N}_n\}\) represents the ordered nodes and their connections. Some frameworks construct nodes dynamically from the input, so the analogy includes dynamic neural networks rather than assuming a permanently fixed sequence.

2. A node packages execution state and symbolic weights Node \(\mathcal{N}_n\) receives natural-language input \(\mathcal{I}_n\), invokes an LLM with prompts \(\mathcal{P}_n\) and tools \(\mathcal{T}_n\), and emits output \(\mathcal{O}_n\) for a later node. A tool’s symbolic representation includes its invocation input, output, and implementation, so tool optimization can change more than a function-call description.

3. The trajectory is the saved state for backward reasoning The trajectory \(\tau\) stores node inputs, outputs, prompts, and tool use from the forward pass. Its role parallels saved activations in automatic differentiation: without this record, the backward prompt could not connect a final failure to the symbolic choices made earlier in execution.

4. Loss and gradient are language objects rather than derivatives The language loss \(\mathcal{L}_{\mathrm{lang}}\) is textual feedback plus an LLM-generated score, and the language gradient \(\nabla_{\mathrm{lang}}\) is textual analysis about how a component should change. These objects borrow the functional roles of a loss and gradient but are not numerical derivatives and provide no mathematical guarantee of descent.

Figure 2 shows the full data flow: a frozen LLM executes a configurable agent, the trajectory is evaluated, and three optimizers consume backward-propagated language gradients.

3.2 Agent Symbolic Learning Procedure

1. The forward pass freezes the model while recording symbolic execution Ordinary agent execution is augmented by saving the input, prompt, tool call, and output at every node. Figure 2 labels the agents as frozen: the procedure changes the surrounding symbolic program, not the LLM parameters.

2. A prompted evaluator produces supervised or unsupervised language loss Given trajectory \(\tau\) and loss prompt \(\mathcal{P}_{\mathrm{loss}}\), the evaluator computes

The prompt combines the task description, input, full trajectory, few-shot demonstrations, evaluation principles, and output-format rules. Supplying the correct label yields supervised agent learning; omitting it lets the evaluator judge against the task description, which the paper calls unsupervised agent learning.

3. Language gradients are generated from downstream to upstream For node \(n\), the backward prompt implements

where \(\nabla_{\mathrm{lang}}^{n+1}\) is empty at the final node. Each critique therefore reflects both the global outcome and the next node’s unmet requirements, producing a chain of natural-language credit assignments.

4. PromptOptimizer edits prompt components separately Prompts are decomposed into task description, few-shot examples, principles, and output-format control. Component-specific rewrite prompts receive the relevant execution examples and language gradients, while format constraints require the new prompt to preserve template variables and remain machine-readable.

5. ToolOptimizer can edit, delete, or create tools The first stage chooses an operation, and a specialized prompt performs tool editing, deletion, or creation. This broadens the mutable substrate beyond natural-language instructions, although code-producing updates introduce validity and security risks that receive only basic safeguards.

6. PipelineOptimizer performs structural mutations The optimizer is taught the Agents configuration language and a small set of atomic graph operations. It analyzes the accumulated language gradients and can add, delete, or move nodes, enabling changes to task decomposition and control flow rather than only parameter-like prompt edits.

7. Retries, rollback, and a textual learning rate stabilize updates An illegal code-space update is retried up to three times and discarded if all attempts fail. After a syntactically valid update, the same example is rerun; if the prompted language-loss score falls, the system rolls back to the previous agent. Each optimizer prompt also includes a “learning rate” instruction controlling how aggressively it should rewrite the component.

8. Batched training aggregates critiques before one update The single-example procedure resembles stochastic gradient descent. Its batched variant independently executes, evaluates, and back-propagates each example, then gives all language gradients for the same node to the optimizer so it can seek a change that addresses the batch rather than overfitting one trajectory.

4 Experiments

4.1 Settings

4.1.1 Tasks

1. Three standard benchmarks test knowledge, mathematics, and code generation The study uses the hard split of HotPotQA for multi-hop question answering, MATH for competition-level mathematics, and HumanEval for program synthesis from docstrings. Metrics are exact match and F1 on HotPotQA, accuracy on MATH, and Pass@1 on HumanEval; tools are disabled so the comparison focuses on the agent pipeline rather than external capabilities.

2. Creative writing supplies open-ended planning pressure Each input contains four random sentences, and the system must write a coherent four-paragraph passage whose paragraphs end with those sentences in the given order. A GPT-4 judge scores the passages, so the task admits language feedback but also inherits the judge model’s preferences and possible self-evaluation bias.

3. Software development uses five products and a four-level rubric Given a short product requirements document, the system produces executable software for Flappy Bird, tank battle, 2048, Snake, and brick breaker. Scores range from 1 for execution failure, through 2 for successful execution and 3 for matching the expected workflow, to 4 for flawless alignment with expectations.

4.1.2 Baselines

1. The baselines separate prompting, fixed scaffolds, local rewriting, and search GPTs use one carefully designed prompt; Agents use manually designed prompts, tools, and a pipeline; Agents with AutoPE let an LLM optimize each node prompt without backward language gradients; and DSPy searches combinations of prompt components. Tree of Thought is additionally reported for creative writing, while DSPy is omitted from complex tasks because those evaluations are not expressed as an executable objective.

2. Two frozen OpenAI backbones isolate symbolic optimization effects Experiments use gpt-3.5-turbo-0125 and gpt-4-turbo-0409. The proposed method starts from the fixed Agents baseline and applies symbolic learning on top, but the paper does not state the number of training examples, update steps, independent runs, decoding parameters, token budget, or total cost.

4.2 Results

1. MATH shows the largest standard-benchmark improvement With GPT-3.5, symbolic learning reaches 38.8%, compared with 23.8% for Agents, 22.5% for AutoPE, and 17.3% for DSPy—a 15.0-point gain over its initialization. With GPT-4 it reaches 60.7%, versus 56.0% for Agents, 57.2% for AutoPE, and 48.4% for DSPy.

2. HotPotQA improves both exact match and F1 GPT-3.5 rises from 27/37.5 exact-match/F1 for Agents to 35/44.8; GPT-4 rises from 39/49.8 to 41/54.0. DSPy ties the proposed method’s GPT-3.5 exact match at 35 but has lower F1 (43.9), while its GPT-4 result is 40/50.5.

3. HumanEval qualifies the claim of universal superiority The proposed method scores 64.5 Pass@1 with GPT-3.5 and 85.8 with GPT-4, improving over the Agents initialization at 59.5 and 85.0. However, GPT-3.5 DSPy reaches 66.7, which exceeds 64.5; the reported values therefore do not support an unqualified claim that symbolic learning beats every baseline in every setting.

The complete standard-benchmark table makes both the strong MATH gains and the HumanEval exception visible.

4. Software development improves from partial execution to near-perfect behavior The mean score is 3.8/4, compared with 2.4 for Agents and 1.6 for GPTs. Symbolic learning scores 4 on tank battle, 2048, Snake, and brick breaker and 3 on Flappy Bird; the fixed Agents baseline ranges from 2 to 3 and never reaches 4.

5. Creative writing benefits from jointly learning prompts and topology The proposed method scores 6.9 with GPT-3.5 and 7.4 with GPT-4. The strongest non-proposed baselines are AutoPE at 4.4 for GPT-3.5 and Tree of Thought at 6.8 for GPT-4, giving margins of 2.5 and 0.6 points respectively; the learned agent discovers a plan–write–revise pipeline.

6. The learned software workflow resembles expert-written multi-agent practice For software development, optimization recovers a standard operating procedure similar to MetaGPT, even though it starts from the simpler Agents configuration. This is evidence that structural rewriting can rediscover useful role decomposition, but no topology ablation isolates how much of the gain comes from pipeline changes rather than prompt changes.

4.3 Case Study & Analysis

1. The creative-writing agent grows an editor and checker The initial graph contains one Write node with a single writer. Optimization preserves the writer, strengthens its instruction to ensure each paragraph ends with the specified sentence and improves narrative flow, adds an editor role for grammar, logic, and readability, and adds a Check node downstream.

Figure 3 makes the structural mutation concrete: symbolic learning changes both the text inside a node and the graph around it.

2. Simple initialization is empirically more stable Starting from the simplest agent and letting the optimizers add complexity works better than beginning with an over-engineered system, whose training becomes unstable. This suggests that symbolic initialization matters, much as weight initialization matters in neural training, but the paper reports the observation qualitatively rather than through a controlled initialization ablation.

3. Complex tasks appear more favorable than narrow automatic benchmarks Gains are larger and described as more stable on software development and creative writing than on exact-match, F1, accuracy, or Pass@1 benchmarks. The proposed explanation is that holistic language feedback better fits complex, multi-criterion outcomes, motivating a future benchmark of diverse agentic tasks and more robust progress measures.

4. Several experimental uncertainties remain unresolved The paper does not report dataset sample counts, training/validation/test separation for symbolic updates, update iterations, stochastic variability, significance tests, judge calibration, compute or API cost, or a component ablation. Because rollback evaluates the updated agent on the current example with the same prompted loss, its protection against held-out regressions and evaluator overfitting is unknown.

5 Conclusion

1. The framework reframes an agent scaffold as a learnable object Language loss, backward language gradients, and symbolic optimizers form a common interface for jointly editing prompts, tools, roles, nodes, and connections. This is the paper’s main conceptual contribution: data-driven optimization applies to the whole program surrounding a frozen LLM.

2. Reported improvements support feasibility rather than convergence Results cover three standard benchmarks, one creative-writing task, and five small software products, showing that the method can produce better scaffolds in varied settings. Natural-language gradients are heuristic critiques rather than true derivatives, so the neural-network analogy does not imply smoothness, unbiased credit assignment, monotonic progress, or convergence.

3. Safety controls are narrow and operational Three retries prevent persistent illegal updates, rollback rejects an immediately lower-scoring candidate, and a prompt-level learning rate moderates edit size. There is no adversarial evaluation of generated tools, sandboxing policy, permission model, held-out release gate, evaluator-robustness test, or analysis of whether an evolving agent can weaken its own controls.

4. Open questions concern scale, generalization, and joint learning The paper leaves open whether large-scale pretraining can produce a reusable agent initialization, how symbolic learning transfers across tasks, how to benchmark complex agent learning reliably, and how scaffold updates should interact with LLM fine-tuning. Further work also needs independent verification, cost-aware optimization, safe tool synthesis, and selection on held-out data rather than the same example that generated the update.

Appendix A Implementation Details

1. A configuration language makes the scaffold machine-editable The implementation adopts the Agents framework, where an agent’s prompts, roles, controllers, tools, nodes, and transitions are represented in configuration files. This representation is essential to the method: symbolic optimizers can emit bounded configuration operations instead of rewriting an opaque application directly.

Appendix B Prompt Templates

1. The loss prompt standardizes scores and suggestions With ground truth, the evaluator must produce a score out of 10 and a concise suggestion, penalizing outputs that are correct but contain unnecessary text. A second form accepts both a ground-truth answer and an external evaluation score, asks the LLM to optimize that score, and permits an explicit no-change suggestion when the output is satisfactory.

2. The backward prompt assigns responsibility at prompt and node levels Prompt-level back-propagation receives the current template, previous-node output, current output, and downstream requirement, then returns both a recommendation for the current prompt and a requirement for the preceding node. Node-level back-propagation may recommend changing a node description or routing controller, adding or deleting a role, or updating a role description.

3. Optimizer prompts enforce machine-readable edits PromptOptimizer must preserve all Python-format variables and emit one tagged replacement prompt; NodeOptimizer emits a JSON-loadable list of atomic actions. Empty outputs represent no change, while strict tags and JSON formatting reduce—but do not eliminate—the chance that an LLM-generated edit cannot be applied.