JudgeFlow: Agentic Workflow Optimization via Block Judge

Zihan Ma Affiliation: KAIST Correspondence to: zihanma@kaist.ac.kr Zhikai Zhao Affiliation: KAIST Chuanbo Hua Affiliation: KAIST Federico Berto Affiliation: KAIST Affiliation: Radical Numerics Jinkyoo Park Affiliation: KAIST Affiliation: Omelet

Abstract

Optimizing LLM-based agentic workflows is challenging for scaling AI capabilities. Current methods rely on coarse, end-to-end evaluation signals and lack fine-grained signals on where to refine, often resulting in inefficient or low-impact modifications. To address these limitations, we propose JudgeFlow, an Evaluation-Judge-Optimization-Update pipeline. We incorporate reusable, configurable logic blocks into agentic workflows to capture fundamental forms of logic. On top of this abstraction, we design a dedicated Judge module that inspects execution traces particularly failed runs and assigns rank-based responsibility scores to problematic blocks. These fine-grained diagnostic signals are then leveraged by an LLM-based optimizer, which focuses modifications on the most problematic block in the workflow. Our approach improves sample efficiency, enhances interpretability through block-level diagnostics, and provides a scalable foundation for automating increasingly complex agentic workflows. We evaluate JudgeFlow on mathematical reasoning and code generation benchmarks, where JudgeFlow achieves superior performance and efficiency compared to existing methods.

Keywords:

Machine Learning, ICML

^†^†affiliationnotice: Equal contribution

1 Introduction

Large language models (LLMs) (Brown et al., 2020) have achieved remarkable success across a wide range of domains. Moving beyond the scope of foundation models (Bommasani et al., 2022), by integrating LLMs into intelligent agent architectures, the emerging foundation agents (Liu et al., 2025a) have attracted more attention. Starting from early work on prompt engineering, such as reasoning-enhanced methods (Wei et al., 2023; Wang et al., 2023b; Yao et al., 2023a), to more recent developments in multi-agent system approaches (Du et al., 2023; Li et al., 2023; Hong et al., 2024), these handcrafted strategies have achieved strong performance across a range of tasks, including mathematical reasoning (Cobbe et al., 2021), code generation (Austin et al., 2021), question answering (Yang et al., 2018), and decision making (Sun et al., 2025).

However, these agentic systems still depend heavily on manual design, making workflow construction complex, costly, and inflexible. AutoML (Hutter et al., 2019) has shown that automating traditionally handcrafted and labor-intensive processes in machine learning can substantially reduce human effort and accelerate the development of high-performance models. Inspired by this success, recent efforts aim to automate the design and optimization of LLM-based agentic workflows (Lee et al., 2025). While these agentic systems still rely on LLMs as core execution engines, optimizing the LLMs themselves through pretraining or fine-tuning (Rafailov et al., 2023) often demands substantial computational resources and massive-scale data, making such approaches expensive in many settings (Kaplan et al., 2020). Instead, keeping the underlying model parameters fixed, and focusing on optimizing the systems structure and behavior leads to a more tractable and efficient optimization.

Automation efforts in agentic systems initially focused on prompt optimization, exemplified by Textual Gradients which leverage LLM feedback for end-to-end optimization (Pryzant et al., 2023; Yuksekgonul et al., 2024; Wang et al., 2024b; Yin and Wang, 2025). Current efforts are expanding to optimize architecture and execution flow of entire agentic systems. Agentic workflow can be modeled as neural network (Liu et al., 2024; Ma et al., 2025), graph (Zhuge et al., 2024a; Zhang et al., 2025a), and code (Hu et al., 2025; Zhang et al., 2025b; Zheng et al., 2025), each offering different levels of representational capacity, interpretability, and optimization difficulty. For instance, Directed Acyclic Graphs (DAGs)-represented workflows facilitate tractable optimization but constrain the ability to represent complex structures such as loops or conditional branching. In contrast, code-represented workflows provide comprehensive expressivity in defining intricate logic and control flow, but error attribution within code execution is difficult, and optimization often has to rely solely on end-to-end evaluation signals rather than fine-grained intermediate feedback. Building on code-represented workflows, Zhang et al. (2025b) introduce operators as modular units that encapsulate common agentic actions and propose a Monte Carlo Tree Search (MCTS) framework that employs LLMs to iteratively optimize workflows using past experience. However, the expansion phase in MCTS and the subsequent evaluation of candidate workflows can be expensive, and the effectiveness of the optimization process is constrained by the granularity of guidance available for modifications. In the absence of sufficiently fine-grained diagnostic information to precisely identify which specific part within the complex workflow requires modification, the search may explore ineffective or low-impact alterations. Furthermore, complex code structural interactions such as conditional constructs where only one branch of an if-else statement is executed along a trajectory leave certain components without informative signals, thereby hindering fine-grained analysis.

To address these challenges, we introduce JudgeFlow, an Evaluation-Judge-Optimization-Update pipeline. First, we incorporate reusable and configurable logic blocks into agentic workflows. These blocks capture three fundamental forms of logic: sequential, loop, and conditional, which are able to broadly represent code-based workflows. Compared with operators, which abstract specific agentic operations or functionalities (Zhang et al., 2025b), logic blocks serve as higher-level structural abstractions. By introducing logic blocks that abstract such common control structures, we retain the structural diversity of code-represented workflows while providing an intermediate level of abstraction between operators and workflows. This additional layer facilitates interpretability and exposes more meaningful diagnostic information for subsequent optimization.

Second, we incorporate a dedicated Judge module that analyzes the execution trace, with particular emphasis on failed runs. We hypothesize that optimizers should receive both evaluation and optimization signals. For each unsuccessful execution, the Judge attempts to identify the most problematic block within the workflow as illustrated in Figure 1. To further improve the precision of diagnosis, we adopt a rank-based approach at the block level. The resulting targeted diagnostic signals are propagated to the subsequent optimization stage, enabling more focused and efficient refinement of weak blocks. In this way, optimization efforts can be concentrated on repairing underperforming components, resulting in more effective and reliable improvements in overall workflow performance. Besides relying solely on end-to-end evaluation signals, our approach leverages block-level diagnostic information, enabling the optimizer to focus on the most problematic components.

Figure 1: Block-level Judge guides agentic workflow optimization by identifying the most problematic block in failed executions.

In summary, our contributions are as follows:

•

We propose a novel Evaluation-Judge-Optimization-Update pipeline named JudgeFlow;
•

We introduce reusable and configurable logic blocks as higher-level structural units, which balance the expressivity of code-based workflows with tractable optimization, while supporting interpretability and intermediate execution tracing;
•

We design a Judge module that analyzes execution traces, especially failed runs, and assigns rank-based responsibility scores to problematic blocks, enabling fine-grained error localization and targeted refinement for subsequent optimization.
•

We evaluate JudgeFlow on mathematical reasoning and code generation benchmarks, showing that it outperforms existing methods.

Figure 2: The illustration of logic blocks.

2 Related Work

LLM-based (Multi-)Agent Systems

In recent years, LLM-based (multi-)agent systems have achieved notable successes (Wang et al., 2024a; Huang et al., 2024; Tran et al., 2025). At the single-agent level, foundational works have enabled agents to reason and act by interleaving thought and action (Yao et al., 2023b), to enhance complex problem-solving through structured exploration of thoughts (Yao et al., 2023a), and to interact effectively with external tools or APIs (Wu et al., 2024). At the multi-agent level, frameworks such as CAMEL (Li et al., 2023), AutoGen (Wu et al., 2023), and MetaGPT (Hong et al., 2024) have facilitated sophisticated collaboration on complex tasks, demonstrating strong performance across diverse domains. Despite these advances, existing systems remain constrained by a reliance on handcrafted prompts and rigid communication topologies, which limit adaptability as task complexity scales. This has spurred a shift toward automated agentic systems capable of optimizing their own architectures and behaviors.

Agentic Systems Automation

Early automation efforts in agentic systems primarily focused on prompt optimization (Pryzant et al., 2023; Ramnath et al., 2025; Li et al., 2025), with approaches such as LLMs-as-optimizers (Yang et al., 2024), self-referential evolution (Fernando et al., 2023), textual gradients (Yuksekgonul et al., 2024), and self-supervised optimization (Xiang et al., 2025). More recent research has expanded beyond prompt-level tuning toward optimizing the architectures and execution flows of entire agentic systems. For example, Liu et al. (2024) explores dynamic communication structures for adaptive collaboration, while Zhuge et al. (2024a) models agents as computational graphs to refine both prompts and inter-agent orchestration. Shang et al. (2024) proposes a novel modular design automatically searching for high-performance agent structures. Zhou et al. (2024) investigates agents capable of self-optimization using symbolic optimizers. Hu et al. (2025) introduces a meta agent that automatically discovers novel, high-performing, and generalizable agentic system designs. Yin et al. (2025) introduces a self-referential framework that enables agents to recursively improve themselves. Zhang et al. (2025b) employs LLMs as optimizers with a Monte Carlo Tree Search (MCTS) variant to discover effective workflows. Zhang et al. (2025a) automatically evolve agentic supernet systems leading to query-specific workflows. Su et al. (2025) leverages debate and reflexion to collaboratively refine workflows while reducing search redundancy. Zheng et al. (2025) introduces safety-constrained evolutionary programming in a declarative graph space, ensuring structural validity and robustness. While these efforts mark significant progress, most existing approaches still focus on end-to-end or global architectural optimization, often leading to inefficient search and a lack of fine-grained diagnostic feedback, which limits both scalability and interpretability as task complexity grows.

LLM as a Judge

The LLM-as-a-judge paradigm leverages large language models to automate the evaluation of generated content, addressing the scalability limitations of human assessment (Gu et al., 2025). This approach has been widely adopted for assessing complex outputs based on predefined criteria (Li et al., 2024). To mitigate the potential bias of the LLM-as-a-Judge (Wang et al., 2023a), various methods have been proposed. Liu et al. (2025b) propose a ranking-based alignment method that significantly improves the judging performance of LLMs. In addition, Zhuge et al. (2024b) proposed the framework to use agentic systems to evaluate agentic systems. In a related application, Zhang et al. (2025c) attempts to automate the failure attribution for LLM multi-agent systems, revealing that providing stronger ground-truth signals can substantially improve attribution quality, and aggregated analysis across multiple failures can uncover reliable error patterns.

Figure 3: The main pipeline of JudgeFlow

3 Methodology

3.1 Problem Formulation

Our framework models an agentic workflow by hierarchically composing basic agentic actions (Operators) into structured logical units (Blocks) as follows.

A configured operator $O(D)$ is the basic unit of agentic action, where $O$ represents a categorical label for its core function like generate or self_refine (details in Appendix A), and $D$ is the operator configuration, which includes the LLM backbone, prompt template, and other hyperparameters (Zhang et al., 2025b). Building upon operators, a logic block $(B,C)$ is a higher-level structural unit that orchestrates one or more configured operators, where $B\in\mathcal{B}$ is the logic block type, dictating how the operators are orchestrated. The set of available types $\mathcal{B}$ includes three fundamental forms of logic as shown in Figure 2 (details in Appendix B):

•

SequenceLogic (seq): A sequential execution block where operators are executed one after another. Each operator consumes the output of its predecessor, ensuring a linear flow of intermediate results until the final operator produces the block output.
•

LoopLogic (for): An iterative block that repeatedly invokes its internal operators. The iteration continues until the stopping condition is satisfied.
•

ConditionalLogic (cond): A branching block that first executes a designated condition operator. Based on the evaluation outcome, it then activates one of two operator sequences. Only the operators in the selected branch are executed to generate the block output.

Correspondingly, $C$ is the logic block configuration, which contains the set of configured operators $O(D)$ in the block and block-level parameters (e.g., stopping condition in LoopLogic). Finally, the agentic workflow $W$ is defined as a tuple $W=\left(\left\{(B_{i},C_{i})\right\}_{i=1}^{M},S\right)$ , where $M$ is the total number of logic blocks in the workflow, and $S$ denotes the ordered sequence of logic blocks at the top level while each individual block may internally contain conditional or iterative control. This definition not only preserves the common logic patterns in code-represented workflows ensuring expressive diversity (Hu et al., 2025; Zhang et al., 2025b), but also enhances interpretability, including the explicit semantic characteristics of each logic block and the overall execution trajectory of the workflow facilitating subsequent optimization.

Given an input query $q$ from the dataset $\mathcal{D}$ which is available to every block, the execution function $\phi_{\text{exe}}$ processes workflow $W$ by sequentially applying its logic blocks along the execution order $S$ . Each block $(B_{i},C_{i})$ receives the state from the previous block, $a^{\prime}_{i-1}$ , and produces a new state, $a^{\prime}_{i}$ , formally defined as:

a^{\prime}_{i}=\phi_{\text{exe}}^{(i)}(a^{\prime}_{i-1},q;B_{i},C_{i}),i=1,2,\ldots,M,

(1)

where $\phi_{\text{exe}}^{(i)}$ is the execution function for block $i$ and $a^{\prime}_{0}=\varnothing$ . The final workflow output is obtained as $a^{\prime}_{M}$ , and then scored by the evaluation function $\phi_{\text{eval}}$ against the ground-truth answer $a$ corresponding to $q$ . The objective of agentic workflow optimization is to find the optimal workflow $W^{*}$ that maximizes evaluation performance across the dataset:

W^{*}=\underset{W\in\mathcal{W}}{\operatorname{argmax}}\ \mathbb{E}_{(q,a)\sim\mathcal{D}}\left[\phi_{\text{eval}}\left(a^{\prime}_{M},a\right)\right],

(2)

where $\mathcal{W}$ denotes the search space of candidate workflows.

3.2 JudgeFlow

A challenge in optimizing agentic workflows is credit assignment: identifying which component of a complex execution is responsible for failures and should be refined. Existing methods largely rely on coarse, end-to-end evaluation signals, making it difficult to perform targeted modifications. Building on the representation of workflow using logic blocks, JudgeFlow incorporates a dedicated Judge module and implements an iterative Evaluation-Judge-Optimization-Update pipeline for the efficient optimization of agentic workflows as shown in Figure 3.

3.2.1 Evaluation-Judge

The combined Evaluation-Judge stage, detailed in Algorithm 1, processes each input query from the dataset. If the workflow $W$ fails on a given query, the stage identifies and logs specific problematic block within $W$ . This provides targeted diagnostic signals for subsequent workflow optimization, enabling a more efficient and focused approach on refining these identified weak logic to improve overall optimization efficiency.

Specifically, for each input query $q$ (with a corresponding ground-truth answer $a$ ), we have $\{a_{i}^{\prime}\}_{i=1}^{M}=\phi_{\text{exe}}(q,W)$ , and score $s=\phi_{\text{eval}}(a_{M}^{\prime},a)$ . The score $s$ is recorded in a list $\mathcal{P}_{\text{scores}}$ for later calculation of $W$ ’s overall performance. Providing a threshold $\varepsilon$ that indicates successful execution, if $s\geq\varepsilon$ , the instance is marked as successful, and the algorithm simply proceeds to the next input.

However, if $s<\varepsilon$ , a quadruple $Q=(W,q,a,\{a_{i}^{\prime}\}_{i=1}^{M})$ is defined to encapsulate the full context of the failure. The Judge proceeds to examine the quadruple, assessing each block’s $\{B_{i}\}_{i=1}^{M}$ responsibility for the failure and ranking them accordingly. This procedure, guided by specific Judge prompts (detailed in Appendix C), yields a rank-based score vector (Liu et al., 2025b) $(r_{i})_{i=1}^{M}$ for the blocks where $r_{i}=1$ refers to the block deemed most responsible for the failure and $r_{i}=M$ denotes the least responsible, each rank from $1$ to $M$ is assigned exactly once. These block scores $(r_{i})_{i=1}^{M}$ are appended to $\mathcal{R}_{\text{ranks}}$ . The $\texttt{RoundWorst}((r_{i})_{i=1}^{M},W)$ function then utilizes this score vector to identify $B_{\text{rw}}$ , the block deemed most problematic for the current instance (i.e. $B_{\text{rw}}=\{\,B_{i}\mid r_{i}=1\,\}$ ) . Subsequently, the instance details $(q,a,\{a_{i}^{\prime}\}_{i=1}^{M})$ are logged into $\mathcal{L}_{B_{\text{rw}}}$ , the dedicated log for $B_{\text{rw}}$ , providing targeted few-shot examples for its potential future optimization.

Upon completion of all instances in $\mathcal{D}$ , the accumulated diagnostic information is processed. The $\texttt{OverallWorst}(\mathcal{R}_{\text{ranks}},W)$ function analyzes all block rank-based score vectors in $\mathcal{R}_{\text{ranks}}$ to identify $B_{\text{sel}}$ , the block deemed the most consistently problematic over the whole dataset. This statistical filtering mechanism is designed to ensure robustness against noisy outputs. We follow the finding (Zhang et al., 2025c) that while individual LLM-based failure attribution might contain noise, the aggregated distribution across multiple failures is more consistent with the true causes. In practice, we aggregate rank vectors across all failing instances in $\mathcal{R}_{\text{ranks}}$ by summing the scores $r_{k}$ assigned to each block $B_{k}$ , and then selects the block achieving the minimum sum (i.e. $B_{\text{sel}}=\arg\min_{B_{k}\in W}\sum_{t=1}^{T}r_{k}^{(t)}$ , where $T$ is the number of the failure executions). Concurrently, the overall performance $P_{W}$ of $W$ on $\mathcal{D}$ is computed by $\texttt{CalPerformance}(\mathcal{P}_{\text{scores}})$ . Finally, this stage returns $P_{W}$ , $B_{\text{sel}}$ , and $\mathcal{L}_{B_{\text{sel}}}$ , providing actionable insights for subsequent optimization.

Algorithm 1 Evaluation-Judge

1: Input: Workflow

W

, Dataset

\mathcal{D}

, executor

\phi_{\text{exe}}

, evaluator

\phi_{\text{eval}}

, Judge, threshold

\varepsilon

2: Output: Performance

P_{W}

, Selected Block

B_{\text{sel}}

and the corresponding Log

\mathcal{L}_{B_{\text{sel}}}

3: For

k\leftarrow 1\text{ to }M

: Initialize

\mathcal{L}_{B_{k}}\leftarrow\emptyset

\mathcal{R}_{\text{ranks}}\leftarrow\emptyset

\mathcal{P}_{\text{scores}}\leftarrow\emptyset

5: for each

(q,a)\in\mathcal{D}

\{a_{i}^{\prime}\}_{i=1}^{M}\leftarrow\phi_{\text{exe}}(q,W)

s\leftarrow\phi_{\text{eval}}(a^{\prime}_{M},a)

\mathcal{P}_{\text{scores}}\leftarrow\textsc{Append}(\mathcal{P}_{\text{scores}},s)

9: if

s\geq\varepsilon

then

10: continue

11: else

12:

(r_{i})_{i=1}^{M}\leftarrow\text{Judge}(W,q,a,\{a_{i}^{\prime}\}_{i=1}^{M})

13:

\mathcal{R}_{\text{ranks}}\leftarrow\textsc{Append}(\mathcal{R}_{\text{ranks}},(r_{i})_{i=1}^{M})

14:

B_{\text{rw}}\leftarrow\texttt{RoundWorst}((r_{i})_{i=1}^{M},W)

15:

\mathcal{L}_{B_{\text{rw}}}\leftarrow\textsc{Append}(\mathcal{L}_{B_{\text{rw}}},(q,a,\{a_{i}^{\prime}\}_{i=1}^{M}))

16: end if

17: end for

18:

B_{\text{sel}}\leftarrow\texttt{OverallWorst}(\mathcal{R}_{\text{ranks}},W)

19:

P_{W}\leftarrow\texttt{CalPerformance}(\mathcal{P}_{\text{scores}})

20: Return:

P_{W},B_{\text{sel}},\mathcal{L}_{B_{\text{sel}}}

Table 1: Performance comparison with baselines on GSM8K, MATH, MBPP, and HumanEval. The results are evaluated averaged over three independent runs. We use gpt-4o-mini-0718 in the experiments.

Method	GSM8K	MATH	MBPP	HumanEval	Avg.
Single-agent System
IO	87.8	48.6	73.9	87.0	74.3
CoT (Wei et al., 2023)	87.0	48.8	74.2	88.6	74.7
CoT SC (Wang et al., 2023b)	86.9	50.4	73.3	91.6	75.6
Hand-crafted Multi-agent System
SELF-REFINE (Madaan et al., 2023)	85.5	46.1	71.8	87.8	72.8
LLM-Debate (Du et al., 2023)	89.5	48.6	70.3	88.8	74.3
LLM-Blender (Jiang et al., 2023)	88.4	46.9	77.1	88.7	75.3
DyLAN (Liu et al., 2024)	90.0	48.5	77.3	90.4	76.6
Autonomous Multi-agent System
GPTSwarm (Zhuge et al., 2024a)	89.1	47.9	77.4	89.3	75.9
ADAS (Hu et al., 2025)	88.4	43.2	77.1	84.2	73.2
AFlow (Zhang et al., 2025b)	90.1	52.8	81.7	90.1	78.7
MaAS (Zhang et al., 2025a)	91.5	52.2	82.2	91.6	79.4
MermaidFlow (Zheng et al., 2025)	92.4	55.4	82.3	92.9	80.8
JudgeFlow (Ours)	93.0	58.5	83.8	93.4	82.2

3.2.2 Optimization-Update

In the subsequent Optimization-Update stage, the LLM-based optimizer utilizes the insights from the previous stage and refines $W$ to produce an improved version $W^{\prime}$ guided by specific optimization prompts (detailed in Appendix D), which can be formally expressed as

W^{\prime}\leftarrow\text{Optimizer}(W,B_{\text{sel}},A,\text{sample}(\mathcal{L}_{B_{\text{sel}}}))

(3)

where $\text{sample}(\mathcal{L}_{B_{\text{sel}}})$ refers to few-shot samples drawn from the logs $\mathcal{L}_{B_{\text{sel}}}$ and $A\in\mathcal{A}$ , where $\mathcal{A}$ is a predefined set of available modification actions as follows:

•

Add Block : Introduce a new block $B_{\text{new}}$ with configuration $C_{\text{new}}$ , and connect it directly with the low-performing block $B_{\text{sel}}$ ;
•

Remove Block: Remove the low-performing block $B_{\text{sel}}$ together with all of its incident edges while reconnecting its predecessor and successor to preserve sequential flow;
•

Modify Block: Reconfigure the existing $B_{\text{sel}}$ by updating its configuration $C_{\text{sel}}\mapsto C^{\prime}_{\text{sel}}$ .

In practice, the LLM-based optimizer selects $A$ adaptively based on the diagnostic signals in $\mathcal{L}_{B_{\text{sel}}}$ . Following Zhang et al. (2025b), the refined workflow $W^{\prime}$ is first evaluated to obtain its performance score $P_{W^{\prime}}$ . The pair $(W^{\prime},P_{W^{\prime}})$ is then added to the candidate pool $\mathcal{W}_{\text{pool}}$ , which retains at most $K$ workflows by keeping the top- $K$ highest-scoring entries:

\mathcal{W}_{\text{pool}}\leftarrow\text{Top-}K\big(\mathcal{W}_{\text{pool}}\cup\{(W^{\prime},P_{W^{\prime}})\}\big).

(4)

At the beginning of the next iteration, the optimizer selects a starting workflow $W_{\text{start}}\sim\mathcal{W}_{\text{pool}}$ according to a softmax distribution:

\Pr(W_{i})=\frac{\exp\!\left(\tfrac{s_{i}-\max_{j}s_{j}}{\tau}\right)}{\sum_{k=1}^{|\mathcal{W}_{\text{pool}}|}\exp\!\left(\tfrac{s_{k}-\max_{j}s_{j}}{\tau}\right)},

(5)

where $s_{i}$ is the evaluation score of workflow $W_{i}$ .

4 Experiments

4.1 Experimental Setups

Benchmarks

We evaluate JudgeFlow on widely used benchmarks, covering math reasoning tasks (GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), AIME (Ye et al., 2025)) and code generation tasks (MBPP (Austin et al., 2021), HumanEval (Chen et al., 2021)). Following previous studies (Zhang et al., 2025b; Zhang et al., 2025a), each dataset is divided into training and test sets. We report the solve rate (%) on GSM8K, MATH, AIME, and pass@1 on MBPP, HumanEval to evaluate.

Baselines

We compare our JudgeFlow with a series of baselines, including (1) Single-agent System: Standard prompting (IO), Chain-of-Thought prompting (CoT) (Wei et al., 2023), and Self-Consistency (Wang et al., 2023b); (2) Hand-crafted Multi-agent System: MultiPersona (Wang et al., 2024c), SELF-REFINE (Madaan et al., 2023), LLM-Debate (Du et al., 2023), LLM-Blender (Jiang et al., 2023), and DyLAN (Liu et al., 2024); (3) Autonomous Multi-agent System: GPTSwarm (Zhuge et al., 2024a), ADAS (Hu et al., 2025), AFlow (Zhang et al., 2025b), MaAS (Zhang et al., 2025a), and MermaidFlow (Zheng et al., 2025). Compared with domain-specialized agents, these methods provide a more appropriate setting to evaluate the effectiveness of our block-level Judge in workflow optimization.

Implementation Details

In our main experiments, to keep consistent with prior literature (Zheng et al., 2025), we use gpt-4o-mini-0718 (OpenAI, 2024b), gpt-4.1-mini (OpenAI, 2025) as the optimization, Judge and execution LLM accessed via API. The number of iteration rounds is set to 20. When optimizing, we set $M\leq 3$ , $\varepsilon=1$ , and $K=3$ .

4.2 Experimental Results

As shown in Table 1, JudgeFlow achieves superior performance compared to several strong baselines across all the tasks. Notably, for some challenging benchmarks such as MATH and MBPP, JudgeFlow outperforms the strongest prior baseline by +3.1(5.6%) and +1.5(1.8%), respectively. At the same time, for relatively simpler benchmarks such as GSM8K and HumanEval, JudgeFlow still achieves consistent gains of +0.6 and +0.5. Taken together, JudgeFlow achieves the average score of 82.2, representing a +1.4(1.7%) increase. As shown in Figure 4, in significantly more challenging AIME benchmark, JudgeFlow achieves an average score of 44.67. The results highlight the effectiveness of our Judge-guided block-level optimization across both reasoning and code generation tasks.

Figure 4: Performance on AIME 2025. The results are evaluated averaged over five independent runs. We use gpt-4.1-mini in the experiments.

4.3 Analysis

We take the MBPP dataset as an illustrative example to analyze JudgeFlow.

Best Performing Workflow

Figure 5 is the best-performing workflow found by JudgeFlow. First, a seq block b1 applies a generate operator to produce an initial candidate function. Second, a for block b2 repeatedly invokes the test operator until the stopping condition is satisfied. Finally, a cond block b3 runs the test operator to check correctness: if the candidate doesn’t pass, it routes the solution to a self_refine operator.

Figure 5: The optimal workflow found by JudgeFlow on the MBPP dataset.

Ablation

The core of JudgeFlow is the synergy between logic blocks and the Judge module, where logic blocks provide well-defined execution boundaries that enable fine-grained failure attribution. To evaluate this design, we compare JudgeFlow and AFlow which lacks Judge on the block-level abstraction as shown in Figure 6. JudgeFlow exhibits performance gains within the first five optimization iterations, with both the training and testing curves showing rapid improvements. Beyond this early stage, JudgeFlow continues to achieve gains, ultimately converging to higher accuracy. In contrast, AFlow remains stagnant across most iterations and only shows noticeable improvements in the later stage, and its final training and testing performance remain consistently lower than those of JudgeFlow.

Figure 6: Training and testing curves of JudgeFlow and AFlow on the MBPP dataset.

Figure 7: The illustration of the case study in the GSM8K dataset.

Impact of LLMs

According to Table 2, we keep gpt-4o-mini-0718 fixed as the executor LLM, while varying the optimization and Judge models. Particularly, we consider gpt-4o (OpenAI, 2024a) and Gemini-2.5-flash (Google-Cloud, 2025) as alternatives for these roles and report the resulting performance. The experiment confirms that increasing the capacity of optimization and Judge models consistently improves performance. While all models yield competitive results, gpt-4o attains the best score 84.5.

Table 2: Performance with different LLMs on MBPP.

Models	Score
GPT-4o-mini	83.8
GPT-4o	84.5
Gemini-2.5-flash	84.4

Cross-benchmark Generalization

We evaluate the cross-benchmark transferability of JudgeFlow by optimizing the workflow using the MATH(MBPP) dataset and zero-shot evaluating on the GSM8K(HumanEval) dataset. The results show that JudgeFlow yields better transferability on both math transfer and code transfer as shown in Table 3.

Table 3: Cross-Benchmark Transfer Performance

	AFlow	JudgeFlow
MATH $\rightarrow$ GSM8K	91.95	92.89
MBPP $\rightarrow$ HumanEval	90.84	93.89

Optimization Cost

Although JudgeFlow introduces an additional Judge module for LLM calls, the dominant cost in agentic workflow optimization lies in the Evaluation phase rather than the Judge module in our proposed methods. We monitor the cost for a single optimization round on the GSM8K dataset, the Evaluation cost ($0.45) is considerably higher than the Judge cost ($0.01). The Judge/Evaluation cost ratio is approximately 2%, demonstrating that the fine-grained diagnosis provides significant optimization guidance at a marginal overhead.

4.4 Case Study

To illustrate how JudgeFlow works in practice, we present a case study of workflow optimization on the GSM8K dataset as shown in Figure 7. The initial workflow consists of two logic blocks: b1, a seq block consisting of one multi_generate_ensemble operator designed to generate and ensemble multiple candidate solutions (with num_solutions set to 3), and b2, a seq block consisting of one programmer operator, which takes the output from the previous block and generates the final answer using programming. When processing a batch of GSM8K instances, this workflow failed multiple times, triggering the Evaluation-Judge stage. The Judge module analyzed execution traces of these failures and assigned rank-based responsibility scores to each block. For example, in one failed run, it output {"b2": 1, "b1": 2}, attributing the primary blame to b2, while in another it output {"b1": 1, "b2": 2}, assigning higher responsibility to b1. By aggregating these rank-based scores across failures, the system identified b1 as the OverallWorst block, indicating that low-quality initial solutions from b1 were the main bottleneck, making it difficult for the workflow to generate correct final answers. In the Optimization-Update stage, the LLM-based Optimizer received this diagnostic signal and selected the Add Block action. It introduced a new logic block, b3, of type seq, with operator self_refine, which iteratively improves candidate solutions. This block was inserted between b1 and b2, producing the new workflow ["b1", "b3", "b2"]. The updated workflow first generates multiple ideas with b1, then refines them with b3, and finally produces the polished answer through b2. This case study demonstrates how block-level diagnostics enable targeted workflow improvements.

5 Conclusion

In this paper, we presented a novel Evaluation-Judge-Optimization-Update pipeline named JudgeFlow for automating agentic workflow optimization. By introducing reusable logic blocks as higher-level structural abstractions, JudgeFlow achieves a balance between the expressive flexibility of code-based workflows and the tractability of optimization. On top of this representation, the Judge module provides block-level diagnostic signals by analyzing execution traces and assigning responsibility to the problematic block, enabling more interpretable and fine-grained optimization. Through extensive experiments on mathematical reasoning and code generation benchmarks, we demonstrate that JudgeFlow consistently outperforms strong baselines. Future work may include exploring more robust Judge for agentic systems optimization.

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

References

Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program Synthesis with Large Language Models. arXiv (en-US). Note: arXiv:2108.07732 [cs] External Links: Link, Document Cited by: §1, §4.1.
Bommasani et al. (2022) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. v. Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang On the Opportunities and Risks of Foundation Models. arXiv. Note: arXiv:2108.07258 [cs] External Links: Link, Document Cited by: §1.
Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language Models are Few-Shot Learners. arXiv. Note: arXiv:2005.14165 [cs] External Links: Link, Document Cited by: §1.
Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating Large Language Models Trained on Code. arXiv. Note: arXiv:2107.03374 [cs] External Links: Link, Document Cited by: §4.1.
Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training Verifiers to Solve Math Word Problems. arXiv (en-US). Note: arXiv:2110.14168 [cs] External Links: Link, Document Cited by: §1, §4.1.
Du et al. (2023) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv (en-US). External Links: Link, Document Cited by: §1, Table 1, §4.1.
Fernando et al. (2023) C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution. arXiv (en-US). External Links: Link, Document Cited by: §2.
Google-Cloud (2025) Google-Cloud Gemini 2.5 flash— vertex ai. Note: https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flashAccessed: 2025-05-18 Cited by: §4.3.
Gu et al. (2025) J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y. Wang, W. Gao, L. Ni, and J. Guo A Survey on LLM-as-a-Judge. arXiv (en-US). External Links: Link, Document Cited by: §2.
Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring Mathematical Problem Solving With the MATH Dataset. arXiv. Note: arXiv:2103.03874 [cs] External Links: Link, Document Cited by: §4.1.
Hong et al. (2024) S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv (en-US). External Links: Link Cited by: §1, §2.
Hu et al. (2025) S. Hu, C. Lu, and J. Clune Automated Design of Agentic Systems. arXiv (en-US). External Links: Link, Document Cited by: §1, §2, §3.1, Table 1, §4.1.
Huang et al. (2024) X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, R. Tang, and E. Chen Understanding the planning of LLM agents: A survey. arXiv. Note: arXiv:2402.02716 [cs] External Links: Link, Document Cited by: §2.
Hutter et al. (2019) F. Hutter, L. Kotthoff, and J. Vanschoren Automated machine learning: methods, systems, challenges. 1st edition, Springer Publishing Company, Incorporated. External Links: ISBN 3030053172 Cited by: §1.
Jiang et al. (2023) D. Jiang, X. Ren, and B. Y. Lin LLM-blender: ensembling large language models with pairwise ranking and generative fusion. External Links: 2306.02561, Link Cited by: Table 1, §4.1.
Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling Laws for Neural Language Models. arXiv. Note: arXiv:2001.08361 [cs] External Links: Link, Document Cited by: §1.
Lee et al. (2025) Y. Lee, G. Yi, M. Liu, J. Lu, G. Yang, and Y. Chen Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions. arXiv. Note: arXiv:2506.08234 [cs] External Links: Link, Document Cited by: §1.
Li et al. (2023) G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem CAMEL: Communicative Agents for ”Mind” Exploration of Large Language Model Society. arXiv (en-US). External Links: Link Cited by: §1, §2.
Li et al. (2024) H. Li, Q. Dong, J. Chen, H. Su, Y. Zhou, Q. Ai, Z. Ye, and Y. Liu LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv. External Links: Link, Document Cited by: §2.
Li et al. (2025) W. Li, X. Wang, W. Li, and B. Jin A Survey of Automatic Prompt Engineering: An Optimization Perspective. arXiv (en-US). External Links: Link, Document Cited by: §2.
Liu et al. (2025a) B. Liu, X. Li, J. Zhang, J. Wang, T. He, S. Hong, H. Liu, S. Zhang, K. Song, K. Zhu, Y. Cheng, S. Wang, X. Wang, Y. Luo, H. Jin, P. Zhang, O. Liu, J. Chen, H. Zhang, Z. Yu, H. Shi, B. Li, D. Wu, F. Teng, X. Jia, J. Xu, J. Xiang, Y. Lin, T. Liu, T. Liu, Y. Su, H. Sun, G. Berseth, J. Nie, I. Foster, L. Ward, Q. Wu, Y. Gu, M. Zhuge, X. Tang, H. Wang, J. You, C. Wang, J. Pei, Q. Yang, X. Qi, and C. Wu Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems. arXiv (en-US). External Links: Link, Document Cited by: §1.
Liu et al. (2025b) Y. Liu, H. Zhou, Z. Guo, E. Shareghi, I. Vulić, A. Korhonen, and N. Collier Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators. arXiv (en-US). External Links: Link, Document Cited by: §2, §3.2.1.
Liu et al. (2024) Z. Liu, Y. Zhang, P. Li, Y. Liu, and D. Yang A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration. arXiv (en-US). External Links: Link, Document Cited by: §1, §2, Table 1, §4.1.
Ma et al. (2025) X. Ma, C. Lin, Y. Zhang, V. Tresp, and Y. Ma Agentic Neural Networks: Self-Evolving Multi-Agent Systems via Textual Backpropagation. arXiv. Note: arXiv:2506.09046 [cs] External Links: Link, Document Cited by: §1.
Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark Self-refine: iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 1, §4.1.
OpenAI (2024a) OpenAI GPT-4o — openai platform documentation. Note: https://platform.openai.com/docs/models/gpt-4o Cited by: §4.3.
OpenAI (2024b) OpenAI GPT-4o-mini — openai platform documentation. Note: https://platform.openai.com/docs/models/gpt-4o-miniAccessed: 2025-05-18 Cited by: §4.1.
OpenAI (2025) OpenAI GPT-4_1-mini — openai platform documentation. Note: https://platform.openai.com/docs/models/gpt-4.1-miniAccessed: 2025-11-27 Cited by: §4.1.
Pryzant et al. (2023) R. Pryzant, D. Iter, J. Li, Y. Lee, C. Zhu, and M. Zeng Automatic Prompt Optimization with “Gradient Descent” and Beam Search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 7957–7968 (en-US). External Links: Link, Document Cited by: §1, §2.
Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv (en). External Links: Link Cited by: §1.
Ramnath et al. (2025) K. Ramnath, K. Zhou, S. Guan, S. S. Mishra, X. Qi, Z. Shen, S. Wang, S. Woo, S. Jeoung, Y. Wang, H. Wang, H. Ding, Y. Lu, Z. Xu, Y. Zhou, B. Srinivasan, Q. Yan, Y. Chen, H. Ding, P. Xu, and L. L. Cheong A Systematic Survey of Automatic Prompt Optimization Techniques. arXiv (en-US). Note: arXiv:2502.16923 [cs] External Links: Link, Document Cited by: §2.
Shang et al. (2024) Y. Shang, Y. Li, K. Zhao, L. Ma, J. Liu, F. Xu, and Y. Li AgentSquare: Automatic LLM Agent Search in Modular Design Space. arXiv (en-US). External Links: Link Cited by: §2.
Su et al. (2025) J. Su, Y. Xia, R. Shi, J. Wang, J. Huang, Y. Wang, T. Shi, Y. Jingsong, and L. He DebFlow: Automating Agent Creation via Agent Debate. arXiv. Note: arXiv:2503.23781 [cs] External Links: Link, Document Cited by: §2.
Sun et al. (2025) C. Sun, S. Huang, and D. Pompili LLM-based multi-agent decision-making: challenges and future directions. IEEE Robotics and Automation Letters 10 (6), pp. 5681–5688. External Links: Document Cited by: §1.
Tran et al. (2025) K. Tran, D. Dao, M. Nguyen, Q. Pham, B. O’Sullivan, and H. D. Nguyen Multi-Agent Collaboration Mechanisms: A Survey of LLMs. arXiv (en-US). External Links: Link, Document Cited by: §2.
Wang et al. (2024a) L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6) (en). External Links: ISSN 2095-2228, 2095-2236, Link, Document Cited by: §2.
Wang et al. (2023a) P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui Large Language Models are not Fair Evaluators. arXiv (en-US). Note: arXiv:2305.17926 [cs] External Links: Link, Document Cited by: §2.
Wang et al. (2024b) W. Wang, H. A. Alyahya, D. R. Ashley, O. Serikov, D. Khizbullin, F. Faccio, and J. Schmidhuber How to Correctly do Semantic Backpropagation on Language-based Agentic Systems. arXiv (en-US). External Links: Link, Document Cited by: §1.
Wang et al. (2023b) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. External Links: 2203.11171, Link Cited by: §1, Table 1, §4.1.
Wang et al. (2024c) Z. Wang, S. Mao, W. Wu, T. Ge, F. Wei, and H. Ji Unleashing the emergent cognitive synergy in large language models: a task-solving agent through multi-persona self-collaboration. External Links: 2307.05300, Link Cited by: §4.1.
Wei et al. (2023) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. External Links: 2201.11903, Link Cited by: §1, Table 1, §4.1.
Wu et al. (2023) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv (en-US). External Links: Link Cited by: §2.
Wu et al. (2024) S. Wu, S. Zhao, Q. Huang, K. Huang, M. Yasunaga, K. Cao, V. N. Ioannidis, K. Subbian, J. Leskovec, and J. Zou AvaTaR: optimizing llm agents for tool usage via contrastive reasoning. External Links: 2406.11200, Link Cited by: §2.
Xiang et al. (2025) J. Xiang, J. Zhang, Z. Yu, F. Teng, J. Tu, X. Liang, S. Hong, C. Wu, and Y. Luo Self-Supervised Prompt Optimization. arXiv (en-US). External Links: Link, Document Cited by: §2.
Yang et al. (2024) C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen Large Language Models as Optimizers. arXiv (en-US). External Links: Link, Document Cited by: §2.
Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. External Links: 1809.09600, Link Cited by: §1.
Yao et al. (2023a) S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv. Note: arXiv:2305.10601 [cs] External Links: Link, Document Cited by: §1, §2.
Yao et al. (2023b) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: Synergizing Reasoning and Acting in Language Models. arXiv. Note: arXiv:2210.03629 [cs] External Links: Link, Document Cited by: §2.
Ye et al. (2025) Y. Ye, Y. Xiao, T. Mi, and P. Liu AIME-preview: a rigorous and immediate evaluation framework for advanced mathematical reasoning. Note: https://github.com/GAIR-NLP/AIME-PreviewGitHub repository Cited by: §4.1.
Yin and Wang (2025) L. Yin and Z. Wang LLM-AutoDiff: Auto-Differentiate Any LLM Workflow. arXiv (en-US). External Links: Link, Document Cited by: §1.
Yin et al. (2025) X. Yin, X. Wang, L. Pan, X. Wan, and W. Y. Wang Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement. arXiv (en-US). External Links: Link, Document Cited by: §2.
Yuksekgonul et al. (2024) M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou TextGrad: Automatic ”Differentiation” via Text. arXiv (en-US). External Links: Link, Document Cited by: §1, §2.
Zhang et al. (2025a) G. Zhang, L. Niu, J. Fang, K. Wang, L. Bai, and X. Wang Multi-agent Architecture Search via Agentic Supernet. arXiv (en-US). External Links: Link, Document Cited by: Appendix A, §1, §2, Table 1, §4.1, §4.1.
Zhang et al. (2025b) J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. Wu AFlow: Automating Agentic Workflow Generation. arXiv (en-US). External Links: Link, Document Cited by: Appendix A, §1, §1, §2, §3.1, §3.1, §3.2.2, Table 1, §4.1, §4.1.
Zhang et al. (2025c) S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, and Q. Wu Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. arXiv (en-US). Note: arXiv:2505.00212 [cs] External Links: Link, Document Cited by: §2, §3.2.1.
Zheng et al. (2025) C. Zheng, J. Chen, Y. Lyu, W. Z. T. Ng, H. Zhang, Y. Ong, I. Tsang, and H. Yin MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming. arXiv (en). Note: arXiv:2505.22967 [cs] External Links: Link, Document Cited by: Appendix A, §1, §2, Table 1, §4.1, §4.1.
Zhou et al. (2024) W. Zhou, Y. Ou, S. Ding, L. Li, J. Wu, T. Wang, J. Chen, S. Wang, X. Xu, N. Zhang, H. Chen, and Y. E. Jiang Symbolic Learning Enables Self-Evolving Agents. arXiv (en-US). External Links: Link, Document Cited by: §2.
Zhuge et al. (2024a) M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber Language Agents as Optimizable Graphs. arXiv (en-US). External Links: Link, Document Cited by: §1, §2, Table 1, §4.1.
Zhuge et al. (2024b) M. Zhuge, C. Zhao, D. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, Y. Shi, V. Chandra, and J. Schmidhuber Agent-as-a-Judge: Evaluate Agents with Agents. arXiv (en-US). External Links: Link Cited by: §2.

Appendix A Operators

Following Zhang et al. (2025b), Zhang et al. (2025a) and Zheng et al. (2025), we adopt the following set of operators:

1.

generate, a generation operator that produces candidate solutions based on the problem description and optional previous results.
2.

test, a testing operator that executes generated solutions against test cases and provides feedback for refinement.
3.

self_refine, a refinement operator that improves a given solution through self-refinement.
4.

multi_generate_ensemble, an ensemble operator that generates multiple solutions and combine them to the best one via self-consistency.
5.

programmer, a synthesis-and-execution operator that generates Python code for solving math problems, runs it in a restricted environment, and iteratively repairs errors.

Appendix B Logic Blocks

We implement three common logic types in code-represented workflows: SequenceLogic (seq), LoopLogic (for), and ConditionalLogic (cond), whose descriptions and interfaces are listed below.

⬇

{

”SequenceLogic”: {

”type”: ”seq”,

”description”: ”Execute operators strictly in order. Required fields: name (string), type (must be ’seq’), operators (array of operator aliases). No optional fields. Use this for linear processing flows where you need sequential execution of operators.”,

”structure”: {

”name”: ”block_name”,

”type”: ”seq”,

”operators”: [”operator”]

”input_flow”: ”block_input -> op1 -> op2 -> … -> block_output”

”LoopLogic”: {

”type”: ”for”,

”description”: ”Iteratively execute a sequence of operators until the optional asynchronous condition returns False or the max iteration limit is reached. Required fields: name (string), type (must be ’for’), operators (array of operator aliases). Optional fields: max_iterations (integer, default 3), condition (object with ’field’ and ’equals’ properties, or null for no condition). Use this for retry mechanisms and iterative refinement.”,

”structure”: {

”name”: ”block_name”,

”type”: ”for”,

”operators”: [”operator”],

”max_iterations”: num_iterations,

”condition”: {

”field”: ”field_name”,

”equals”: ”some_value”

}

”input_flow”: ”block_input -> repeat [op1 -> op2 -> …] until stop -> block_output”

”ConditionalLogic”: {

”type”: ”cond”,

”description”: ”Run a dedicated condition operator first, then choose the success or failure branch based on the field specified by ’condition_field’. The chosen branch runs sequentially with the same data-passing semantics as SequenceLogic. Required fields: name (string), type (must be ’cond’), condition_operator (string, operator alias to evaluate condition), success_operators (array of operator aliases for success path), failure_operators (array of operator aliases for failure path). Optional fields: condition_field (string, field name to check for condition result, default ’result’). The condition operator evaluates criteria and sets a result field, which determines whether to execute success_operators or failure_operators. Use this for branching logic and conditional processing. ”,

”structure”: {

”name”: ”block_name”,

”type”: ”cond”,

”condition_operator”: ”condition_operator”,

”success_operators”: [”success_op”],

”failure_operators”: [”failure_op”],

”condition_field”: ”field_name”

”input_flow”: ”block_input -> condition operator -> select branch -> branch sequence -> block_output”

}

Appendix C Judge Prompt

The user prompt provides the problem, correct answer, incorrect answer, workflow structure, execution trace in XML format, and the list of blocks to rank.

JudgeFlow: Agentic Workflow Optimization via Block Judge

Abstract

Keywords:

1 Introduction

2 Related Work

LLM-based (Multi-)Agent Systems

Agentic Systems Automation

LLM as a Judge

3 Methodology

3.1 Problem Formulation

3.2 JudgeFlow

3.2.1 Evaluation-Judge

3.2.2 Optimization-Update

4 Experiments

4.1 Experimental Setups

Benchmarks

Baselines

Implementation Details

4.2 Experimental Results

4.3 Analysis

Best Performing Workflow

Ablation

Impact of LLMs

Cross-benchmark Generalization

Optimization Cost

4.4 Case Study

5 Conclusion

Impact Statement

References

Appendix A Operators

Appendix B Logic Blocks

Appendix C Judge Prompt

Appendix D Optimization Prompt