Agent Loop and Runtime Design

An agent becomes useful when the model is placed inside a runtime loop.

The loop is the part of the system that keeps asking: what does the model need to see, what action did it choose, is that action allowed, what happened, should we continue, and what evidence proves we are done?

This is why agent design is less like writing one prompt and more like building a small operating system around a model. The model proposes. The runtime coordinates.

Level 1: Fundamentals

The classic ReAct pattern showed why reasoning and acting belong together: the model can interleave a reasoning trace with task-specific actions, use observations from external systems, update its plan, and recover from exceptions [1]. Modern agent runtimes implement the same idea in a more productized form.

OpenAI's current agent-running docs describe a runner loop that calls the model, inspects the output, executes tool calls, handles handoffs, and returns only when there is a final answer and no more tool work to perform [2]. That loop is the heart of an agent.

At the simplest level:

goal
  -> prepare context
  -> call model
  -> parse model output
  -> execute allowed action
  -> observe result
  -> update state
  -> repeat or stop

The runtime is responsible for the parts that should not be left to vibes:

  • preserving state across turns
  • validating tool calls
  • enforcing approvals
  • handling timeouts and retries
  • streaming progress
  • saving resumable state when a run pauses
  • stopping when the environment proves the task is complete

The Hitchhiker guide calls this surrounding infrastructure the agent harness: the layer that turns an LLM from a stateless text generator into a stateful, goal-directed system with tool use, memory retrieval, observability, and error handling [8].

Level 2: Concepts

The agent loop has five core moments.

Moment Runtime Question Common Bug
Prepare What instructions, state, tools, and context should the model see? prompt stuffed with stale or irrelevant context
Decide What did the model ask to do? ambiguous final answer vs action
Act Is the action valid, authorized, and executable? tool called with invalid arguments or hidden side effects
Observe What happened in the outside world? tool result lost, summarized incorrectly, or not persisted
Stop Is the task actually done? model says "done" without objective evidence

The loop does not need to be complicated, but it does need to be explicit.

State

State is the difference between a one-shot response and a run.

OpenAI lists several continuation strategies: replay-ready history, SDK-backed sessions, managed conversation IDs, and lightweight continuation with a previous response ID [2]. Those are implementation choices. The design choice is deciding what state the runtime owns.

Useful state often includes:

  • conversation history
  • current objective and subtasks
  • tool results and artifacts
  • approval state
  • retries and errors
  • trace identifiers
  • cost and token budget

OpenAI's result surface exposes final output, replay-ready history, the last agent, response IDs, interruptions, and resumable state [3]. That is a clue: a production loop should return more than text. It should return enough structure for the application to resume, inspect, audit, or recover.

Interruptions

Not every run ends in an answer. Some runs should pause.

Human review is not a decorative feature. OpenAI describes guardrails and human-in-the-loop approvals as controls that can validate input, output, tool behavior, and side-effecting actions [4]. When the agent needs approval, the runtime should save the pending action and return an interruption, not invent a final answer.

For example:

{
  "status": "interrupted",
  "reason": "approval_required",
  "pending_tool_call": {
    "name": "send_customer_email",
    "arguments": {
      "case_id": "case_123",
      "template": "refund_denied"
    }
  }
}

That object is not just UI metadata. It is how the system avoids losing the run.

Latency Mode

Runtime design also decides how work reaches the user.

For conversational tasks, streaming matters because the application can process early tokens while the model continues generating [5]. For long-running tasks, background mode matters because deep research, coding, or multi-step tool work can take minutes, and an asynchronous run can be polled until it reaches a terminal state [6].

The same agent may need both modes:

  • stream visible progress for short interactive steps
  • move long work to the background
  • resume after approval or connectivity loss
  • save artifacts as they appear

An agent loop that works only for a single synchronous request is usually a prototype, not a product.

Level 3: Engineering Detail

A minimal runtime loop can be expressed as a small state machine:

READY
  -> RUNNING
  -> WAITING_FOR_TOOL
  -> WAITING_FOR_APPROVAL
  -> RUNNING
  -> COMPLETED | FAILED | CANCELLED

The important move is to persist the state transition, not only the message transcript.

Loop Skeleton

def run_agent(run_id, user_input):
    state = load_state(run_id)
    state.add_user_input(user_input)

    while state.turns < state.max_turns:
        model_input = build_model_input(state)
        result = call_model(model_input, tools=state.allowed_tools)
        state.record_model_result(result)

        if result.final_output:
            checked = validate_final_output(result.final_output, state)
            if checked.ok:
                state.complete(checked.output)
                return state.to_response()
            state.add_repair_instruction(checked.error)
            continue

        for call in result.tool_calls:
            decision = authorize_tool_call(call, state)
            if decision.requires_approval:
                state.pause_for_approval(call)
                return state.to_response()
            if not decision.allowed:
                state.record_blocked_call(call, decision.reason)
                continue

            observation = execute_tool(call)
            state.record_observation(call, observation)

        if state.should_stop_for_budget():
            state.fail("budget_exceeded")
            return state.to_response()

    state.fail("max_turns_exceeded")
    return state.to_response()

This is deliberately boring. Boring is good. The model can be flexible; the runtime should be legible.

Runtime Contracts

Contract What It Should Specify
Input contract message roles, instruction hierarchy, context sources, user identity
Tool contract schema, timeout, permission tier, side effects, retry behavior
State contract persisted fields, resumability, retention, privacy policy
Approval contract who can approve, what is shown, what is resumed
Observation contract how tool results become model-visible evidence
Stop contract objective completion criteria, max turns, max cost, failure states

The stop contract is often the weakest part. "The answer sounds good" is not a stop condition. Better stop conditions are:

  • tests pass
  • file was written
  • ticket status changed
  • database row exists
  • approval request was created
  • source evidence is sufficient
  • human chose to stop

Failure Handling

Agent runs fail in recurring ways.

Failure Runtime Response
invalid tool arguments ask model to repair arguments or block the call
tool timeout retry with backoff, try alternate tool, or pause
stale context refresh source, prefer fresh tool result over memory
looping enforce max turns and detect repeated actions
unsafe action require approval or deny tool execution
incomplete evidence retrieve more, ask user, or escalate
final output fails schema repair through structured validation

Anthropic's guidance is useful here: add complexity only when it demonstrably improves outcomes, and keep the system transparent enough that developers can inspect what happened [7]. That is especially true for runtime recovery logic. Hidden magic is where debugging goes to die.

Current Landscape

Agent runtimes are converging on a shared shape: a model loop, typed tools, explicit state, tracing, guardrails, streaming, and resumability.

OpenAI's Agents platform surfaces this through runner loops, result objects, state strategies, guardrails, human review, streaming responses, and background mode [2][3][4][5][6]. The details differ by SDK or API, but the runtime problem is the same: coordinate model decisions with tools and state.

The ReAct paper remains conceptually important because it framed the interleaving of thought, action, and observation [1]. Current systems add engineering layers around that pattern: permissions, approval, persistence, telemetry, and user experience.

The Hitchhiker guide places the loop in the larger harness layer, alongside context-window management, memory, communication, and observability [8]. That is the right mental model for production: the loop is not just a prompt pattern; it is runtime infrastructure.

Practical Takeaways

Design the loop before choosing the framework. Write down the states, transitions, approval points, and stop conditions.

Return structured run results, not just final text. The application needs history, artifacts, interruptions, trace IDs, and resumable state.

Treat tools as side-effecting APIs. Validate arguments, set timeouts, capture observations, and attach permission tiers.

Use streaming for user-visible progress and background mode for long work. Do not force every agent task through one synchronous request.

Make completion objective. The agent is done when the environment or evaluator proves it is done.

Keep runtime logic inspectable. When a run fails, the trace should show which instruction, tool call, guardrail, or state transition caused the failure.

References

[1] "ReAct: Synergizing Reasoning and Acting in Language Models." Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. https://arxiv.org/abs/2210.03629. Published/updated: 2023-03-10. Accessed: 2026-07-06. Used for: interleaved reasoning, acting, observations, exception handling, and the conceptual origin of the modern agent loop.

[2] "Running agents." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/running-agents. Published/updated: not listed. Accessed: 2026-07-06. Used for: runner loop, tool execution, handoffs, continuation strategies, and state management.

[3] "Results and state." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/results. Published/updated: not listed. Accessed: 2026-07-06. Used for: result surfaces, final output, history, response IDs, interruptions, and resumable state.

[4] "Guardrails and human review." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/guardrails-approvals. Published/updated: not listed. Accessed: 2026-07-06. Used for: input, output, and tool guardrails plus human-in-the-loop approvals.

[5] "Streaming API responses." OpenAI API documentation. https://developers.openai.com/api/docs/guides/streaming-responses. Published/updated: not listed. Accessed: 2026-07-06. Used for: server-sent events, early response processing, and interactive runtime behavior.

[6] "Background mode." OpenAI API documentation. https://developers.openai.com/api/docs/guides/background. Published/updated: not listed. Accessed: 2026-07-06. Used for: asynchronous long-running tasks, polling, response storage windows, and timeout avoidance.

[7] "Building effective agents." Anthropic. https://www.anthropic.com/engineering/building-effective-agents. Published/updated: 2024-12-19. Accessed: 2026-07-06. Used for: start-simple guidance, agent autonomy tradeoffs, ground-truth observations, and stopping conditions.

[8] "The Hitchhiker's Guide to Agentic AI: From Foundations to Systems." Haggai Roitman. Local source: agent-101/The Hitchhiker’s Guide to Agentic AI/book.tex. Published/updated: 2026. Accessed: 2026-07-06. Used for: agent harness framing, runtime infrastructure, context management, memory, observability, and error handling.