Production Agent Architecture

A demo agent can be a prompt and a tool call.

A production agent is a service.

It has identity, state, queues, tool permissions, observability, evals, incident response, deployment controls, cost budgets, and user experience. It survives retries, approvals, timeouts, stale data, partial failures, and changing models.

The question is not "can the agent do the task once?" The question is "can the system do the task repeatedly, safely, observably, and economically?"

Level 1: Fundamentals

OpenAI's production best-practices docs frame the move to production around security, robust architecture, and high traffic [1]. Its deployment checklist highlights choices that matter for agent systems: stateful workflows, tool selection, reasoning effort, compaction, prompt caching, background mode, and real-time interaction [2].

The production architecture has to wrap the model loop with service infrastructure:

user/app
  -> API gateway
  -> agent service
  -> state store
  -> tool gateway
  -> queues/workers
  -> retrieval/data layer
  -> approval service
  -> observability/evals

OpenAI's current agent docs expose many of the primitives: runner loops, state strategies, result objects, interruptions, streaming, background mode, and guardrails [3][4][5].

The Hitchhiker guide calls out the production gap directly: reliability, observability, testability, deployment, and iteration are the hard parts after a prototype works [7].

Level 2: Concepts

A production agent needs clear boundaries.

Component Responsibility
API gateway authentication, rate limits, request routing
Agent service prompt assembly, loop control, model calls
State store sessions, run state, artifacts, approvals
Tool gateway schema validation, permission enforcement, execution
Retrieval layer source access, indexing, evidence packets
Queue/workers long-running tasks and retries
Approval service human review, resumes, audit records
Observability traces, metrics, logs, dashboards
Eval service regression and capability testing

Keep these responsibilities separate. If the model loop also owns auth, retrieval, side effects, and persistence, debugging becomes painful and security becomes fragile.

Stateful Runs

Agents often run longer than one request. OpenAI's background-mode docs call out long-running agents such as coding and research tasks, where asynchronous execution avoids timeouts and connectivity problems [5].

Stateful runs need:

  • run ID
  • user and tenant
  • agent version
  • current status
  • prior steps
  • pending approvals
  • artifacts
  • error state
  • resume token or continuation reference

Versioning

Version everything that can change behavior:

  • model
  • prompt/instructions
  • tool schema
  • tool implementation
  • retrieval corpus
  • memory policy
  • guardrails
  • eval suite
  • UI approval flow

When a run fails, you need to know which system produced it. When a new version ships, you need to compare it against the old one.

Cost and Latency

Agent cost is not just one model call. It includes:

  • model tokens
  • repeated turns
  • retrieval
  • tool calls
  • evaluator calls
  • retries
  • background workers
  • human review
  • engineering investigation

OpenAI's deployment checklist calls out reasoning effort, compaction, prompt caching, tool choices, and background mode as deployment levers [2]. Those are cost and latency levers as much as quality levers.

Level 3: Engineering Detail

A production run record should be boring and complete.

Run Table

CREATE TABLE agent_runs (
  run_id TEXT PRIMARY KEY,
  user_id TEXT NOT NULL,
  tenant_id TEXT NOT NULL,
  agent_name TEXT NOT NULL,
  agent_version TEXT NOT NULL,
  model TEXT NOT NULL,
  status TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  completed_at TIMESTAMP,
  trace_id TEXT,
  cost_cents INTEGER DEFAULT 0,
  token_input INTEGER DEFAULT 0,
  token_output INTEGER DEFAULT 0,
  error_code TEXT,
  resume_state JSONB
);

That schema is not glamorous. It is the difference between "the agent broke" and "run 123 failed after tool approval because policy lookup timed out."

Tool Gateway

All tools should pass through a gateway that can:

  • validate schema
  • enforce permissions
  • check risk tier
  • require approval
  • add idempotency keys
  • time out execution
  • redact sensitive outputs
  • emit trace events
model request -> tool gateway -> policy check -> tool execution -> observation packet

Do not let every tool invent its own safety and logging behavior.

Deployment Pipeline

draft prompt/tool change
  -> unit tests for tools
  -> offline eval suite
  -> red-team cases
  -> shadow traffic or canary
  -> monitor traces and metrics
  -> staged rollout
  -> regression watch

NIST's AI RMF is useful here because it frames trustworthy AI as a lifecycle concern across design, development, use, and evaluation [6]. Production architecture should make that lifecycle real.

Operational Failure Modes

Failure Architecture Response
model output changes after upgrade versioned evals and canary release
tool provider outage retry, fallback, degraded mode
approval delayed run pause, reminder, expiration
user disconnects background run and resumable state
cost spike budget enforcement and per-run limits
data corpus changes index versioning and retrieval evals
unsafe tool request tool gateway blocks and logs
incident investigation trace, run record, artifacts, approvals

Current Landscape

The modern platform direction is clear: agent features are becoming runtime features.

OpenAI's Responses/Agents stack emphasizes stateful workflows, built-in tools, guardrails, tracing, background mode, streaming, and result state [2][3][4][5]. That is a sign that production agents need platform support around the model, not only better prompts.

Anthropic's agent guidance gives the strategic caution: start simple, add complexity when it helps, and keep interfaces inspectable [8]. Production architecture should preserve that discipline even as the system grows.

The Hitchhiker guide's production-framework discussion is useful because it treats lifecycle, testing, deployment, observability, prompt versioning, failure logging, and iteration as first-class concerns [7].

Practical Takeaways

Build the agent as a service, not a prompt wrapper.

Persist run state and trace IDs. Every long-running or tool-using task should be inspectable and resumable.

Put tools behind a gateway. Authorization, validation, approvals, idempotency, and telemetry should be centralized.

Version prompts, tools, models, retrieval corpora, and eval suites. You cannot debug what you cannot identify.

Use background work for long tasks and streaming for visible progress. Different latency modes serve different user needs.

Release through evals and canaries. Production agent changes should be measured before and after deployment.

References

[1] "Production best practices." OpenAI API documentation. https://developers.openai.com/api/docs/guides/production-best-practices. Published/updated: not listed. Accessed: 2026-07-06. Used for: production-readiness framing, security, architecture, and high-traffic considerations.

[2] "Deployment checklist." OpenAI API documentation. https://developers.openai.com/api/docs/guides/deployment-checklist. Published/updated: not listed. Accessed: 2026-07-06. Used for: Responses API, reasoning effort, tool search, built-in tools, compaction, prompt caching, background mode, and real-time interaction.

[3] "Running agents." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/running-agents. Published/updated: not listed. Accessed: 2026-07-06. Used for: runner loop, tool execution, handoffs, and state strategies.

[4] "Results and state." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/results. Published/updated: not listed. Accessed: 2026-07-06. Used for: final output, history, response IDs, interruptions, and resumable state.

[5] "Background mode." OpenAI API documentation. https://developers.openai.com/api/docs/guides/background. Published/updated: not listed. Accessed: 2026-07-06. Used for: asynchronous long-running tasks, polling, timeout avoidance, and response lifecycle.

[6] "AI Risk Management Framework." National Institute of Standards and Technology. https://www.nist.gov/itl/ai-risk-management-framework. Published/updated: 2026-04-07. Accessed: 2026-07-06. Used for: AI lifecycle governance across design, development, use, and evaluation.

[7] "The Hitchhiker's Guide to Agentic AI: From Foundations to Systems." Haggai Roitman. Local source: agent-101/The Hitchhiker’s Guide to Agentic AI/book.tex. Published/updated: 2026. Accessed: 2026-07-06. Used for: production gap framing, lifecycle, testing, deployment, observability, failure logging, prompt versioning, and iteration.

[8] "Building effective agents." Anthropic. https://www.anthropic.com/engineering/building-effective-agents. Published/updated: 2024-12-19. Accessed: 2026-07-06. Used for: start-simple guidance, inspectable interfaces, and autonomy tradeoffs.