AI Agents 101 | 10. Evaluation and Observability
Evaluation and Observability
You cannot improve an agent you cannot see.
Agents are harder to evaluate than single-turn chat because they act over time. They call tools, change state, branch, recover, fail partially, ask for approval, and sometimes reach the right answer for the wrong reason. A final response alone is not enough evidence.
Evaluation asks whether the agent did the right thing. Observability shows what happened.
Level 1: Fundamentals
OpenAI's agent-evals docs describe the core pieces: traces capture model, tool, guardrail, and handoff activity; graders score outputs or traces; datasets and eval runs make measurement repeatable [1].
Anthropic's agent-evals guidance makes the same point from another angle: agent evaluations involve tasks, trials, graders, transcripts or traces, outcomes, evaluation harnesses, and suites [3]. It also emphasizes that agent mistakes can propagate because agents use tools across many turns and modify state [3].
Observability is the runtime side. OpenAI's integration docs describe tracing as an end-to-end record of runs before deeper evaluation [2]. OpenTelemetry defines observability tooling around telemetry data such as traces, metrics, and logs, with vendor-neutral APIs and conventions [5].
The Hitchhiker guide treats evaluation environments and observability as part of the agent system, not a late testing activity [7].
Level 2: Concepts
Agent evaluation has three main targets:
| Target | Question | Example |
|---|---|---|
| Output | Did the final answer satisfy the user? | correct answer, clear citations, safe wording |
| Trajectory | Did the agent take acceptable steps? | used allowed tools, did not loop, asked approval |
| Outcome | Did the external state become correct? | tests pass, ticket updated, booking exists |
Single-turn evals usually focus on output. Agent evals need all three.
Traces
A trace is the story of a run in machine-readable form:
- user input
- model calls
- tool calls
- tool observations
- guardrail decisions
- handoffs
- approvals
- retries
- errors
- final output
- timing and token usage
Without traces, debugging becomes anecdotal. With traces, failures can be grouped: bad retrieval, wrong tool choice, missing permission, schema failure, hallucinated citation, overlong loop, or weak final synthesis.
Graders
Anthropic divides graders into code-based, model-based, and human categories [3]. Each has a role.
| Grader Type | Strength | Weakness |
|---|---|---|
| Code-based | fast, objective, reproducible | limited nuance |
| Model-based | flexible, scalable for qualitative checks | non-deterministic, needs calibration |
| Human | expert judgment | slow and expensive |
The best suites often combine them. A coding-agent task might run tests, run static analysis, check tool trajectory, and use a rubric for maintainability.
Capability vs Regression
Capability evals ask: can the agent solve hard tasks it currently struggles with?
Regression evals ask: did we break behavior that used to work?
Both matter. Capability suites help climb the hill. Regression suites protect the base camp.
Red Teaming
OpenAI's red-teaming guidance frames adversarial testing as a complement to evals, especially for misuse, failure modes, and high-risk interactions [4]. For agents, red teaming should include:
- prompt injection through retrieved documents
- malicious tool outputs
- unsafe plugin behavior
- privacy boundary tests
- approval bypass attempts
- loop/cost abuse
- misleading citations
Level 3: Engineering Detail
Start by defining the event schema.
Trace Event
{
"trace_id": "tr_123",
"run_id": "run_456",
"span_id": "span_789",
"event_type": "tool_call",
"timestamp": "2026-07-06T10:00:00Z",
"agent": "support_agent",
"model": "gpt-5.4",
"tool": "lookup_order",
"arguments": {"order_id": "ord_123"},
"permission_decision": "allowed",
"duration_ms": 842,
"status": "success",
"tokens": {"input": 1800, "output": 250}
}
Every event does not need every field, but every run should be reconstructable.
Eval Task
task:
id: "refund_policy_edge_case_017"
input: "Can I get a refund for a subscription renewed 33 days ago?"
environment:
customer_id: "cust_123"
policy_version: "refund_policy_2026_04"
expected:
outcome: "escalation_created"
must_cite:
- "refund_policy_2026_04#late-renewals"
must_not:
- "promise_refund"
graders:
- type: "tool_call"
required: ["search_policy", "create_escalation"]
- type: "output_schema"
schema: "support_answer_v2"
- type: "model_rubric"
rubric: "clear_boundary_no_false_promise"
The task encodes success as behavior and outcome, not just wording.
Metrics
Track metrics at multiple levels:
| Metric | Why It Matters |
|---|---|
| task success rate | primary capability signal |
| regression pass rate | release safety |
| tool error rate | integration health |
| approval rate | risk and UX load |
| citation support rate | grounding quality |
| turns per task | loop efficiency |
| latency | user experience |
| cost per successful task | business viability |
| escalation rate | boundary clarity |
| user correction rate | field quality signal |
Cost per successful task is often more useful than raw cost. A cheap failed run is not cheap.
Failure Taxonomy
Label failures consistently:
retrieval_missing
retrieval_wrong_source
permission_denied
tool_schema_error
tool_runtime_error
unsafe_tool_request
bad_plan
loop_limit_exceeded
approval_missing
unsupported_claim
wrong_final_answer
ui_confusion
This taxonomy turns eval failures into roadmap data.
Current Landscape
Agent evaluation is becoming trace-centered.
OpenAI's agent-eval docs explicitly connect traces, graders, datasets, and eval runs [1]. Anthropic's evals post emphasizes transcripts, outcomes, harnesses, trial variance, and mixed grader types [3]. That is the current best practice: evaluate the run, not just the reply.
Observability is also becoming less optional. OpenAI exposes tracing around agent runs and MCP-backed integrations [2]. OpenTelemetry provides the broader vendor-neutral vocabulary for traces, metrics, logs, collectors, APIs, and semantic conventions [5].
Anthropic's broader agent guidance adds a product-engineering constraint: agents should stay simple and inspectable enough that teams can understand how ground-truth observations shaped the run [6].
Red teaming sits alongside evals. It is the adversarial pressure test for misuse and boundary failure [4].
Practical Takeaways
Instrument the agent before scaling it. You need traces, metrics, logs, and artifacts from the beginning.
Evaluate outputs, trajectories, and outcomes. A good final answer reached by unsafe steps is still a failure.
Use mixed graders. Code-based checks, model rubrics, and human review answer different questions.
Keep capability and regression suites separate. One helps you improve; the other keeps you from breaking what works.
Label failures consistently. A taxonomy turns messy traces into engineering priorities.
Red-team the full loop. Attack retrieval, tools, memory, approvals, and UI, not only final text.
References
[1] "Evals for AI agents." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agent-evals. Published/updated: not listed. Accessed: 2026-07-06. Used for: traces, graders, datasets, eval runs, repeatability, and agent-performance measurement.
[2] "Integrations and observability." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/integrations-observability. Published/updated: not listed. Accessed: 2026-07-06. Used for: tracing, MCP-backed agent capabilities, and inspecting runtime behavior.
[3] "Demystifying evals for AI agents." Anthropic. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents. Published/updated: 2026-01-09. Accessed: 2026-07-06. Used for: tasks, trials, graders, transcripts/traces, outcomes, eval harnesses, eval suites, and capability-vs-regression evaluation.
[4] "Red teaming." OpenAI API documentation. https://developers.openai.com/api/docs/guides/red-teaming. Published/updated: not listed. Accessed: 2026-07-06. Used for: adversarial testing, misuse probing, and security-risk evaluation.
[5] "What is OpenTelemetry?" OpenTelemetry documentation. https://opentelemetry.io/docs/what-is-opentelemetry/. Published/updated: not listed. Accessed: 2026-07-06. Used for: observability framework, traces, metrics, logs, vendor-neutral telemetry, APIs, collectors, and semantic conventions.
[6] "Building effective agents." Anthropic. https://www.anthropic.com/engineering/building-effective-agents. Published/updated: 2024-12-19. Accessed: 2026-07-06. Used for: agent autonomy, ground-truth observations, and the value of simple inspectable designs.
[7] "The Hitchhiker's Guide to Agentic AI: From Foundations to Systems." Haggai Roitman. Local source: agent-101/The Hitchhiker’s Guide to Agentic AI/book.tex. Published/updated: 2026. Accessed: 2026-07-06. Used for: evaluation environments, observability, harness design, failure logging, and production iteration.
Enjoy Reading This Article?
Here are some more articles you might like to read next: