AI Agents 101 | 02. From Prompt Engineering to Agent Systems Engineering
From Prompt Engineering to Agent Systems Engineering
Prompt engineering is still useful. It is just no longer the main bottleneck.
The old mental model was: write a better prompt, get a better answer. That still works for single-turn tasks. But when an AI system can inspect a repository, call tools, run tests, read logs, open pull requests, ask other agents for review, persist state, and wake up again on a schedule, the engineering problem changes. You are no longer just shaping a model response. You are designing the environment, harness, and loop that let the model do reliable work.
The shift is from "what should I ask the model?" to "what system should repeatedly turn goals into verified progress?"
Level 1: Fundamentals
Prompt engineering means shaping the input so a model is more likely to produce the output you want. OpenAI's prompt engineering guide frames prompts as the way developers ask models to generate text, code, structured JSON, mathematical output, or prose, and it highlights patterns such as examples, clear instructions, and prompt caching for reused context [1].
That is the right starting point. A model is not a mind reader. If you want a customer-support classifier to return only Hardware, Software, or Other, you should say that. If you want citations, specify the citation format. If you want Markdown, say where Markdown is appropriate. Prompting is the interface between human intent and model behavior.
But agents add new moving parts. A useful agentic system may have:
- prompts that define the role and task
- context that supplies current facts and project rules
- tools that let the model act
- a harness that runs and constrains one agent execution
- loops that rediscover work, retry, verify, schedule, and persist state
- evaluations that measure behavior
- safety controls that limit damage
Cobus Greyling's "Loop Engineering" draws a clean distinction between these layers: the harness equips a single run, while the loop repeatedly prompts agents, spawns helpers, persists state, and decides what should happen next [2]. OpenAI's "Harness engineering" article makes the same shift concrete in software engineering: the team's job became designing environments, specifying intent, and building feedback loops so Codex agents could do reliable work [3].
The beginner mistake is to keep polishing the prompt after the problem has moved somewhere else. If the model lacks the current policy, the fix is context. If it calls the wrong API, the fix is tool design. If it cannot observe the app, the fix is harness engineering. If it needs to check CI every morning, triage failures, assign work, verify changes, and remember what happened last time, the fix is loop engineering.
Level 2: Concepts
A helpful mental model is:
prompt engineering = shape one model call
context engineering = shape what the model can know
tool engineering = shape what the model can do
harness engineering = shape one reliable agent run
loop engineering = shape repeated goal-seeking runs over time
agent systems engineering = compose all of the above
Prompt engineering answers: What should the model do in this moment?
Context engineering answers: What information should the model see, in what order, at what level of detail, with what provenance?
Tool engineering answers: What actions can the model take, and how obvious, typed, constrained, and recoverable are those actions?
Harness engineering answers: What execution environment makes one run legible, observable, testable, and enforceable?
Loop engineering answers: What keeps discovering work, assigning it, verifying it, persisting state, and deciding the next action?
Eval engineering answers: How do we know the system is getting better rather than merely sounding better?
Safety engineering answers: What should the system refuse, constrain, review, sandbox, or escalate?
Anthropic's "Building effective agents" gives a useful bridge from single prompts to systems. It describes workflows as systems where LLMs and tools follow predefined code paths, and agents as systems where the LLM dynamically directs its process and tool use [4]. It also recommends starting with the simplest solution possible, because agentic systems often trade latency and cost for better task performance [4].
That advice matters even more once harnesses and loops enter the picture. A harness is leverage when it makes work verifiable. A loop is leverage when it repeatedly converts state into useful next actions. Both are liabilities when they simply automate confusion.
The stack is easier to reason about if each layer has a contract:
| Layer | Contract |
|---|---|
| Prompt | express the local task and success criteria |
| Context | supply relevant, current, permissioned information |
| Tools | expose safe, typed actions with clear consequences |
| Harness | make one run observable, isolated, testable, and reviewable |
| Loop | decide when to run, what to run, how to verify, and when to stop |
| Evals | measure behavior against scenarios, not vibes |
| Safety | prevent, detect, and escalate unsafe behavior |
The work of agent systems engineering is turning those contracts into code, docs, tests, runtime boundaries, and feedback loops.
Level 3: Engineering Detail
The layers of agent systems engineering show up in different failure modes.
| Layer | What You Design | Common Failure If Neglected |
|---|---|---|
| Prompt | role, instructions, examples, output format | vague or inconsistent responses |
| Context | retrieved docs, project rules, memory, source ordering | stale, irrelevant, or missing facts |
| Tools | schemas, names, descriptions, permissions, errors | wrong calls, invalid arguments, unsafe actions |
| Harness | runtime, isolation, observability, review, app legibility | invisible failures or unverifiable work |
| Loop | schedule, state, task discovery, maker/checker split, stopping policy | endless automation, duplicated work, unattended mistakes |
| Evals | datasets, graders, scenarios, metrics | no evidence that changes helped |
| Safety | guardrails, sandboxing, human review, abuse controls | prompt injection, data leaks, excessive agency |
Prompt engineering
Prompting still matters because it defines intent. Modern prompt guidance for agentic work often includes persistence, tool-use, and planning reminders: continue until the task is actually resolved, use tools instead of guessing, and plan or reflect around tool calls when useful [5].
But prompts should not carry every responsibility. A giant prompt that includes API docs, policies, examples, product data, safety rules, and workflow state becomes hard to inspect and easy to break. Past a certain scale, the prompt should become a clean interface to a better-engineered context, harness, and loop.
A good agent prompt should be closer to a control header than a knowledge dump:
role: support operations agent
goal: resolve the current ticket or escalate with evidence
success: ticket has a documented resolution, approved action, or escalation reason
tools: read_ticket, search_policy, stage_refund, add_case_note, escalate_case
constraints: do not move money; do not email the customer; approval required for refunds
stop: final answer only after a note, approval request, or escalation has been written
The prompt states intent and boundaries. The system around it supplies the policy, state, tools, traces, and approval mechanism.
Context engineering
Context engineering is the work of deciding what information enters the model's context window. That includes retrieved documents, project instructions, memory, examples, source snippets, prior tool results, and user state.
OpenAI's harness article gives a practical version of this rule: repository knowledge became the system of record, with a short AGENTS.md acting as a map and deeper documentation living in structured files that agents can discover progressively [3]. The article also warns that a monolithic instruction file can crowd out the task, rot quickly, and become hard to verify [3].
This is also visible outside one company. A 2026 arXiv paper on context engineering for open-source software agents studied AGENTS.md files across 466 repositories and found that projects use these files to provide project structure, code style, build and test workflows, policies, and other agent-relevant context [6].
The practical lesson: context is not just "more text." Good context has relevance, order, freshness, provenance, and boundaries. Bad context creates its own failure modes. Too little context makes the model guess. Too much context buries the signal. Stale context teaches the model the wrong world.
A practical context pipeline looks like this:
task
-> classify domain and risk
-> load stable instructions
-> retrieve task-specific docs
-> filter by permission and freshness
-> rank by relevance and authority
-> summarize only when the raw source is too large
-> attach citations, paths, versions, or timestamps
For code agents, this often becomes a repository-local knowledge architecture: a short root instruction file, domain docs, architecture docs, command references, generated schema docs, active plans, completed plans, and quality or security checklists. The key design principle is progressive disclosure: the agent starts with a small map, then loads deeper sources only when needed.
Tool engineering
Tools turn language into action. OpenAI's tools guide describes platform tools such as function calling, web search, remote MCP servers, skills, shell, computer use, file search, and tool search; the model can decide whether to use configured tools, and advanced workflows can load more tool definitions as needed [7].
Tool engineering is more than exposing an API. It means designing an "agent-computer interface." Anthropic argues that tool definitions deserve as much prompt-engineering attention as the overall prompt, including examples, edge cases, input requirements, clear boundaries, and parameter names that make the intended use obvious [4].
Good tools are narrow and boring. A tool called refund_order(order_id, reason, amount_cents) is easier to supervise than a tool called do_customer_action(payload). A tool that returns structured errors is easier to recover from than a tool that returns a paragraph. A tool that requires approval before money moves is safer than one that hides the consequence behind a friendly function name.
Design each tool around five fields:
| Field | Why It Matters |
|---|---|
| name | tells the model when to use it |
| description | defines consequence and boundary |
| input schema | constrains arguments before runtime |
| output schema | makes recovery and evaluation possible |
| permission tier | decides whether the call can run automatically |
A useful tool output should look more like an event than prose:
{
"ok": false,
"error_code": "POLICY_NOT_FOUND",
"message": "No refund policy matched region=AU and product_type=subscription.",
"recoverable": true,
"suggested_next_tools": ["search_policy", "escalate_case"]
}
That shape gives the model something to reason over. It also gives the harness something to trace, test, and evaluate.
Hugging Face's smolagents work adds another useful angle: not every agent action has to be expressed as a JSON tool call. Code-agent patterns let the model write actions as code, which can improve composability for calculations, loops, and intermediate variables, but they also make sandboxing and execution security more important [8].
Harness engineering
The harness is the program and environment around one agent run. It gives the agent tools, context, isolation, observability, and a path to verification.
The Hitchhiker guide puts the same idea in systems language: an agent harness is runtime infrastructure that wraps an LLM into a stateful, goal-directed agent, separating concerns such as reasoning, execution, memory, communication, and observability [15].
OpenAI's harness engineering case study is useful because it is not hypothetical. The team describes building an internal beta with no manually written code: Codex generated application logic, tests, CI, documentation, observability, and internal tooling; humans steered while agents executed [3]. The important lesson is not "never write code manually." The lesson is that human work moved up a level: making the environment legible and enforceable for the agent.
In that case study, when Codex struggled, engineers did not simply ask it to try again. They asked what capability was missing and how to make that capability visible and enforceable [3]. Examples included making the app bootable per worktree, wiring browser control into the agent runtime, exposing logs, metrics, and traces, and creating repository-embedded skills so Codex could gather context without humans copying and pasting [3].
That is harness engineering. It is the difference between:
"fix startup latency"
and:
run isolated app -> measure startup -> inspect traces -> patch code -> rerun workload -> verify threshold
The prompt names the goal. The harness makes the goal observable.
A good harness captures a run record:
{
"run_id": "run_2026_06_17_001",
"task": "reduce startup latency below 800ms",
"environment": "worktree:feature-startup-latency",
"allowed_tools": ["shell", "browser", "logs", "metrics", "apply_patch"],
"approval_policy": "ask_before_external_side_effects",
"observations": [
{ "type": "metric", "name": "startup_ms", "value": 1240 },
{ "type": "trace", "span": "load_user_config", "duration_ms": 620 }
],
"verification": {
"command": "npm test",
"result": "passed",
"metric": "startup_ms",
"threshold": 800
}
}
That record is not just logging. It is the substrate for review, evals, debugging, retries, and loop state.
Loop engineering
Loop engineering begins when you stop being the person who manually prompts the agent at every step.
Greyling defines loop engineering as designing the system that discovers work, hands tasks to agents or sub-agents, verifies results, persists state, and decides the next action on a schedule or until a goal is met [2]. That is one level above the harness. The harness makes a run capable. The loop decides when and why runs happen.
A useful loop has a few boring parts:
- a heartbeat: a schedule, trigger, queue, or goal condition
- a work finder: CI failure, ticket, bug report, stale doc, metric regression, or inbox item
- a maker/checker split: one agent does the work, another verifies it
- isolation: worktrees, sandboxes, or dedicated sessions so parallel work does not collide
- durable state: what is active, what was tried, what failed, and what needs a human
- stop and escalation rules: what counts as done, blocked, risky, or too expensive
Greyling emphasizes that unattended loops create real economics and verification problems: frequent cadences and sub-agents multiply cost, and a verifier improves confidence but does not remove the need for human judgment [2]. That is the sober version of loop engineering. It is not "let the agent do everything." It is "replace repetitive prompting with a small control system that still knows when to hand back to you."
A loop state file can be very small:
{
"goal": "keep main branch green",
"cadence": "every_30_minutes",
"last_run_at": "2026-06-17T08:30:00Z",
"active_items": [
{
"id": "ci-4821",
"status": "needs_fix",
"evidence": "unit test failure in billing/refunds",
"assigned_run": "run_2026_06_17_001",
"human_needed": false
}
],
"stop_rules": {
"max_cost_usd_per_day": 20,
"max_attempts_per_item": 3,
"escalate_on": ["security", "data_loss", "unclear_requirement"]
}
}
Without state, every loop run starts from scratch. With state, the system can avoid duplicate work, remember failed attempts, control spend, and know when to ask for help.
Eval and observability engineering
Once the system has loops and tools, you need traces and evals.
Tracing records what happened during a run. OpenAI's Agents SDK observability docs say tracing is built into the SDK and can emit structured records of model calls, tool calls, handoffs, guardrails, and custom spans, giving developers an end-to-end record before formalizing evals [9].
Evals measure whether the system meets expectations. OpenAI's evals guide describes evaluation workflows built from datasets and testing criteria, then shows how those criteria compare model output against expected behavior [10].
Anthropic's evals guidance makes the agent-specific problem sharper: agents operate over many turns, call tools, modify environment state, and adapt based on intermediate results, so mistakes can propagate and compound [11]. A single-turn output eval is not enough. The evaluation has to measure the model, the harness, the tools, the environment, and the final state they produce together.
Anthropic uses a useful vocabulary for agent evals [11]:
| Term | Meaning |
|---|---|
| task | one test case with inputs and success criteria |
| trial | one attempt at a task; multiple trials reduce noise from non-determinism |
| grader | code, model, or human logic that scores some aspect of behavior |
| transcript | the trace or trajectory: messages, tool calls, reasoning, intermediate results |
| outcome | the final environment state, such as a database row, file, PR, or booking |
| evaluation harness | infrastructure that runs tasks, records steps, grades outputs, and aggregates results |
| evaluation suite | a collection of tasks targeting a capability or regression area |
This distinction between transcript and outcome matters. A support agent might say "your refund is processed," but the outcome is whether the refund record exists with the right status. A coding agent might write a persuasive summary, but the outcome is whether tests pass, security checks pass, and the intended code path changed.
For agents, evals should measure the whole behavior, not just the final answer:
| Eval Dimension | Example Check |
|---|---|
| source use | did the agent cite the current policy rather than memory? |
| tool choice | did it call lookup before action? |
| argument quality | did schemas, units, IDs, and regions match expectations? |
| approval behavior | did it pause before irreversible actions? |
| recovery | did it handle a failed tool call without inventing success? |
| stop condition | did it stop only after objective verification? |
| cost and latency | did it stay under budget and time limits? |
The grader design should match the task. Anthropic groups common graders into code-based, model-based, and human graders: deterministic tests, static analysis, outcome checks, tool-call checks, transcript analysis, rubric-based LLM judges, pairwise comparisons, expert review, spot checks, and A/B tests [11]. Use deterministic graders wherever possible because they are cheap, objective, reproducible, and easy to debug. Use LLM graders when the task is open-ended or subjective. Use human graders to calibrate model graders and validate quality where domain judgment matters.
That gives a more concrete eval design pattern:
task:
id: "support-refund-policy-001"
input: "Customer asks for a refund on a late subscription renewal."
success_criteria:
- "Agent verifies identity before taking account action."
- "Agent grounds refund decision in current policy."
- "Agent stages the refund but does not move money without approval."
graders:
- type: state_check
expect:
refund: {status: staged}
approval_request: {status: pending}
- type: tool_calls
required:
- {tool: verify_identity}
- {tool: search_policy}
- {tool: stage_refund}
forbidden:
- {tool: process_refund}
- type: llm_rubric
rubric: "Was the explanation clear, empathetic, and grounded in retrieved policy?"
tracked_metrics:
- n_turns
- n_tool_calls
- total_tokens
- latency_ms
- estimated_cost_usd
Anthropic also separates capability evals from regression evals. Capability evals ask "what can this agent do well?" and should include tasks the agent currently struggles with. Regression evals ask "does the agent still handle the tasks it used to handle?" and should stay close to 100% pass rate [11]. As a capability eval saturates, it can graduate into a regression suite.
Non-determinism needs its own measurement. Anthropic highlights two useful metrics: pass@k, the chance an agent succeeds at least once across k attempts, and pass^k, the chance it succeeds in all k attempts [11]. The first is useful when one working solution is enough. The second is more important for customer-facing agents where users expect consistent behavior every time.
Good evals are also maintained artifacts, not one-off scripts. Anthropic recommends starting early with 20-50 tasks drawn from real manual checks or failures, writing unambiguous tasks with reference solutions, testing both positive and negative cases, isolating trials in clean environments, preferring outcome grading over brittle path grading, reading transcripts to verify the grader is fair, watching for eval saturation, and treating eval suites like unit tests that product and domain experts help maintain [11].
A pretty final paragraph can hide a bad trajectory. Loop engineering makes this more urgent. If a loop runs on a schedule, a small bug can repeat. If it spawns sub-agents, a bad assumption can fan out. If it auto-opens pull requests, weak verification becomes noise. Observability and evals are not polish; they are the brakes, dashboard, and regression suite.
Safety engineering
Safety engineering is what makes the system robust under pressure. Guardrails are one mechanism. OpenAI's current guardrails and human review docs describe input guardrails, output guardrails, tool guardrails, and approval boundaries for sensitive actions [12].
Security needs its own lens too. The OWASP Top 10 for LLM Applications lists risks such as prompt injection, insecure output handling, sensitive information disclosure, insecure plugin design, excessive agency, and overreliance [13]. These map directly onto agent systems: agents read untrusted content, call tools, pass data across boundaries, and may be trusted too much because their prose sounds confident.
This is why safety cannot be a paragraph at the end of the prompt. It needs to be built into:
- tool permission tiers
- sandboxed execution
- source isolation for untrusted content
- approval gates for external side effects
- structured logging and audit trails
- evals for unsafe requests and prompt injection
- deployment gates for high-risk workflows
- escalation paths when the model is uncertain
The hard part is not writing "be safe." The hard part is making unsafe behavior difficult, visible, and reviewable.
Current Landscape
The direction of travel is clear: AI engineering is becoming systems engineering.
OpenAI's agent tooling emphasizes APIs, built-in tools, orchestration, tracing, and evaluation [14]. Anthropic's production advice emphasizes simple composable patterns, transparency, and careful agent-computer interface design [4]. Anthropic's eval guidance adds an important operational point: agent quality depends on eval harnesses, outcome checks, transcript review, grader calibration, and maintained suites, not just prompt tweaks [11]. Context engineering is becoming visible enough that researchers are studying real-world context files such as AGENTS.md across open-source repositories [6].
The newer terms sharpen the stack:
- context engineering makes the right knowledge available
- tool engineering makes actions legible and constrained
- harness engineering makes one run observable and enforceable
- loop engineering makes repeated runs discover, act, verify, and persist
- eval and safety engineering keep the system from confusing activity with progress
The old workflow was:
write prompt -> inspect answer -> write next prompt
The agent-systems workflow is:
encode intent -> expose context -> design tools -> build harness -> run agent -> trace behavior -> evaluate outcome -> persist state -> schedule next loop
The second workflow is slower at first. It is also the path to software you can debug, test, and trust.
Practical Takeaways
Start with a prompt, but do not stay there too long. If the same bug repeats, classify it by layer: prompt, context, tool, harness, loop, eval, or safety.
Keep prompts short enough to inspect. Move durable facts into context sources, actions into tools, and control policy into the harness.
Design tools as product APIs. Give them typed parameters, clear names, precise descriptions, examples, explicit permissions, structured errors, and approval behavior.
Design the harness before asking for autonomy. Give the agent a way to inspect the app, run tests, read logs, measure outcomes, and verify its own work.
Design loops only after the harness can produce trustworthy evidence. A schedule without reliable verification is just automated noise.
Persist state in boring files or systems. The loop needs to know what it is working on, what it tried, what failed, and what requires a human.
Build traces before you need them. You cannot debug an agent from the final answer alone.
Write evals around realistic scenarios, not just isolated outputs. Include failure cases, stale context, tool errors, ambiguous user requests, unsafe requests, and repeated loop execution. Track both capability evals and regression evals, and read transcripts so the numbers do not drift away from reality.
Keep humans at the judgment points. Loops can discover, assign, attempt, verify, and report. Humans still own taste, risk, priorities, and accountability.
The core skill is no longer clever wording. It is designing the world the model operates in, and the control system that decides what happens next.
References
[1] "Prompt engineering." OpenAI API documentation. https://developers.openai.com/api/docs/guides/prompt-engineering. Published/updated: not listed. Accessed: 2026-07-05. Used for: prompt engineering fundamentals, examples, clear instructions, structured outputs, and prompt caching framing.
[2] "Loop Engineering." Cobus Greyling. https://cobusgreyling.substack.com/p/loop-engineering. Published/updated: 2026-06-09. Accessed: 2026-07-05. Used for: loop engineering definition, harness-vs-loop distinction, scheduling, maker/checker split, memory, and loop risks.
[3] "Harness engineering: leveraging Codex in an agent-first world." OpenAI. https://openai.com/index/harness-engineering/. Published/updated: 2026-02-11. Accessed: 2026-07-05. Used for: harness engineering, agent legibility, repository knowledge, observability, worktrees, constraints, and feedback loops.
[4] "Building effective agents." Anthropic. https://www.anthropic.com/engineering/building-effective-agents. Published/updated: 2024-12-19. Accessed: 2026-07-05. Used for: workflow-vs-agent distinction, composable agent patterns, simplicity guidance, and tool-interface design.
[5] "Prompt guidance." OpenAI API documentation. https://developers.openai.com/api/docs/guides/prompt-guidance. Published/updated: not listed. Accessed: 2026-07-05. Used for: agentic prompting patterns such as persistence, tool use, planning, and stop-condition guidance.
[6] "Context Engineering for AI Agents in Open-Source Software." Seyedmoein Mohsenimofidi, Matthias Galster, Christoph Treude, and Sebastian Baltes. https://arxiv.org/abs/2510.21413. Published/updated: 2026-02-05. Accessed: 2026-07-05. Used for: context engineering in real open-source agent configuration files.
[7] "Using tools." OpenAI API documentation. https://developers.openai.com/api/docs/guides/tools. Published/updated: not listed. Accessed: 2026-07-05. Used for: current OpenAI tool categories and tool-use behavior.
[8] "Introducing smolagents, a simple library to build agents." Hugging Face. https://huggingface.co/blog/smolagents. Published/updated: 2024-12-31. Accessed: 2026-07-05. Used for: code-agent framing, agency spectrum, and sandboxing implications for action execution.
[9] "Integrations and observability." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/integrations-observability. Published/updated: not listed. Accessed: 2026-07-05. Used for: tracing, structured run records, model calls, tool calls, handoffs, guardrails, custom spans, and workflow observability.
[10] "Working with evals." OpenAI API documentation. https://developers.openai.com/api/docs/guides/evals. Published/updated: not listed. Accessed: 2026-07-05. Used for: evaluation workflow, criteria, datasets, and iteration guidance.
[11] "Demystifying evals for AI agents." Anthropic. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents. Published/updated: 2026-01-09. Accessed: 2026-07-05. Used for: task/trial/grader/transcript/outcome vocabulary, grader types, capability-vs-regression evals, pass@k and pass^k, eval harness design, transcript review, and eval maintenance.
[12] "Guardrails and human review." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/guardrails-approvals. Published/updated: not listed. Accessed: 2026-07-05. Used for: input, output, and tool guardrails, approval boundaries, and human review for sensitive actions.
[13] "OWASP Top 10 for Large Language Model Applications." OWASP Foundation. https://owasp.org/www-project-top-10-for-large-language-model-applications/. Published/updated: 2025. Accessed: 2026-07-05. Used for: LLM application security risks including prompt injection, insecure output handling, excessive agency, and overreliance.
[14] "New tools for building agents." OpenAI. https://openai.com/index/new-tools-for-building-agents/. Published/updated: 2025-03-11. Accessed: 2026-07-05. Used for: Responses API, built-in tools, Agents SDK, orchestration, and observability as agent-building blocks.
[15] "The Hitchhiker's Guide to Agentic AI: From Foundations to Systems." Haggai Roitman. Local source: agent-101/The Hitchhiker’s Guide to Agentic AI/book.tex. Published/updated: 2026. Accessed: 2026-07-05. Used for: agent harness framing as runtime infrastructure and separation of reasoning, execution, memory, communication, and observability concerns.
Enjoy Reading This Article?
Here are some more articles you might like to read next: