Safety, Security, and Governance

Agent safety is not one filter at the end of the conversation.

An agent reads untrusted content, chooses tools, calls external systems, writes state, and may act over many turns. That means safety has to be designed into the loop: before the model sees data, before tools run, before outputs reach users, and before side effects become real.

Security is about what can go wrong. Governance is about who is accountable when it does.

Level 1: Fundamentals

OWASP's Top 10 for Large Language Model Applications lists risks that map directly onto agent systems, including prompt injection, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and supply-chain vulnerabilities [1]. The list is useful because it does not treat LLM risk as just "bad answers." It includes tools, permissions, dependencies, and human trust.

OpenAI's guardrails docs divide controls into input guardrails, output guardrails, tool guardrails, and human-in-the-loop approvals [2]. That is the right shape for agents: different risks appear at different boundaries.

NIST's AI Risk Management Framework frames AI governance around improving trustworthiness across design, development, use, and evaluation [4]. For agentic systems, this translates into policies for capability scope, testing, monitoring, incident response, human oversight, and auditability.

The Hitchhiker guide treats safety as a cross-cutting concern across the agentic stack: RAG, memory, tools, harnesses, orchestration, protocols, and UI all create different failure surfaces [6].

Level 2: Concepts

Safety starts by admitting that agents have agency only because the system gives it to them.

Risk	Agent-Specific Version	Control
Prompt injection	retrieved text or tool output tells agent to ignore rules	instruction hierarchy and untrusted-data handling
Excessive agency	agent can take broad side-effecting actions	narrow tools, approvals, and scoped permissions
Sensitive disclosure	agent sees or reveals restricted data	ACL-aware retrieval and output redaction
Insecure plugin/tooling	installed capability exposes unsafe actions	plugin review, version pinning, permission gating
Overreliance	user accepts unsupported output	citations, uncertainty, and human checkpoints
Model DoS/cost abuse	loop or user request burns resources	budgets, rate limits, max turns, abuse detection

Prompt Injection

Prompt injection is especially dangerous for agents because the model reads untrusted text from web pages, files, emails, tickets, or tool outputs. Some of that text may be written to manipulate the agent.

The defense is not "write a stronger prompt" alone. Use layered controls:

• mark retrieved/tool content as untrusted data
• preserve instruction authority hierarchy
• prevent untrusted text from changing tool permissions
• restrict side-effecting tools
• validate outputs and tool arguments
• log suspicious instructions found in data

Excessive Agency

Excessive agency means the system gives the agent more ability to act than the task requires. A support agent that can read refund policy may not need to issue refunds directly. A coding agent may need to edit a branch but not deploy to production.

The safest capability is the narrowest one that completes the job.

Human Oversight

Human review should be attached to risk, not sprinkled randomly. Require approval when:

• money moves
• customer-visible messages are sent
• files are deleted
• production systems change
• sensitive data leaves a boundary
• the agent is uncertain but the cost of being wrong is high

OpenAI's human-review pattern lets runs pause when approval is needed, preserving state for later continuation [2].

Level 3: Engineering Detail

A useful safety design starts with a risk tier for each tool and workflow.

Risk Tiering

Tier	Example	Controls
0: read-only public	search public docs	log and cite
1: internal read	search company docs	identity, ACL, audit
2: reversible write	draft ticket note	sandbox, validation, undo path
3: external communication	send email or Slack message	preview, approval, audit
4: irreversible or regulated	refund, deploy, delete, legal decision	human approval, policy check, dual control

This table should live near tool definitions. The runtime should enforce it.

Guardrail Placement

user input
  -> abuse/risk classification
  -> context retrieval with ACLs
  -> model call
  -> tool-call validation
  -> approval if required
  -> tool execution
  -> output validation/redaction
  -> final answer with evidence

Input guardrails are useful because they can stop unsafe or expensive runs early [2]. Tool guardrails are useful because they protect the boundary where language becomes action. Output guardrails are useful because they protect the user and downstream systems.

Tool Authorization

{
  "tool": "send_customer_email",
  "risk_tier": 3,
  "allowed_roles": ["support_agent"],
  "requires_approval": true,
  "approval_payload": {
    "recipient": "customer@example.com",
    "subject": "Refund decision",
    "body_preview": "...",
    "source_citations": ["policy_refunds_2026_04#s3"]
  },
  "blocked_when": [
    "missing_policy_citation",
    "contains_internal_notes",
    "customer_id_mismatch"
  ]
}

Approval should show the actual action, not a vague summary.

Governance Records

For auditability, each meaningful run should record:

• user identity and role
• agent version
• model version
• tools available
• tool calls requested
• tool calls executed or denied
• approvals requested and granted
• source evidence used
• final output
• errors and retries
• trace ID

OpenAI's red-teaming guidance emphasizes adversarial probing for misuse, failures, and security risks [3]. Those tests become much more useful when traces expose what the agent actually did.

Current Landscape

The safety conversation has broadened from model output moderation to system risk.

OWASP's current LLM application risks include prompt injection, plugin/tool risk, excessive agency, and overreliance [1]. OpenAI's guardrails and human-review docs provide platform-level controls around inputs, outputs, tools, and approvals [2]. NIST's AI RMF supplies a broader governance frame for trustworthy AI design and evaluation [4].

Anthropic's agent guidance adds a practical engineering warning: autonomous agents can be costly and can compound errors, so they should be tested in sandboxed environments with appropriate guardrails [5].

The main lesson is that agent safety is architecture. It belongs in tool schemas, permission systems, state stores, approval UX, eval suites, and incident processes.

Practical Takeaways

Classify every tool by risk. Permissions should follow consequence, not convenience.

Treat retrieved content and tool output as untrusted data. They may contain instructions, but they should not gain authority.

Use guardrails at multiple boundaries: input, retrieval, tool call, tool output, and final response.

Require human approval for high-impact side effects. Preserve resumable state so approval is part of the loop, not an afterthought.

Keep audit logs. Governance depends on reconstructing what the agent saw, requested, executed, and claimed.

Red-team the full system. Attack prompts, tools, retrieval, memory, plugins, and UI flows, not just final answers.

References

[1] "OWASP Top 10 for Large Language Model Applications." OWASP GenAI Security Project. https://owasp.org/www-project-top-10-for-large-language-model-applications/. Published/updated: 2025. Accessed: 2026-07-06. Used for: prompt injection, insecure output handling, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and supply-chain risk.

[2] "Guardrails and human review." OpenAI API documentation. https://developers.openai.com/api/docs/guides/agents/guardrails-approvals. Published/updated: not listed. Accessed: 2026-07-06. Used for: input guardrails, output guardrails, tool guardrails, human review, approvals, pauses, and run continuation.

[3] "Red teaming." OpenAI API documentation. https://developers.openai.com/api/docs/guides/red-teaming. Published/updated: not listed. Accessed: 2026-07-06. Used for: adversarial testing, misuse probing, failure discovery, and security-risk evaluation.

[4] "AI Risk Management Framework." National Institute of Standards and Technology. https://www.nist.gov/itl/ai-risk-management-framework. Published/updated: 2026-04-07. Accessed: 2026-07-06. Used for: trustworthy AI framing, design/development/use/evaluation governance, and generative-AI risk management.

[5] "Building effective agents." Anthropic. https://www.anthropic.com/engineering/building-effective-agents. Published/updated: 2024-12-19. Accessed: 2026-07-06. Used for: autonomy tradeoffs, compounding-error risk, sandboxing, and guardrail guidance.

[6] "The Hitchhiker's Guide to Agentic AI: From Foundations to Systems." Haggai Roitman. Local source: agent-101/The Hitchhiker’s Guide to Agentic AI/book.tex. Published/updated: 2026. Accessed: 2026-07-06. Used for: safety as a cross-cutting concern across memory, RAG, harnesses, tools, protocols, multi-agent systems, and UI.