AI Agents 101 | 05. Context and Memory
Context and Memory
Agents live and die by what they can see.
A model call has a context window. A user has a history. A project has conventions. A company has private knowledge. A task has progress. A production agent has to decide which of those things belong in the next turn, which belong in durable storage, and which should never enter the model at all.
Context is what the model sees now. Memory is what the system can bring back later.
Level 1: Fundamentals
Context is the immediate working set: instructions, user request, retrieved documents, tool results, prior turns, artifacts, and any other content placed into the model input.
Memory is persisted information that can survive beyond the current turn or run. It may include user preferences, project rules, summaries, facts, embeddings, task state, or learned patterns.
The distinction matters because context is expensive and temporary. Memory is durable but must be selected. If you put everything into context, the model becomes slow, expensive, distracted, and easier to manipulate. If you store everything as memory, the agent becomes stale, noisy, and privacy-hostile.
OpenAI's Codex docs show one practical form of context layering: AGENTS.md files provide project instructions, and nearer files can override broader ones as Codex works within a repository tree [1]. Anthropic's Claude Code docs make a similar distinction with CLAUDE.md files and auto memory notes, while warning that context is not an enforcement mechanism for hard policy [2].
OpenAI's Codex memory docs draw the line clearly: memories are useful for carrying context from earlier threads, but required team guidance should live in project files or docs rather than only in memory [3].
The Hitchhiker guide frames memory as part of the broader agentic stack: it sits next to retrieval, harness design, context-window management, and observability [8].
Level 2: Concepts
There are several kinds of context and memory. Mixing them into one chat transcript is where many agent systems start to wobble.
| Layer | Example | Lifetime | Risk |
|---|---|---|---|
| Instructions | system policy, project rules, task constraints | durable or per-run | conflicting authority |
| Conversation | user turns, assistant turns, tool messages | session | irrelevant drift |
| Working state | current plan, blockers, approvals, artifacts | run | lost progress |
| Retrieval context | docs, files, tickets, search results | per-query | stale or untrusted data |
| User memory | preferences, stable personal facts | long-lived | privacy and stale recall |
| Project memory | repo conventions, architecture notes | long-lived | hidden outdated assumptions |
| Audit memory | traces, decisions, approvals | retained by policy | sensitive data retention |
Retrieval
Retrieval-augmented generation gives the model access to non-parametric knowledge: documents, databases, code, or files that are not baked into model weights.
The original RAG paper describes combining a parametric seq2seq model with a non-parametric dense vector index, improving knowledge-intensive tasks and making generated responses more specific and factual [6]. OpenAI's retrieval docs describe the same basic value: semantic search can surface relevant information even when the query does not share exact keywords with the source text [7].
For agent systems, retrieval is not just "search before answer." It is a context assembly step:
user intent
-> query planning
-> permission filtering
-> retrieval
-> reranking
-> compression
-> citation/evidence packaging
-> model input
Retrieval should be permission-aware. If the user cannot access a document, the model should not see it.
Compaction
Long-running agents need context management. OpenAI's compaction docs describe reducing context size while preserving state, balancing quality, cost, and latency [4]. In practice, compaction turns a long transcript into a smaller state object that carries forward the important decisions, artifacts, constraints, and open questions.
The danger is lossy compression. A bad summary can delete the one detail that matters. Use compaction for narrative continuity, but preserve structured state for things that must be exact:
- approved actions
- file paths
- identifiers
- decisions
- test results
- source citations
- tool outputs that will be audited
Prompt Caching
Prompt caching is a performance technique, not a memory architecture. OpenAI describes automatic prompt caching for repeated prompt prefixes and recommends placing stable content early and variable content later [5].
That encourages a useful context layout:
stable policy and developer instructions
stable tool definitions
stable project context
variable user input
variable retrieved context
variable tool results
Caching can reduce latency and cost, but it does not decide what should be true, fresh, or authorized.
Level 3: Engineering Detail
A good context system has three contracts: what can be stored, what can be retrieved, and what can be injected.
Memory Record
{
"id": "mem_01JZ",
"scope": "project",
"subject": "repo_convention",
"content": "Use pnpm for package scripts in this repository.",
"source": "AGENTS.md",
"created_at": "2026-07-06T09:00:00Z",
"last_verified_at": "2026-07-06T09:00:00Z",
"confidence": "high",
"ttl_days": 90,
"visibility": ["agent", "developer"],
"sensitivity": "internal",
"authority": "project_file"
}
The fields matter. A memory without source, freshness, authority, and scope is just folklore with a database row.
Context Assembly
1. Start with high-authority instructions.
2. Add task-specific user request.
3. Load relevant project or user memory by scope.
4. Retrieve fresh external evidence.
5. Filter by permission and sensitivity.
6. Rank for relevance and recency.
7. Compress low-risk narrative context.
8. Preserve exact structured facts.
9. Attach citations and tool observations.
10. Track what was injected.
The last step is underrated. If the model makes a claim, you need to know whether it came from user input, memory, retrieval, a tool result, or model inference.
Context Budget
The Hitchhiker guide describes the context window as a budget shared by instructions, memory, tools, conversation history, and reasoning space [8]. A simple working model:
context_budget =
instructions
+ tool_definitions
+ conversation_history
+ retrieved_sources
+ working_state
+ response_space
When the budget is tight, prefer:
- exact structured state over verbose transcript
- recent tool observations over old summaries
- authoritative docs over remembered guesses
- source snippets over whole documents
- summaries only when the original can be reloaded
Failure Modes
| Failure | What It Looks Like | Fix |
|---|---|---|
| memory poisoning | agent stores user-provided instructions as durable policy | require source and authority labels |
| stale recall | old preference overrides fresh tool result | use freshness and source precedence |
| context stuffing | model sees too much irrelevant text | rank, summarize, and prune |
| lost state | agent repeats work after compaction | persist task state outside transcript |
| citation drift | final answer cites the wrong source | bind evidence to claims at retrieval time |
| data leak | retrieval ignores user permissions | enforce ACLs before model context |
Current Landscape
The current ecosystem is moving toward explicit context architecture.
OpenAI's Codex uses repository instructions as a durable source of project guidance [1], and its memory docs position memory as useful recall rather than the sole home for required rules [3]. Anthropic's Claude Code docs make the same practical split between checked-in instruction files and auto-generated memory notes [2].
On the API side, compaction and prompt caching show that long-context agent work is an operational concern, not just a model-size issue [4][5]. Retrieval and file search make private corpora usable by agents, but they also push teams to solve permissions, freshness, chunking, ranking, and citation handling [7].
The Hitchhiker guide is useful as a survey here because it connects RAG, memory, context management, and harness design into one architecture [8].
Practical Takeaways
Separate context, memory, and state. Conversation history is not a database.
Store memories with source, scope, freshness, authority, and sensitivity. Do not let arbitrary chat content become permanent policy.
Use retrieval for fresh or private knowledge. Use memory for stable preferences and conventions. Use project files for mandatory team guidance.
Preserve exact state outside summaries. Compaction is helpful, but approvals, IDs, file paths, and test results should remain structured.
Apply permissions before retrieval context reaches the model. Security cannot be patched in after the model has already seen restricted text.
Track what context was injected into each run. Debugging and evaluation both depend on knowing what the model could see.
References
[1] "AGENTS.md." OpenAI Codex documentation. https://developers.openai.com/codex/guides/agents-md. Published/updated: not listed. Accessed: 2026-07-06. Used for: repository instruction layering, project-level guidance, and nested override behavior.
[2] "Memory." Anthropic Claude Code documentation. https://code.claude.com/docs/en/memory. Published/updated: not listed. Accessed: 2026-07-06. Used for: CLAUDE.md, auto memory, persistent instructions, and warning that context is not hard enforcement.
[3] "Memories." OpenAI Codex documentation. https://developers.openai.com/codex/memories. Published/updated: not listed. Accessed: 2026-07-06. Used for: memories as recall from earlier threads and guidance to keep required rules in project files or docs.
[4] "Compaction." OpenAI API documentation. https://developers.openai.com/api/docs/guides/compaction. Published/updated: not listed. Accessed: 2026-07-06. Used for: long-running conversation compaction, context-size reduction, and preserving prior state.
[5] "Prompt caching." OpenAI API documentation. https://developers.openai.com/api/docs/guides/prompt-caching. Published/updated: not listed. Accessed: 2026-07-06. Used for: stable prompt prefixes, latency/cost reduction, and context ordering.
[6] "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Patrick Lewis et al. https://arxiv.org/abs/2005.11401. Published/updated: 2021-04-12. Accessed: 2026-07-06. Used for: RAG architecture, parametric and non-parametric knowledge, and retrieval for factual generation.
[7] "Retrieval." OpenAI API documentation. https://developers.openai.com/api/docs/guides/retrieval. Published/updated: not listed. Accessed: 2026-07-06. Used for: semantic search, vector stores, and retrieval as context for model synthesis.
[8] "The Hitchhiker's Guide to Agentic AI: From Foundations to Systems." Haggai Roitman. Local source: agent-101/The Hitchhiker’s Guide to Agentic AI/book.tex. Published/updated: 2026. Accessed: 2026-07-06. Used for: RAG, memory, context-window budgeting, compaction patterns, and harness-level context management.
Enjoy Reading This Article?
Here are some more articles you might like to read next: