AI Agents Series | Lessons from Engineers and Scientists

Verification: stop making the human carry every observation

One of the most useful stories comes from Tan’s first week working on Cursor’s agent window, a React application. With a launch approaching, she was investigating performance in an unfamiliar codebase. She would inspect a Chrome DevTools trace, send evidence to an agent, and receive a confident explanation. After trying the suggested fix, she would discover that it had identified the wrong cause.

This created a slow loop: the agent changed code, Tan ran the application, Tan collected the next screenshot or error, and the agent tried again. More agents would have increased the number of requests for her attention. The missing capability was a way for each agent to observe whether its own hypothesis survived contact with the running application.

She built a control skill that let agents launch and interact with the application and collect runtime evidence. In the talk, verification includes tools such as performance traces, heap snapshots, and simulator interaction. This gives an agent a way to reproduce a problem and check a change against the behavior that originally failed. Verification and the performance investigation, 05:56–10:06.

Lesson learned: identify every point where a person must manually move evidence between the application and the agent. Automating that observation can remove a bottleneck from the whole investigation. Verification still has a scope: demonstrating that one reported interaction works does not establish that the implementation is maintainable or that unrelated behavior is unaffected.

The feature map: tool access needs product knowledge

The first control skill exposed another failure. An agent could operate the application, but it did not know how to reach the relevant feature. A report about a slow sidebar or a broken PR tab could send it searching through code and clicking around without making progress.

Tan added a feature map: a guide connecting user-facing concepts to the steps needed to exercise them. Her examples include where a feature appears, which subfeatures it contains, keyboard shortcuts, and DOM attributes useful for selecting controls. A separate maintenance skill helps keep this knowledge current.

Her anecdote about internal feedback makes the value concrete. A bug report might contain little more than a screenshot and a few question marks. With a feature map, an agent has a better chance of connecting that fragment to an actual product workflow. Without it, the agent has to rediscover the interface before it can investigate the bug. Feature maps and sparse bug reports, 10:17–13:29.

Lesson learned: document the route from a user’s description to an observable behavior. An API reference explains which operations exist; a feature map explains how those operations relate to the product someone is using. This is particularly valuable when the person reporting a problem does not know the implementation vocabulary.

Skills grew out of specific failures

Tan says pstack emerged incrementally. She did not begin with a complete catalog of ideal agent behaviors. She watched agents fail, inspected their actions, and turned recurring problems into reusable instructions.

One example was an agent confidently diagnosing a bug without reading the code she expected to be involved. Looking at the tool calls revealed a gap between the explanation and the investigation. Her response was to encode a more disciplined investigation process, including examining the relevant code and gathering evidence before reaching a conclusion.

Her management analogy is useful here: imagine a capable programmer who joined the team seconds ago and has no business context. Technical ability does not automatically supply knowledge of the product, its history, or the team’s standards. A skill can preserve some of the guidance that an experienced colleague would otherwise repeat in every session. How pstack developed, 13:41–17:04.

Lesson learned: a recurring correction is evidence about a missing part of the workflow. Before adding another broad instruction, name the actual failure: skipping the relevant code, confusing a hypothesis with a finding, or failing to reproduce the symptom. That makes the resulting skill easier to evaluate and revise.

Evaluate the skills, including the verification skill

Instructions can sound rigorous while producing little change in behavior. Tan describes evaluating skill changes with a coordinator that defines a rubric and launches agents in separate directories. She also runs evaluations across models and sometimes uses a different model to judge the results.

A memorable detail is her attempt to avoid making it obvious to the agents that they are being evaluated. She reports that agents can behave differently when they recognize an evaluation setting. Whether a particular evaluation successfully avoids that problem needs its own evidence, but her concern is worth preserving: performance in an artificial test may differ from ordinary work.

She applies this process to the control and verification skills themselves, iteratively improving them against an evaluation. She also emphasizes that maintenance takes close observation and judgment. A skill collection needs attention as the product and agent behavior change. Skill evaluation and iteration, 17:40–23:21.

Lesson learned: evaluate the behavior an instruction is supposed to change. For an investigation skill, useful questions include whether the agent examined the relevant implementation and connected its conclusion to evidence. For a verification skill, ask whether it exercised the intended behavior and detected a known failure.

My interpretation: optimizing until a judge awards a perfect score can improve the evaluated workflow, but that score alone cannot establish general reliability. I would also want fresh cases that were not used to revise the skill, including cases where the correct response is to report insufficient evidence. A second model’s judgment adds another perspective; it does not replace execution evidence.

Autonomy followed reliable local work

Tan recommends starting locally, where a person can watch an agent operate the application and inspect how it uses the control interfaces. Once that loop is dependable, the same capabilities can support cloud agents and shared automations.

Her example is Benny, an agent that investigates incoming bug reports. It runs the application in its own environment and tries to reproduce the reported behavior. In one case, it established that the bug had already been fixed on the main branch, so the remaining action was to release a new build. That finding saved the team from doing another investigation or writing an unnecessary patch.

This is a useful way to judge automation: a successful run can resolve uncertainty without producing code. Tan also cautions against jumping directly from supervising a few unreliable agents to launching large numbers of cloud agents; that can consume many tokens without fixing the underlying workflow. Starting locally and the Benny example, 23:57–28:18.

Lesson learned: increase concurrency after establishing that one agent can complete the relevant observation-and-verification loop. Reusable verification also changes the economics of the initial investment: the same capability can support other engineers and automated triage across the team.

Architecture makes preferred behavior easier to repeat

The second half of the talk shifts from agent instructions to codebase structure. Tan argues that large engineering organizations already face a version of the problem: many contributors with different levels of context can introduce changes faster than a few experienced reviewers can inspect every detail.

In her account, established codebases with strong conventions may give agents a better starting point than quickly generated prototypes with few constraints. Agents tend to copy nearby patterns and choose convenient solutions. Those tendencies make the existing architecture consequential: each new change can reinforce either a useful convention or an accumulating shortcut. Refactoring and inherited constraints, 29:29–35:18.

She describes an internal Electron application framework called Dune, with explicit structure and strict checks. Examples include colocating feature code, defining conventional extension points, and checking dependency boundaries between main-process and renderer code. The renderer-boundary example addresses a recurring performance failure: computational work entering the UI process and competing with rendering.

Some choices are deliberately opinionated. She reports banning useEffect in that application and restricting code comments after seeing agents turn local review feedback into misleading permanent explanations. These are examples of decisions for her environment, rather than universal rules for React or documentation. The transferable mechanism is to identify a repeated problem and encode an enforceable constraint around it. CI rules and architectural examples, 37:55–44:30.

Tan distinguishes these hard constraints from softer guidance such as skills, review bots, and written rules. All can contribute, but a dependency check that fails CI gives a different guarantee from a paragraph an agent may overlook. Her recurring question is whether a repeated human review comment can become a lint rule, a CI failure, or an architectural change that removes the problem. Layers of enforcement, 44:30–47:05.

Lesson learned: design the repository so that a straightforward implementation follows the intended architecture. Human review remains useful for choices that require judgment; repeatedly correcting the same mechanically detectable violation is a signal that the tooling could do more.

Personal working habits and the cost behind the result

Several smaller observations make this account more useful than a list of agent tips:

• She watches behavior closely while developing a workflow. Reading tool calls exposed failures that a polished final answer hid. Her later autonomy grew out of a period of detailed supervision.
• She favors PRs that each describe one coherent change. She gives no fixed size cap, but values Git history as context and wants regressions to be easier to locate and revert. PR size and history, 35:23–37:41.
• She uses a head-chef analogy. The engineer increasingly designs the environment, distributes work, and sets standards. This captures the work involved in making delegation reliable. The analogy, 22:30–23:21.
• She acknowledges unusually generous token access. Her workplace gives her effectively unlimited tokens, and she explicitly avoids presenting her exact approach as the right expenditure for everyone. The refactoring and verification infrastructure required substantial upfront effort. Costs and return on investment, 47:43–50:50.
• She sees benefits beyond her own output. She reports designers and product managers contributing changes successfully once the application has stronger conventions and checks. This is a reported team benefit, rather than evidence that all engineering judgment has become automatic. Other contributors, 52:23–55:01.

These details help explain why copying the visible endpoint, such as automatic merging, would miss much of the work that made it possible. Her account supports investing in a reliable process; it does not provide a controlled comparison of costs, defect rates, or long-term maintenance outcomes.

Related foundations in this series: Tools, MCP, Skills, and Plugins, Evaluation and Observability, and Coding Agents as a Case Study.

Start with a shared design, and involve the people who know the answers

Pocock describes a frustration with planning modes: the agent would quickly decide it had enough information and generate a plan, while he still felt that they were imagining different things. He connects the missing element to Frederick Brooks’s idea of a shared design concept. The useful outcome of the conversation is agreement about what is being built and why.

His grill-me skill turns the agent into an interviewer. In the workshop version, it asks one question at a time, supplies a recommended answer, and investigates the codebase. The gamification exercise quickly reveals decisions that the original brief omitted: which actions earn points, whether previous completions count, how levels progress, whether streaks earn additional points, and where the interface appears. Retroactive points are a particularly useful example because an apparently small product choice changes the implementation and backfill requirements. The interview and its motivation, 12:14–21:47.

He reports that these sessions can run to dozens of questions, and that he normally prefers dictation. He also makes the limits of the live demonstration visible: to save workshop time, he asks the agent to supply a list of recommendations and answer more of its own questions. That shortcut should not be mistaken for the human alignment process he normally advocates.

Lesson learned: use the interview to expose decisions and find their owners. When a developer cannot answer a product question, Pocock recommends bringing the domain expert into the discussion with the agent. A confident recommendation does not establish the missing business requirement. Team participation and human-in-the-loop work, 24:59–27:31.

Treat context as a working budget and make tasks resumable

Pocock frames long conversations in terms of a useful working region and a region where the agent’s decisions become less dependable. He prefers a small amount of always-present instruction, bounded tasks, and a fresh start when moving between pieces of work. He dislikes repeatedly compacting one long session because he prefers the predictable starting state of a cleared conversation.

His rough marker of around 100,000 tokens is presented as his current rule of thumb. The workshop does not establish a universal threshold, and its attention analogy does not by itself prove when coding quality declines. The more transferable observation is that a large advertised context window does not tell a developer how much accumulated conversation will remain useful for a particular task. Context, task size, and compaction, 03:00–11:03; his long-context preference, 37:24–38:31.

The live exploration also illustrates context isolation. A subagent uses roughly 94,000 tokens while exploring, then returns a summary to the parent. This keeps the exploration transcript out of the parent’s working context; it does not mean those tokens were never consumed. Exploration in a separate context, 17:35–18:32.

Lesson learned: design the handoff so that a new session can recover the task, relevant decisions, and current state without inheriting the whole conversation. In his later implementation loop, issues and recent commits provide that durable starting point. Clearing context is useful only when the information needed to continue has somewhere else to live.

Separate the destination from the route, then review the task boundaries

After the interview, Pocock creates a product requirements document, or PRD, containing the problem, proposed solution, user stories, implementation decisions, and testing decisions. It records the destination and definition of done. He then breaks the work into independently selectable issues with dependencies and labels indicating whether a human must remain involved. Creating the PRD, 28:40–35:56; task dependencies, 39:32–41:43.

His most instructive intervention concerns vertical slices. He observes that agents tend to divide work horizontally: database first, service or API second, interface last. That delays the first opportunity to check whether the layers work together. He wants an early task to cross the necessary layers and produce a small, usable behavior.

Even with vertical slicing written into the skill, the agent proposes building the gamification service first. He rejects that boundary. The revised task awards points for completing a lesson and displays the result on the dashboard. It is still bounded, but now there is a real flow to exercise. This is the workshop’s clearest example of a short human review changing the quality of everything that follows. Vertical slices and the live correction, 41:47–46:56.

Pocock makes an unusual distinction about review effort: he says he generally does not reread the generated PRD after a thorough interview, because he trusts the summarization step. He does inspect the task breakdown, where he can spot consequential mistakes quickly. That is his personal allocation of attention; a summary can still omit a constraint, especially when several stakeholders were involved.

Lesson learned: review whether each task creates useful feedback, rather than judging a plan only by how organized it looks. Dependencies also reveal which tasks can run concurrently. A numbered list often hides that structure; explicit blocking relationships make it available to the person or agent assigning work.

The unattended loop runs over a curated backlog

Pocock describes planning as the human’s day shift and implementation as work that can run during the night shift. His first demonstration is deliberately simple: a script provides the agent with local issue files, the last five commits, and an implementation prompt. The agent chooses an eligible task, explores the repository, implements with tests, runs checks, and produces a commit. The prompt distinguishes tasks suitable for unattended work from tasks that still need human participation. The single-run script and prompt, 52:15–57:48.

He recommends watching individual runs before repeating them automatically. That makes it possible to see where the prompt or environment needs adjustment. Later, he demonstrates a parallel setup using Sandcastle: a planner selects compatible issues, implementers work in separate sandboxed branches, reviewers inspect their commits, and a merger combines the results and addresses integration failures. This describes the setup shown in the recording, rather than every configuration of the current library. Parallel implementation, review, and merging, 01:29:50–01:32:42.

The human workflow continues around this loop. QA findings become new backlog issues, including blockers, while agents work on other eligible tasks. Pocock also explains that real projects can move back from research or prototypes into earlier design discussions; the neat sequence on his diagram is not a claim that product development is linear. Team iteration, 01:00:31–01:03:48; QA feeding the backlog, 01:34:44–01:35:22.

Lesson learned: unattended implementation depends on the quality of the work queue. An unresolved product decision does not become an implementation task merely because it has been put in a ticket. Useful task boundaries, clear dependencies, and a way to return findings to the backlog are part of the automation.

Tests provide feedback, and the live failure shows their boundary

Pocock strongly favors test-driven development: write a failing test, implement enough to make it pass, then refactor. He reports that agents produce weaker tests when they write the entire implementation first and add tests afterward. The small red–green cycle gives the agent feedback while it builds, and makes it harder to justify an implementation solely with tests that mirror what it already wrote. TDD in the demonstration, 01:06:38–01:08:34.

After the first task, the agent runs the tests and type checks, fixes a type error, and reports completion. Pocock explains that his own review would start with the tests: are they checking reasonable behavior? Then he would inspect the implementation and try the dashboard. This ordering matters because a green test suite only establishes what its assertions actually cover.

The manual check immediately finds a problem. Completing a lesson fails with a SQLite error because the running database lacks the points-events table. He starts investigating the migration command, but does not demonstrate a completed fix before moving on. The recording therefore shows a successful automated check followed by a failed real interaction, rather than a fully verified finished feature. Checks, test review, and the missing table, 01:09:34–01:12:27.

Lesson learned: include the application environment in the definition of completion. Service tests and type checks can pass while the actual database state prevents the user flow from working. This is also a concrete connection to Tan’s emphasis on runtime verification: the relevant question is whether the agent exercised the behavior under the conditions that matter.

Deep modules preserve testability and a human mental model

Pocock draws on John Ousterhout’s concept of deep modules: substantial behavior behind a small, understandable interface. He contrasts this with a web of tiny interconnected units whose dependencies are difficult to follow and whose tests become coupled to internal wiring. The distinction concerns how much complexity the interface hides, rather than simply how many lines a file contains. Module boundaries and testability, 01:14:19–01:19:01.

For agents, a useful boundary makes it easier to determine what a component promises and test that promise. For people, it offers a way to retain architectural understanding without memorizing every generated function. Pocock says he wants to design the interfaces, understand how the major components behave, and delegate more of their internal implementation. He describes this as a way to preserve his sense of the codebase while keeping the workload manageable. Architectural understanding, 01:19:19–01:21:08.

His personal example is the browser-based video editor in his course-video-manager application. He describes using his architecture-improvement skill to find a boundary around the frontend-to-backend flow, allowing it to be tested from the outside. He reports a substantial improvement in the agent’s ability to change that part of the application once it could observe and test the complete flow. This is an experience report, not a measured comparison of architectures. The video-editor example, 01:21:12–01:23:07.

Lesson learned: when agents repeatedly struggle in an area, investigate whether the component boundaries make its behavior difficult to understand or test. A stronger prompt may help, but changing the interface and feedback available to the agent can address a more persistent cause.

Review in a fresh context and supply the review criteria explicitly

Pocock recommends an automated review step before human QA. He prefers a fresh context for that review so it is not appended to an already crowded implementation conversation. In the setup shown, he assigns different models to implementation and review, prioritizing reasoning ability for the latter. Those model choices are part of the recorded setup, rather than a permanent ranking. Separate review context, 01:04:52–01:06:24; model roles, 01:32:23–01:32:42.

He also distinguishes information an agent can retrieve from information supplied directly in its prompt. For implementation, he wants coding standards available on demand. For review, he wants the relevant standards explicitly included alongside the code, so the reviewer has a concrete basis for comparison. Supplying standards to implementers and reviewers, 01:27:40–01:29:42.

Lesson learned: give the reviewer a defined job and the evidence needed to perform it. A new session alone does not establish independence or correctness. Pocock still reserves human attention for the behavior, design, and quality of the result, and candidly says he does not have a complete answer to the growing volume of code review. The review bottleneck, 58:58–01:00:18.

Own the workflow and retire obsolete planning context

Several preferences reveal how Pocock adapts his process. He favors small, editable skills because he wants to understand and repair the workflow when it fails. He abandoned an earlier instruction to compress plans aggressively once he realized that an interactive design conversation was more useful to him than a shorter plan. For interface design, he suggests generating several throwaway prototypes and using human feedback to decide what should enter the real implementation. Owning the process, 23:26–24:29; changing his planning habit, 47:55–49:15; prototype feedback, 01:02:26–01:03:48.

He is also wary of completed PRDs remaining in the repository as if they still describe the current system. Names, structure, and requirements can change, leaving an agent to rediscover an obsolete document and treat it as authoritative. His normal GitHub-issue workflow marks completed work as closed: the history remains retrievable, but its status distinguishes it from active work. Planning history and documentation decay, 01:23:23–01:25:05.

Lesson learned: preserve the useful decisions while making their status clear. A historical plan, an active requirement, and a maintained description of the current system serve different purposes. The workshop’s larger message is to keep adapting the process using observed failures, rather than assuming that a particular framework or prompt will remain sufficient.

Taken alongside Tan’s experience, this account emphasizes a different allocation of human attention. Tan describes extensive investment in verification and enforceable constraints before allowing automatic merging. Pocock emphasizes human alignment, architectural understanding, and QA around delegated implementation. Both accounts make the surrounding engineering process central to the quality of agent output; their preferred levels of autonomy reflect their own settings and experience.