Stanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon Tasks
mindmap
root((Agentic Evaluations and Long-Horizon Tasks))
Evaluation dimensions
Task duration and complexity
Reliability at explicit success thresholds
Economic usefulness versus experts
Synthesis quality and verifiability
METR time horizons
Human-time-calibrated task suites
Repeated runs fit success curves
Seven-month doubling at 50 percent
Much shorter horizon at 80 percent
Planning tool and loop failures
GDPVal professional work
Multimodal deliverables across occupations
Pairwise preference against experts
Linear aggregate win-rate gains
Iterative attempts improve cost and speed
Performance depends on task and modality
DeepScholar-Bench
Live recent literature avoids contamination
Retrieval synthesis and citation evaluation
Foundational sources and key facts are missed
Fluent prose can remain unverifiable
Large headroom below 19 percent overall
Cross-benchmark lessons
Complete context makes tasks easier
Human orchestration supplies tacit priorities
Long duration does not guarantee quality
Long-tail reliability remains difficult
Use multiple metrics and human validation
Agentic evaluation needs multiple complementary measures of progress
A portfolio of agent evaluations
- METR: measures the human-equivalent duration of tasks an agent completes at a chosen success probability.
- GDPVal: asks whether a professional would prefer an AI deliverable to expert human work on an economically valuable task.
- DeepScholar-Bench: tests whether an agent can retrieve literature, synthesize it coherently, and support claims with citations.
Central lesson: benchmark progress must represent sustained, useful, and verifiable work. A saturated chatbot or single-answer test cannot establish all three.
Task duration, economic value, and reliability answer different evaluation questions
Keep the axes separate
Duration asks how long or complex a task can be. Reliability asks how often the agent actually finishes it. Economic usefulness asks whether the resulting artifact is competitive with professional work.
METR combines duration with a success threshold such as 50% or 80%; GDPVal uses expert pairwise preference. Consequently, an impressive time horizon can coexist with frequent failure or an unusable deliverable. Claims about capability should always name both the metric and its success criterion.
METR derives time horizons from three suites and repeated agent runs
How the time horizon is estimated
1. Assemble and vet approximately 170 tasks across short atomic actions, HCAST engineering work, and RE-Bench ML research work.
2. Measure successful professional completion times and task difficulty.
3. Run agents repeatedly and estimate success for each task.
4. Fit success against human completion time and read off the 50% or 80% horizon.
Transcript caveat: it describes HCAST as spanning roughly one minute to 30 hours, but also describes the combined evaluated range as reaching eight hours. These may refer to different suite versus selected-task ranges, but the lecture does not resolve that distinction; retain both values as reported.
Human-calibrated examples reveal both the breadth and bias of time estimates
Human time is a useful but imperfect anchor
The suite intentionally spans seconds-to-hours work: shell-file lookup, BibTeX retrieval, simulation debugging, JSON transformation, literature review, refactoring, and CUDA optimization. Skilled professionals with about five years of experience record successful completion times, and the benchmark aggregates difficulty with a geometric mean.
Calibration risks: experts carry tacit shortcuts and may underestimate how hard a task is; a task difficult for a model need not be difficult for a human; and a small human sample can be noisy. The lecturer recommends checking inter-human agreement and collecting more observations when estimates are far apart.
The reported 50-percent task horizon has doubled rapidly but remains a coin flip
Fast growth with a crucial denominator
The lecture reports a progression from about 2 seconds for GPT-2, to about 8 minutes for GPT-4, to 59 minutes for Claude 3.7 Sonnet at the 50% success threshold—an estimated doubling roughly every seven months.
Better reasoning, coding, tool use, loop avoidance, error recovery, goal tracking, and memory all contribute. Yet the headline remains a median-like capability: at the stated horizon, an agent may succeed only half the time. Longer tasks also amplify opportunities to drift or compound errors.
Context engineering, planning, feedback, and memory help agents sustain longer work
The long-horizon control stack
- Context engineering: retain and compact the information needed across many steps.
- Planning: decompose complex work before execution.
- Re-planning: revise the plan after tools reveal new evidence.
- Feedback: treat user corrections and recurring failures as signals for another attempt or workflow improvement.
- Memory: preserve an evolving model of the codebase or environment and the remaining distance to the goal.
These mechanisms explain why an agent product can improve even when its underlying language model changes little.
High-reliability horizons remain much shorter and expose structural failure modes
Reliability changes the practical conclusion
Claude 3.7 is presented with a 59-minute horizon at 50% success, but a much shorter horizon at 80% success. Transcript caveat: the lecturer first reads the 80% plot as approximately 8–10 minutes, then later states 15 minutes. The source does not reconcile those two readings, so the defensible claim is simply that the high-reliability horizon is dramatically shorter.
Failure analysis points to planning, tool choice, arithmetic/reasoning, completion detection, premature abandonment, and repeated high-probability actions after failure. o1 loops less than GPT-4 in the shown analysis, but the taxonomy remains relevant across models.
METR is informative but favors clean tasks and low-context comparisons
What the horizon does not measure
METR is a useful first attempt at duration-based evaluation, but clean, automatically judgeable tasks are not representative of every workplace problem. Messy tasks may have several acceptable answers. Public SWE-bench repositories create possible training exposure, and experts can underestimate familiar work.
The private-code comparison is especially revealing: contractors lacking local context take 5–18 times as long as maintainers, and models look more like those contractors. A human-time horizon should therefore be interpreted as low-context task execution, not proof of maintainer-level domain knowledge.
GDPVal compares multimodal professional deliverables against experienced experts
A benchmark of deliverables, not quiz answers
GDPVal sources well-specified professional work across nine sectors, 44 occupations, and roughly 1,320 tasks, including text, CAD, video, audio, spreadsheets, and presentations. Around 70% require reference files, and 89% are described as well specified. Outputs are compared pairwise with work from professionals who have more than ten years of experience.
Transcript caveat: the lecture alternates between 200 and 220 tasks for the open gold subset on Hugging Face. It does not resolve which count is current, so report it as approximately 200–220 rather than selecting one. Subjective expert preference measures quality, while authorization and safety remain separate deployment questions.
GDPVal reports steady gains alongside instruction-following and quality failures
Progress looks different when judged as real work
Aggregate expert win rate rises from 12.4% for GPT-4o to 47.6% for Claude Opus 4.1 in the displayed comparison. The lecturer calls this roughly linear, but it should not be treated as disagreement with METR because the axes differ: professional preference is not a time horizon.
Model strengths vary by modality. Recurring failures are more operational: ignoring required reference files, hallucinating instead of inspecting them, and violating output format. GPT-5’s distribution is reported as roughly half acceptable-but-subpar, 20% genuinely better, and 29% bad or catastrophic. Human grader disagreement is another source of uncertainty.
Iterative attempts improve economics, but capability varies sharply by occupation
Self-improvement changes the comparison
A single attempt understates what an agentic loop can do. GDPVal’s analysis allows repeated attempts in which each result is improved, and the lecture reports approximately 1.6× cost improvement and 1.4× speed improvement for GPT-5 versus an unaided expert, with successful runs costing under 10% of expert salary.
Near-parity pockets appear across clerical, purchasing, real-estate, software, administrative, compliance, health-management, editing, research, service, and sales work. These are task-level averages—not evidence that every task, entire occupation, or top specialist has been matched.
Fully specified prompts overstate how independently models can perform real jobs
Execution is not problem formulation
Under-specifying a task causes both a modest win-rate decline and a deeper failure: the model may not know what problem to solve. GDPVal supplies the context an experienced worker holds mentally, then tests execution. Real jobs also require discovering priorities, defining scope, gathering context, and deciding where effort should go.
The practical reading is human architecture plus model execution. AI assistance can already be economical when supervised, but a task-level win does not imply occupation-level replacement. The benchmark’s durable value is a quantitative taxonomy of where delegation is promising.
DeepScholar-Bench evaluates live related-work generation across retrieval, synthesis, and citation support
A live benchmark for evidence-backed synthesis
DeepScholar-Bench generates related-work sections from recent arXiv literature across 22 domains and refreshes monthly. Using papers after major training cutoffs reduces contamination.
It separates three abilities:
1. Synthesis: coherent organization and key-fact coverage.
2. Retrieval: relevant, important, and sufficiently comprehensive sources.
3. Verifiability: whether citations support claims and whether claims are adequately cited.
Human agreement is roughly 70–80%, and no evaluated system exceeds 19% overall, leaving unusually large headroom compared with saturated benchmarks.
Research agents produce polished prose but miss foundational evidence and key facts
The bottleneck is evidence coverage, not fluency
Current systems can write persuasive prose while missing foundational papers and essential findings. Reported document importance remains below 12.5%; DeepScholar-base reaches around 90% citation precision, while OpenAI Deep Research is coherent but less verifiable. Supplying ideal papers still yields only about 50% key-fact coverage.
A useful related-work section must jointly retrieve important sources, extract their decisive facts, organize comparisons, and attach the correct evidence to each claim. Optimizing fluency alone can conceal—not solve—those failures. Human expertise remains valuable for knowing which work is foundational.
The three benchmarks illuminate different omissions of real-world agent work
Each benchmark controls away a different hard problem
- METR measures time and success on automatically judged, single-agent tasks, but not multi-agent collaboration, resource limits, or costly mistakes.
- GDPVal measures professional output quality, but supplies unusually complete context and permits no iterative dialogue.
- DeepScholar-Bench measures live retrieval and evidence use, exposing source-importance and citation failures.
Research synthesis connects the three: it can last 30 minutes to eight hours, has clear economic value, and still fails through incomplete retrieval or unverifiable claims. Real work combines all these dimensions rather than isolating one.
Models are strongest on specified knowledge work but weak on context and long-tail reliability
Balanced capability assessment
High confidence: isolated, well-specified software, ML, and digital knowledge tasks; organized outputs; meaningful hour-scale assistance.
Lower confidence: ambiguous context acquisition, adversarial environments, 95% reliability, non-knowledge domains, comprehensive retrieval, and simultaneous citation precision and coverage.
A seven-month METR extrapolation suggests month-long tasks in 2028–2031, but that is a conditional forecast from a 50% horizon. GDPVal’s steadier win-rate curve and DeepScholar’s quality failures warn against converting it directly into an automation timeline. Duration, utility, quality, and human comparison must be measured together.
Future progress depends on solving the last mile of reliability and domain-specific context
The last mile is the research frontier
The lecturer expects continued coding gains but highlights distributed systems and domain-specific edge cases as persistent challenges. Current models are strong co-scientists and brainstorming partners; the proposed stronger milestone is an AI scientist that forms hypotheses, runs experiments, and builds the next model without human control of the loop.
METR success is estimated from multiple attempts, so a 50% horizon remains a coin flip per comparable task. Closing the long tail requires representative tasks and environments, new domain data, stronger core capabilities, and reliable context acquisition. Humans still add fundamental reasoning and the context that tells an agent what matters.
Enjoy Reading This Article?
Here are some more articles you might like to read next: