iii
Scenario · Academic / Research
Research agent
A literature-review agent for a research lab. Reads PDFs, scrapes arXiv, runs Python notebooks, generates LaTeX. Reproducibility, transparency, and the ability to spawn parallel sub-agents matter more than UX polish.
Reproducibility
Tool transparency
Parallel subagents
Long-horizon
The task
Given a research question, scan the last 3 years of arXiv for relevant papers. Download PDFs, extract methods and results, run their published code where possible to verify reproducibility, synthesize a structured literature review with LaTeX bibliography. Spawn parallel sub-agents per paper to avoid context overflow.
Why this is hard
This is the scenario where context-engineering and OS-level access matter most. The agent needs file system, shell, Python execution, long context, and the ability to fan out work across sub-agents that each maintain their own focused context. Anthropic's own research found multi-agent architectures outperform single-agent benchmarks by up to 90% in this regime.
Defined metrics
Capability & Coverage
- Papers correctly retrieved (recall @ 50) for a known topic
- Method extraction accuracy (LLM-as-judge + human spot-check)
- Code reproducibility rate (% of papers' code runnable)
- Citation accuracy (% of claims with valid attribution)
Process & Transparency
- Trace completeness (every tool call logged and replayable)
- Sub-agent parallelism efficiency (wall-clock speedup vs single-agent)
- Hallucination rate on summaries (per CAR-bench protocol)
- End-to-end token budget per literature review
Claude Agent SDK · LangGraph (for orchestration shell)
Claude Agent SDK is purpose-built for this: Read/Write/Edit/Bash/Glob/Web Search are native tools, subagents with isolated context windows are first-class, the Skills system makes domain expertise (LaTeX, BibTeX, arXiv API) hot-loadable, and Claude Sonnet 4.5 has demonstrated 30+ hours of autonomous coding. LangGraph earns the dual mention when you need the durability layer around it — for week-long, resumable research runs spanning many context windows, wrap Claude SDK calls inside a LangGraph checkpointed graph.
Why not the others?
OpenAI SDK sandbox-agents (Apr 2026) brought it closer, but the harness is younger and the file/shell tool depth lags Claude. CrewAI's role-based abstraction maps awkwardly to research workflows where parallelism is data-driven not role-driven. Google ADK can do it, especially if you need Gemini's long-context strengths — but the framework's enterprise lean shows. MS Agent Framework is least at-home here; research workflows rarely benefit from .NET strengths.