Amazon Science has published SOP-Bench, an open benchmark that evaluates how well AI agents execute authentic standard operating procedures drawn from twelve business domains. The framework includes more than 2,000 tasks, each paired with functioning tool interfaces and ground-truth answers so that an agent's success is measured by completing the procedure rather than by producing text that an automated grader prefers.
The benchmark was presented at KDD 2026 and is available on GitHub along with a technical paper on arXiv. Amazon researchers built SOP-Bench through a human-AI collaborative workflow: domain experts authored the original SOPs from real industrial workflows, while an Anthropic Claude 3.5 Sonnet v2 model handled the mechanical work of turning each procedure into runnable tool specifications, mock APIs, and test datasets that mix ordinary cases with edge cases and outright failures. Every generated artifact was then validated by the experts before inclusion.
What's new
- 12 domains, 2,000+ tasks: healthcare intake, dangerous-goods classification, customer service, content moderation, financial compliance, warehouse inspection, aviation pre-flight checks, retail seller routing, finance entity verification, autonomous-driving object detection, media content flagging, and logistics package assessment.
- Executable interfaces: each task provides the SOP text, tool specifications, mock tool code, and test cases with known correct outcomes.
- Grounded metrics: Execution Complete Rate (ECR), Completed-Task Success Rate (C-TSR), and Task Success Rate (TSR) are computed against ground truth, not model-based judges.
- Extensible framework: teams can drop in their own agents, test against the bundled procedures, or add new SOPs using the same four-component schema (text, tools, specs, test cases).
- Two baseline agents: a Function-Calling (FC) agent and a ReAct-style reasoning agent, evaluated across eleven frontier models.
Why it matters
Most existing agent benchmarks isolate a single capability — API selection, constraint adherence, or multi-step planning — using clean, machine-formatted prompts. Real SOPs demand all of those skills simultaneously: coordinated multi-tool use, interpretation of ambiguous instructions, reliance on implicit domain knowledge, and recovery when a step does not go as expected. SOP-Bench is the first public benchmark to combine realistic procedures with working tools and objective grading, giving practitioners a way to validate agents on the exact workflows they would face in production before deploying them.
Our take
The finding that a routine model upgrade can silently lower success rates — without any obvious regression signal — is the most actionable insight for teams running agents in production. It shifts the burden from "pick the highest-ranked model" to "test your actual procedures after every change," which is exactly the workflow SOP-Bench was designed to support.
