AutomationBench
Zapier's benchmark testing whether AI agents can autonomously orchestrate real cross-application business workflows (CRM, inbox, calendar, ticketing) via REST APIs, under real business policy constraints and misleading data.
Overview
AutomationBench, created by Daniel Shepard and Robin Salimans at Zapier, evaluates AI agents on their ability to complete real business automation workflows that span multiple SaaS applications — the kind of multi-app orchestration tasks that flow through Zapier's own platform. Rather than isolated single-tool calls, tasks require an agent to discover the correct API endpoints itself, read and follow a written business policy document, and correctly write data across systems such as Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot.
The benchmark consists of 657 tasks spanning six business functions — Sales, Marketing, Operations, Support, Finance, and HR — set in simulated environments seeded with realistic data, including irrelevant or deliberately misleading records that agents must filter out. Grading is fully programmatic and end-state focused: it checks whether the correct information reached the correct downstream systems, and separately tracks whether agents violated any stated business/policy guardrails while doing so.
Results reveal a gap between partial-credit and strict scoring: on the public AutomationBench-AA leaderboard (Artificial Analysis), top models complete roughly half of task objectives on average (Grok 4.5 at 51.4%, GPT-5.6 Sol at 51.2%, Claude Fable 5 at 48.6%), but the original paper reports that full end-to-end task completion with zero guardrail violations remains below 10% for the best frontier models — underscoring how much headline 'completion' scores can overstate real operational reliability.
How It Works
Task setup, inputs, outputs, and evaluation.
Example Tasks
Real tasks from this evaluation system.
Cross-application sales-to-support workflow
Business Operations Automation
Agent must complete a multi-step business workflow spanning several SaaS tools (e.g., updating a CRM record, notifying a channel, and logging a ticket) while following a written policy document.
What the Agent Receives
What It Must Produce
How Success Is Judged
Results
Model performance on this evaluation.
| # | Model | Score |
|---|---|---|
| 1 | Grok 4 5 | 51.4% |
| 2 | Gpt 5 6 | 51.2% |
| 3 | Claude Fable 5 | 48.6% |
Key Findings
Top frontier configurations complete only ~50% of task objectives on average on the AutomationBench-AA leaderboard (Grok 4.5: 51.4%, GPT-5.6 Sol: 51.2%, Claude Fable 5: 48.6%).
The original paper reports that full end-to-end task completion with zero guardrail violations remains below 10% for even the best models, revealing a large gap between partial-credit and strict scoring.
657 tasks span 6 business functions and 7 real SaaS platforms (Gmail, Sheets, Slack, Salesforce, Zendesk, Jira, HubSpot), requiring agents to self-discover correct API endpoints amid irrelevant/misleading data.
What Makes It Unique
- ✓
Built directly from real workflow patterns observed on Zapier's own automation platform — grounded in actual production business-automation demand rather than synthetic scenarios.
- ✓
Grades on two axes simultaneously: task completion AND business-policy/guardrail adherence, penalizing agents that finish a task while breaking stated rules.
- ✓
Publicly tracked via a continuously updated Artificial Analysis leaderboard (AutomationBench-AA) and reportedly used internally by Zapier as a continuous agent-improvement loop.
Controversies & Caveats
Created and hosted by a commercial automation vendor (Zapier) whose products stand to benefit from a benchmark framed around SaaS-workflow orchestration, raising some conflict-of-interest concerns in task design.
Large gap between the lenient 'objective share completed' metric (~50%) and the strict full-completion-with-no-violations metric (<10%) means leaderboard headline numbers can look considerably rosier than real-world reliability.