EnterpriseClawBench
Benchmark constructed from sanitized real enterprise agent sessions, testing coding/computer-use agents on genuine workplace deliverables (documents, spreadsheets, web pages) with role-, skill-, and rubric-tagged grading; dataset kept private for confidentiality.
Overview
EnterpriseClawBench evaluates AI agents on realistic enterprise workflows by mining a large archive of proprietary, real-world agent sessions from actual workplace usage, then reconstructing 852 reproducible tasks from them. Each task is paired with recovered environment fixtures, a rewritten self-contained prompt, role and skill-subclass classification, hard rules, and a semantic rubric — reflecting the genuine distribution of requests enterprise agents encounter (reading heterogeneous files, invoking tools, and delivering business artifacts) rather than a curated or synthetic task set.
Because the underlying sessions contain internal enterprise content, the authors do not release the raw benchmark data; instead their reusable contribution is the construction and evaluation protocol itself, along with a standardized run-directory format (task spec, prompt, agent response, generated artifacts, and audit metadata) that lets other groups reproduce the evaluation methodology on their own enterprise data. Evaluation uses LLM-judges to score delivered artifacts against the recovered rubrics and hard rules, and the paper explicitly argues that single aggregate scores are misleading — instead reporting harness–model combination, artifact/visual quality, cost, runtime, and skill-transfer behavior as separate axes.
On the benchmark, the best reported configuration — a Codex harness paired with GPT-5.5 — reaches only a 0.663 average task score across the 852 tasks, underscoring that even strong coding/computer-use agents struggle to reliably deliver correct enterprise artifacts end-to-end.
How It Works
Task setup, inputs, outputs, and evaluation.
Example Tasks
Real tasks from this evaluation system.
Enterprise document/spreadsheet deliverable task
Enterprise Knowledge Work
Agent operates inside a reconstructed enterprise workspace and must inspect files, use tools, and produce a correct business deliverable matching the original real workplace request.
What the Agent Receives
What It Must Produce
How Success Is Judged
Results
Model performance on this evaluation.
| # | Model | Score |
|---|---|---|
| 1 | Gpt 5 5 | 0.663 |
Key Findings
Best-performing configuration (Codex harness + GPT-5.5) reaches only a 0.663 average task score across 852 real-derived enterprise tasks.
The paper argues single aggregate scores obscure large harness-vs-model differences, and reports artifact delivery quality, visual quality, cost, runtime, and skill-transfer separately.
Because sessions contain internal enterprise content, the underlying dataset is not publicly released — only the construction/evaluation protocol is open-sourced.
What Makes It Unique
- ✓
Sourced from a large archive of real, proprietary enterprise agent sessions rather than crowdsourced or synthetic tasks, aiming to reflect the genuine distribution of real workplace requests.
- ✓
Each task ships with recovered fixtures, a rewritten self-contained prompt, role/skill classification tags, hard rules, and a semantic rubric rather than a single pass/fail check.
- ✓
Explicitly measures skill-transfer behavior — how performance on one workplace skill class predicts performance on related but unseen classes.
Controversies & Caveats
Underlying dataset is not publicly released due to confidential enterprise content, limiting independent verification and reproducibility outside the authors' own harness.
Author institutional affiliations are not clearly stated within the paper itself; they were inferred from separate author homepages and profiles.