SW-A²-Bench
Benchmark for automatically converting GitHub repositories into autonomous, interoperable software agents for the 'Agentic Web'.
What It Tests
SW-A²-Bench ("SoftWare Agent generation for Agentic Web Bench") is the first benchmark designed to evaluate software agent *generation* rather than direct task-solving. It studies the pipeline of automatically converting existing code repositories into autonomous, A2A/MCP-compatible software agents using coding agents (e.g., Claude Code, Codex CLI, OpenHands), and asks three questions: can a valid agent be generated from a repo at all, does that generated agent faithfully inherit the repository's underlying capabilities, and can multiple independently generated agents interoperate in multi-agent workflows.
The benchmark curates 35 diverse GitHub repositories spanning 9 real-world domains (document/OCR parsing, web scraping, speech/audio processing, vision and video generation, security/vulnerability detection, NLP and prompt engineering, chemistry/molecular synthesis planning, general web/backend frameworks, and financial backtesting), and builds 522 evaluation instances (336 single-repo and 186 multi-repo tasks) with a difficulty gradient (Easy/Medium/Hard) based on environment constraints, output predictability, and domain expertise required.
What differentiates SW-A²-Bench from typical coding benchmarks (e.g., SWE-bench) is its focus on the meta-task of agent creation and interoperability under modern agent protocols (Google's A2A v0.3.0 and Anthropic's MCP), rather than solving a single coding issue. Headline results show that agent-generation itself is largely solved (Claude Code and the authors' own EnvX pipeline hit 100% Pass@1), but downstream capability inheritance is far from saturated: overall single-agent execution success rates cluster around 34-37%, collapsing to roughly 10-14% on Hard-tier tasks, while multi-agent orchestration success (65-72%) is comparatively higher — suggesting environment setup and precise skill/capability specification, not raw code generation, are the binding bottlenecks.
Task Anatomy
How a single task is structured.
Example Tasks
3 real examples from the benchmark.
Chemistry domain: aizynthfinder retrosynthetic planning (Hard tier)
Problem / Input
Faithfully paraphrased from the paper's discussion of environment-setup bottlenecks; illustrates why Hard-tier single-repo execution success rates are low (~10-14%).
Vision/audio domain: repository-specific function invocation
Problem / Input
Paraphrased from the paper's description of single-repo task construction across the vision/video and speech/audio domains.
Multi-repo sequential agent chain
Problem / Input
A chained task spanning multiple generated agents (drawn from different domains), structured so that each agent in the chain must consume the concrete output of the preceding agent as its sole valid input.Paraphrased from the paper's definition of multi-repo tasks used to test interoperability under the A2A protocol.
Leaderboard Results
Model scores sorted by performance.
4 results
| # | Model | Score |
|---|---|---|
| 1 | Claude Sonnet 4.5 | 36.9% Overall Execution SR (via Claude Code scaffold) |
| 2 | Claude Sonnet 4.5 | 35.1% Overall Execution SR (via EnvX scaffold) |
| 3 | GPT-5.2-Codex | 34.5% Overall Execution SR (via Codex CLI scaffold) |
| 4 | Claude Sonnet 4.5 | 33.9% Overall Execution SR (via OpenHands scaffold) |
V= Self-reported by the model's creator, not independently verified
Score Over Time
Performance progression across model generations.
Key Findings
Agent generation itself is largely solved: Claude Code and the authors' EnvX pipeline both achieve 100% Pass@1 in producing a validly wrapped agent from a repository (Codex CLI and OpenHands reach 94.28% Pass@1, 100% Pass@3).
Downstream capability inheritance is far from solved: overall single-agent Execution Success Rate clusters at 33.9-36.9% across scaffolds, with a steep difficulty gradient — roughly 53-58% on Easy tasks collapsing to 10.4-14.2% on Hard tasks.
Multi-agent orchestration success rates (65.1-71.5%) are notably higher than Hard-tier single-repo execution, suggesting that once agents exist, chaining them via the A2A protocol is comparatively more tractable than deep domain-specific execution.
Environment setup is a major bottleneck independent of model choice — e.g., the aizynthfinder (chemistry) trajectory required 6.5 minutes and 22 package-management operations before the actual task could run.
The paper concludes the key bottlenecks are robust environment setup, reliable skill construction, and precise capability specification — not raw underlying model coding ability.
Controversies & Caveats
Known limitations and criticisms.
As of this research (July 2026), no public leaderboard, GitHub dataset/code release, or independent third-party corroboration (PapersWithCode, HuggingFace, Epoch AI, vals.ai) could be found; all reported figures trace back solely to the paper itself (arXiv v2, June 2026).
The benchmark primarily compares agent scaffolds (Claude Code, Codex CLI, OpenHands, EnvX) rather than raw underlying models — three of the four scaffolds share the same backend LLM (Claude Sonnet 4.5), which complicates direct model-vs-model comparison and means differences partly reflect scaffold engineering rather than model capability.
Part of the evaluation (skill-specification precision/recall) relies on an LLM-as-judge framework rather than fully deterministic execution checks, which is not independently validated in the paper.