OSWorld 2.0

OSWorld 2.0 is the successor to the widely-used OSWorld computer-use benchmark, built by the XLANG Lab at the University of Hong Kong with collaborators from Columbia, UC Santa Barbara, Mila, and Snorkel AI. Where OSWorld 1.0 tasks averaged roughly 30 tool calls, OSWorld 2.0 raises the bar to genuinely long-horizon professional workflows requiring an average of 318 tool calls and a median of 1.6 hours of human completion time, using 31 self-hosted realistic websites to avoid benchmark contamination and enable reproducible, controllable environments.

The benchmark was built because existing computer-use evals failed to capture the messiness of real work: information arriving mid-task, hidden state that must be tracked, ambiguous instructions, and multi-source cross-referencing. OSWorld 2.0 explicitly annotates each of its 108 tasks (spanning research, creative production, engineering, personal services, business/finance, administration/compliance, and healthcare) with the specific 'challenge phenomena' it probes, such as cross-source reasoning (42.6% of tasks), visual-spatial precision (41.7%), implicit-state inference (39.8%), and dynamic/streaming environments.

Scoring uses binary task completion (capped at 500 agent steps) plus an average of 27.25 fine-grained scoring checkpoints per task for partial credit, along with trajectory-level failure attribution. The best model, Claude Opus 4.8, completed only 20.6% of tasks (using 244K output tokens on average), while GPT-5.5 reached about 14% with far higher token efficiency (39K tokens). Performance collapses further as task length grows: no model exceeds 10% completion on tasks over ~2.3 hours, and success drops to zero beyond ~2.7 hours, showing that current frontier agents still cannot sustain professional-length work.