VS-Bench (Visual Strategic Bench)
A CVPR 2026 Oral benchmark testing whether vision-language models can perceive, reason strategically (theory-of-mind), and make good decisions inside 10 vision-grounded multi-agent games.
What It Tests
VS-Bench (Visual Strategic Bench) is the first benchmark to systematically evaluate vision-language models' (VLMs) strategic abilities inside interactive, multi-agent visual environments. It was built to close a gap in existing evaluation: prior work tested either single-agent VLM perception/reasoning or text-only multi-agent LLM strategy, but nothing tested strategic, theory-of-mind reasoning when the state of the world (and other agents) must be read off an image. VS-Bench comprises ten vision-grounded environments spanning cooperative (Hanabi, Overcooked, Knights-Archers-Zombies/KAZ), competitive (Breakthrough, Kuhn Poker, Atari Pong, Multi-Agent Particle Environment/MPE), and mixed-motive (Coin Dilemma, Monster Hunt, Battle of the Colors) game families.
The benchmark decomposes 'strategic ability' into three measurable dimensions: perception (can the model correctly read game state — tokens, cards, grid contents — off a screenshot?), strategic reasoning (can the model predict what other agents will do next, i.e. theory-of-mind via next-action prediction accuracy?), and decision-making (can the model actually play well, measured by normalized episode return against conventional agents or in self-play?). This three-way split is what differentiates VS-Bench from prior multi-agent or single-agent VLM benchmarks: it isolates whether failures come from not seeing the board, not predicting opponents, or not converting correct beliefs into good actions.
The headline finding is a large and growing gap as tasks move from perception to action: across fifteen leading VLMs, models showed strong perception (best model ~85% element-recognition accuracy) but a steep drop-off in strategic reasoning (best model 46.6% next-action prediction accuracy, versus a 23.0% random baseline) and an even steeper drop in decision-making (best model 31.4% normalized return, far below the 100% oracle and below the 62.7% average normalized return achieved by human participants). Several models scored worse than random on decision-making in multiple games. The paper was accepted as a CVPR 2026 Oral presentation.
Task Anatomy
How a single task is structured.
Example Tasks
3 real examples from the benchmark.
Hanabi — Perception task
Problem / Input
Hanabi is a cooperative card game where each player's own cards are visible to teammates but hidden from themselves, creating an information-asymmetry challenge that the paper found many VLMs fail to account for in strategic reasoning.
Kuhn Poker — Perception task
Problem / Input
Kuhn Poker is used as a minimal competitive/imperfect-information game to test strategic reasoning under bluffing and hidden information.
Overcooked — Perception task
Problem / Input
Overcooked is a cooperative environment requiring two agents to coordinate on tasks (chopping, cooking, plating, serving) — used to test strategic reasoning about a teammate's likely next move and decision-making return from joint play.
Leaderboard Results
Model scores sorted by performance.
39 results
| # | Model | Score |
|---|---|---|
| 1 | o3 | 84.9% |
| 2 | Gemini 2.5 Pro | 83.4% |
| 3 | UI-TARS-1.5 | 81.5% |
| 4 | Claude 3.7 Sonnet | 81.1% |
| 5 | GPT-4.1 | 80.3% |
| 6 | Qwen2.5-VL-72B | 80.3% |
| 7 | Qwen-VL-Max | 80.2% |
| 8 | Doubao-1.5-Vision-Pro | 77.6% |
| 9 | Doubao-1.5-Thinking-Pro | 74.7% |
| 10 | QVQ-Max | 74.5% |
| 11 | InternVL3-78B | 74.1% |
| 12 | Grok-2 Vision | 70.2% |
| 13 | Llama 3.2 90B Vision | 67.8% |
| 14 | o3 | 46.6% |
| 15 | Claude 3.7 Sonnet | 40.4% |
| 16 | Gemini 2.5 Pro | 39.6% |
| 17 | UI-TARS-1.5 | 39.0% |
| 18 | Doubao-1.5-Thinking-Pro | 36.3% |
| 19 | GPT-4.1 | 36.0% |
| 20 | Qwen2.5-VL-72B | 32.2% |
| 21 | o3 | 31.4% |
| 22 | Qwen-VL-Max | 31.2% |
| 23 | QVQ-Max | 30.8% |
| 24 | Doubao-1.5-Vision-Pro | 30.1% |
| 25 | InternVL3-78B | 29.1% |
| 26 | Llama 3.2 90B Vision | 26.4% |
| 27 | Grok-2 Vision | 26.2% |
| 28 | Gemini 2.5 Pro | 23.2% |
| 29 | Doubao-1.5-Thinking-Pro | 20.3% |
| 30 | Claude 3.7 Sonnet | 18.1% |
| 31 | UI-TARS-1.5 | 12.9% |
| 32 | GPT-4.1 | 4.8% |
| 33 | QVQ-Max | 3.6% |
| 34 | Qwen2.5-VL-72B | 3.0% |
| 35 | InternVL3-78B | 1.2% |
| 36 | Qwen-VL-Max | 0.2% |
| 37 | Doubao-1.5-Vision-Pro | -0.9% |
| 38 | Llama 3.2 90B Vision | -6.3% |
| 39 | Grok-2 Vision | -6.7% |
V= Self-reported by the model's creator, not independently verified
Score Over Time
Performance progression across model generations.
Key Findings
Perception is largely solved by frontier VLMs (best model o3: 84.9% element-recognition accuracy) but strategic reasoning and decision-making are not: the same model drops to 46.6% next-action prediction accuracy and 31.4% normalized return.
Random-guess baseline for strategic reasoning is 23.0%; most models beat it in at least 8 of 10 games, showing 'non-trivial' but weak theory-of-mind capability.
Four of fifteen models scored worse than random (negative or near-zero normalized return) on decision-making, and several models scored below random on specific games (notably Atari Pong and Monster Hunt).
Human participants average 62.7% normalized return on decision-making; the best model (o3, 31.4%) only surpasses about 12.9% of human results, indicating VLMs remain far below human-level strategic play.
Documented failure modes include ignoring game history and private/hidden information, overlooking information asymmetry (e.g., in Hanabi), and focusing excessively on one's own actions while ignoring other agents' likely behavior.
The 'thinking'/reasoning-enabled variants of models (e.g., Gemini 2.5 with thinking vs. without, Claude 3.7 Sonnet with vs. without extended thinking) showed meaningfully better strategic reasoning and decision-making scores than their non-thinking counterparts.
Controversies & Caveats
Known limitations and criticisms.
The paper's own limitations section notes decision-making evaluation currently relies mainly on self-play or play against a single conventional agent rather than a diverse population of opponents, which may not fully capture robustness of strategic ability.
Most VS-Bench environments are two-player only; the authors acknowledge many real-world multi-agent scenarios involve more than two participants, and report only preliminary three-player decision-making experiments.
The authors note that in some settings, giving models text-only (non-visual) state descriptions performs slightly better than visual input, raising the question of whether the visual modality is adding meaningful difficulty versus just friction.
No major third-party contamination or design criticism was found as of this research pass (July 2026); the benchmark is recent (CVPR 2026 Oral) and public critical discussion appears limited so far.