Benchmarks
Every major LLM benchmark explained — what it tests, how tasks work, and where models stand.
AgentBench
LLM agents across 8 interactive environments: OS, databases, web, games, and more.
AIME
Annual olympiad-level math competition used as a fresh, contamination-proof AI benchmark.
ARC-AGI-2
Second-generation abstract visual reasoning benchmark testing symbolic interpretation, compositional rule application, and context-aware reasoning. Pure LLMs score 0%; humans solve every task in ≤2 attempts.
ARC-Challenge
Grade-school science questions that simple retrieval systems can't answer.
ARC-Challenge + HellaSwag
Two classic saturated benchmarks — ARC tests grade-school science reasoning (2,590 questions), HellaSwag tests commonsense sentence completion (10,000+ questions). Both exceeded by frontier models at ~95–98% accuracy.
BIG-Bench Hard
23 hard reasoning tasks where chain-of-thought is required to exceed human performance.
BigCodeBench
1,140 practical coding tasks requiring invocation of diverse library APIs across 139 libraries; models score ≤60% vs. 97% human baseline. Accepted as ICLR 2025 oral.
FrontierMath
Hundreds of novel, research-grade mathematics problems across number theory, real analysis, algebraic geometry, and category theory, authored by expert mathematicians. Current best AI solves <2%.
GAIA
Multi-step real-world tasks that are conceptually simple for humans but require tool-using agents.
GPQA Diamond
PhD-level science questions so hard that even experts with Google still struggle.
GSM8K
Grade-school math word problems requiring 2-8 step arithmetic reasoning.
HellaSwag
Commonsense reasoning — pick the most plausible continuation of an everyday activity.
HLE
2,500 expert-crafted academic questions spanning math, natural sciences, and humanities, designed to be Google-proof and exceed frontier model capabilities. Published in Nature (Jan 2025).
HumanEval / HumanEval+
Python function completion from docstrings, evaluated by test execution.
LiveBench
Contamination-resistant benchmark refreshed monthly from recent sources with no LLM judge.
LiveCodeBench
Contamination-resistant coding benchmark using freshly released competition problems.
LMSYS Chatbot Arena
Crowdsourced human preference Elo ratings from millions of real user comparisons.
MATH Benchmark
Competition-level math problems across 7 subjects, from AMC to AIME difficulty.
MMLU / MMLU-Pro
Broad academic knowledge across 57 subjects — the standard knowledge benchmark.
MMMU-Pro
Hardened extension of MMMU with 10-option MCQs, vision-only perceptual tasks, and multi-image questions. Models score 16–27%, vs. ~56% on the original MMMU.
OSWorld
369 real-world computer-use tasks on Ubuntu/Windows/macOS requiring GUI navigation, desktop app control, and multi-app workflows. Humans succeed at 72%; best model achieved 12% at launch (NeurIPS 2024).
SWE-bench
Can AI resolve real GitHub issues on production codebases?
SWE-Lancer
Real Upwork freelance software tasks mapped to $1M in economic value.
TheAgentCompany
A simulated software company with 16 AI colleagues testing real office work tasks.
TheAgentCompany
175 professional workplace tasks in a simulated software company environment with 16 AI colleagues, 4 integrated platforms, and 6 job role domains. Best agent: ~43%. Published NeurIPS 2025.
TruthfulQA
Can AI avoid repeating common myths and falsehoods that pervade its training data?
Video-MME
Comprehensive multi-modal LLM video benchmark spanning 30 subfields and 11-second to 1-hour durations, with subtitle and audio-integrated questions. Accepted CVPR 2025; adopted by OpenAI as industry standard.
WebArena / VisualWebArena
Autonomous browser agents completing realistic tasks on functional sandboxed websites.
τ-bench (tau-bench)
AI customer service agents that must follow policy while solving real customer problems.