HealthAgentBench
Microsoft Research suite of 54 agentic healthcare tasks across 7 real clinical/biomedical workflows (EHR auditing, X-ray report correction, clinical trial matching, pathology, CT), scored by task-specific verifiers with cost/time tracking.
Overview
HealthAgentBench is a unified benchmark suite from Microsoft Research (lead authors Qianchu Liu, Sheng Zhang, Guanghui Qin, Hoifung Poon and collaborators) that evaluates frontier AI agents on realistic, end-to-end healthcare workflows rather than static medical QA. Each of its 54 tasks, spanning 7 categories (EHR format conversion, X-ray report correction, CT abnormality classification, pathology tumor-area selection, clinical trial matching, EHR data-quality auditing, and EHR event modeling), drops an agent into a containerized terminal environment built on the Harbor framework, where it must inspect real clinical data (drawn from credentialed sources like MIMIC-IV, MIMIC-CXR, and TREC Clinical Trials), use tools, and produce a verifiable output judged by a task-specific automated verifier or LLM-as-judge.
The benchmark was built to address the gap between narrow medical QA leaderboards and the messy, multimodal, long-horizon nature of actual clinical and biomedical work — covering modalities from 2D X-rays and 3D CT volumes to gigapixel pathology slides and structured EHR data, with documented expert time-to-solve reference points (e.g. clinical trial matching takes experts over 8 hours; X-ray report correction 15–60 minutes).
Results show current agents remain far from reliable: the best agent, Codex GPT-5.5, solves only 42% of tasks overall (at roughly $0.70/task), with EHR/text-based tasks averaging 49% success but medical imaging tasks averaging just 17%. The paper also surfaces a notable cost/performance disconnect — Claude Code agents were both the slowest and most expensive (up to $4.80/task) yet scored below cheaper competitors, largely due to producing ~2x more output tokens per trial.
How It Works
Task setup, inputs, outputs, and evaluation.
Results
Model performance on this evaluation.
| # | Model | Score |
|---|---|---|
| 1 | Gpt 5 5 | 42% |
| 2 | Claude Opus 4.7 | 36% |
Key Findings
Best agent (Codex GPT-5.5) solves only 42% of tasks overall; EHR/text tasks average 49% success vs. just 17% for medical-imaging tasks.
Claude Code agents were both the slowest and most expensive (up to $4.80/task) yet underperformed cheaper models, producing ~2x more output tokens per trial.
EHR data-quality auditing success jumps from 42% to 97% when search clues are provided, showing large search space (not reasoning) is the primary bottleneck.
9 of 10 agents achieve 100% success on EHR format conversion, showing strong performance on pure text/structured-data transformation but sharp drop-off on multimodal imaging.
What Makes It Unique
- ✓
Spans the full patient-journey and data-modality range (2D/3D imaging, whole-slide pathology, structured EHR, free text) in one suite, unlike single-modality medical benchmarks.
- ✓
Reports per-task dollar cost and wall-clock time alongside success rate, enabling Pareto-frontier cost/performance analysis.
- ✓
Uses real credentialed clinical datasets (MIMIC-IV, MIMIC-CXR, TREC Clinical Trials) with anti-cheat measures (scrubbed IDs, credential isolation, no web access) to prevent contamination.
Controversies & Caveats
Requires credentialed access to PhysioNet datasets (MIMIC-IV/MIMIC-CXR), limiting fully open reproducibility.
Some task categories (CT classification, pathology) have small task counts, so per-category scores carry high variance.