Healthcare Bench AIIndependent benchmark analysis / 300-task paper; arXiv v2
MedAgentBench turns an instruction into a sequence of requests against a virtual FHIR record environment. Its useful distinction is between returning information and preparing an action. We analyze the 300-task paper snapshot, including the separate query and action results. Our central interpretation is that an aggregate score hides a decision about workload composition. The baseline also checks proposed write payloads without executing them against the server, which places a precise boundary around the word success. This is an independent reading of the authors’ evaluation, with no new model runs.
01 / What is being tested?
Data origin. Physician-authored tasks grounded in deidentified Stanford STARR records; synthetic identifying details and patient-level date jittering. [1][2][3]
150 query tasks and 150 action tasks.
Section 2.5.1 [1]Profiles are shared across tasks; tasks are not 300 independent patients.
Table 2 [1]Four reported FHIR resource groups.
Table 2 [1]Grouped into seven broad categories in Table 1.
Table 1 [1]Invalid actions or exceeding the budget count as failures.
Sections 2.4.1–2.4.3 [1]Read instruction and hospital context before choosing a FHIR function.
[1]GET requests return raw server responses to the agent.
[1]POST payloads receive local checks and a simulated success response; baseline writes are not executed.
[1]Compare query answers with reference solutions; apply authored checks to action payloads.
[1]Dataset anatomy
GET-only task subgroup.
Requires a proposed POST, sometimes after retrieval.
Counts describe the 300-task paper; they are not deployment traffic weights. Bar lengths use the largest listed count as their reference; they are not percentages of a shared population.
02 / Measurement
Higher is better within the same task/configuration.
Rule-based success for one attempt; query answers and proposed action payloads have different checks.
SR = successful task attempts / all task attempts × 100
Pin task version, orchestrator, parser, reference solution and interaction cap. This is not a patient-outcome rate. [1]
03 / Measured evidence
Paper-reported results / selected rows
arXiv v2, February 2025; 150 queries, pass@1, eight-round baseline; temperature zero for these selected models.
Historical selected rows. Table 3 does not provide uncertainty intervals.
Source: Table 3 [1]
Paper-reported results / selected rows
Same paper snapshot and orchestrator; 150 action tasks, with proposed POST payload validation.
Keep this separate from query scores. No deployment reliability or current model ranking is implied.
Source: Table 3 [1]
04 / Our original analysis
Claude’s query/action scores are 85.33/54.00; Gemini-1.5 Pro’s are 52.67/71.33. [1]
A read-heavy workflow and a write-heavy workflow can favor different systems. Equal weighting is a benchmark design choice, not a universal utility function.
The baseline sends GETs to the server but validates POST payloads locally. [1]
A passing payload leaves server validation, persisted state and downstream execution untested. Treat these as additional integration questions.
300 tasks use 100 patient profiles. [1]
A task-level denominator describes attempts; it does not establish evidence from 300 distinct patients or institutions.
Invalid actions are failures under the specified parser. [1]
Model knowledge and orchestrator compatibility jointly affect this result. Changing a parser changes the evaluated system.
05 / Scope of the evidence
STARR profiles do not establish transportability to other EHR systems or populations. [1]
The task inventory does not cover all specialties, teamwork or the consequences of real actions. [1]
The January 2025 v1 evaluated 100 tasks; do not combine its scores with these 300-task rows. [2][1]
MIT repository licensing does not by itself establish unrestricted reuse of patient-derived records. [3]
Evidence trail
Jiang, Black, Geng et al.; Stanford University. Primary 300-task paper; historical results, not a current leaderboard.
Jiang, Black, Geng et al.; Stanford University. Version history only; its scores must not be mixed with the 300-task evaluation.
Stanford Machine Learning Group. Repository MIT license applies to code; repository access is not a separate patient-data license.
Benchmark authors retain authorship of their work. This publication provides independent analysis; published rows are not Arcophos evaluation runs. Editorial method.