InclusionMed: Benchmarking AI for Inclusive Healthcare
The InclusionMed Collaboration introduces an open evaluation framework to audit and reproduce medical AI benchmarks across diverse healthcare roles and capabilities, revealing that model rankings vary by context and inference costs differ substantially.
The InclusionMed Collaboration introduces InclusionMed, an open evaluation initiative designed to audit and reproduce medical AI benchmarks across diverse healthcare roles and capabilities. The initiative establishes the InclusionMed Atlas, which organizes benchmarks along a matrix of healthcare roles (patients and the public, healthcare professionals, healthcare administration professionals) and AI capabilities (knowledge, reasoning, agency). Selecting four public benchmarks from 388 candidates through structured audits and independent reproduction, alongside a private diagnostic reasoning benchmark called InclusionMed-DxFiles, the initial Atlas covers five benchmarks across three role-capability cells. Evaluating frontier large language models on the Atlas yielded the InclusionMed Index, with overall scores ranging from 17.5 to 49.1 out of 100. The analysis demonstrates that a model's performance rank varies significantly depending on the specific role or capability evaluated, and the ten highest-ranked models consistently scored higher on patient communication than on clinical decision-making. Furthermore, models with comparable Index scores showed substantial differences in estimated inference costs, highlighting the critical need for context-specific model selection and broader benchmark coverage in medical AI.
View selection rationale
Importance 77/100
About this brief
- Content type
- Research summary
- Source type
- First-party / official source
- Original source
- Hugging Face Daily Papers ↗
- Source published
- Process
- AI-assisted discovery, translation, summarisation, classification and scoring; standards and corrections are maintained by the editorial desk.
- Verification
- The original URL and publication metadata are retained. This brief is not independently fact-checked unless explicitly stated.
- Published by
- Polyemergence Editorial Desk · PolyEmergence Pte Ltd.
- Conflict disclosure
- No financial or commercial conflict declared.
