Open Health AI
The benchmark for health AI
Browse 329 open health LLMs and 266 datasets. Run reproducible evals. Compare clinical reasoning, medical knowledge, and RAG reliability — all at temperature 0.
- 329
- Health LLMs
- 266
- Datasets
- 7
- Reproducible evals
- 0°
- Temperature — always
The problem
Why this exists
Health AI needs transparency
Models are being deployed in clinical tools, triage systems, and patient-facing apps. Without open, reproducible benchmarks it is impossible to know which models are actually safe to use with health data.
General benchmarks miss the point
A model that scores well on MMLU can still fail at clinical reasoning, misidentify diseases, or hallucinate drug interactions. We test specifically for healthcare: facts, reasoning, evidence, and structured output.
Open data, open scores
Every eval uses a publicly licensed dataset. Every score comes from a results file checked into this repo. No proprietary data, no closed judging — just runs anyone can reproduce and verify.
RAG changes the picture
Most health deployments use retrieval-augmented generation (RAG) to ground answers in clinical documents. We test JSON reliability and claim verification — the two capabilities that break first in real RAG pipelines.
Stay in the loop
Get notified of updates
New health models, datasets, and leaderboard scores — delivered to your inbox weekly. No spam.
Get involved
Three ways to contribute
Fork on GitHubML engineers · researchers
Add a model
Know a health LLM not in the catalog? Open an issue with the HuggingFace ID and we'll add it in 24 hours.
Add model →Anyone with a GPU or Apple Silicon
Submit eval scores
Run our eval script locally on any model and submit the JSON result. All you need is Ollama and Python.
Submit scores →Clinicians · NLP researchers
Add a dataset
Have a clinical NLP dataset not in the catalog? Add it — EHR, radiology, drug safety, genomics, all welcome.
Add dataset →How it works
How we evaluate every model
Pull the model
Any model available in Ollama works. One command gets it running locally — no cloud, no API keys.
ollama pull meditron:7bLoad a clinical dataset
We use five openly licensed datasets — MedMCQA, MedQA USMLE, PubMedQA, PubHealth, and NCBI Disease. 100 samples each, same seed every time.
# example question from MedMCQA
Which vitamin deficiency causes scurvy?
A. Vitamin A B. Vitamin B12 C. Vitamin C D. Vitamin D
Answer with one letter only.Run prompts at temperature 0
Every prompt runs at temperature 0, seed 42. No randomness — the same model on the same hardware always gives the same answer.
python3 scripts/run_evals.py meditron:7b \
--samples 100 \
--hardware "MacBook Pro M3"Score the replies
Each eval has a strict scoring rule. For MCQ we extract the first letter. For entity extraction we compute F1. For JSON we check parse validity.
# model replied "The answer is C."
# scorer extracts → "C"
# gold answer → "C" ✓ correctCompute health score
The seven scored evals are averaged into a single health score (0–100). A grade from A to E is assigned. Speed is recorded separately — it depends on hardware.
med_knowledge 61.0%
clinical_reasoning 67.0%
evidence_reasoning 55.0%
claim_verification 48.0%
entity_extraction 44.0% F1
json_reliability 72.0%
consistency 74.0%
─────────────────────────
Health Score 60.1% → CSubmit & appear on the leaderboard
Open an issue with the results JSON or use --push to upload directly. Scores are public, reproducible, and linked to the exact run config.
python3 scripts/run_evals.py meditron:7b --push \
--submitted-by "yourhandle"
# → score live on leaderboard in secondsWhat you need