Open Health AI
Contribute to HealthLLM
Add models, run evals, submit datasets. Everything is open, reproducible, and community-driven.
Get involved
Three ways to contribute
ML engineers · researchers
Add a model
Know a health LLM not in the catalog? Open an issue with the HuggingFace ID and we'll add it in 24 hours.
Add model →Anyone with a GPU or Apple Silicon
Submit eval scores
Run our eval script locally on any model and submit the JSON result. All you need is Ollama and Python.
Submit scores →Clinicians · NLP researchers
Add a dataset
Have a clinical NLP dataset not in the catalog? Add it — EHR, radiology, drug safety, genomics, all welcome.
Add dataset →What you need
How it works
How we evaluate every model
Six steps, fully open, fully reproducible. Any model that runs in Ollama can be evaluated using the same pipeline — same datasets, same temperature, same seed.
Pull the model
Any model available in Ollama works. One command gets it running locally — no cloud, no API keys.
ollama pull meditron:7bLoad a clinical dataset
We use five openly licensed datasets — MedMCQA, MedQA USMLE, PubMedQA, PubHealth, and NCBI Disease. 100 samples each, same seed every time.
# example question from MedMCQA
Which vitamin deficiency causes scurvy?
A. Vitamin A B. Vitamin B12 C. Vitamin C D. Vitamin D
Answer with one letter only.Run prompts at temperature 0
Every prompt runs at temperature 0, seed 42. No randomness — the same model on the same hardware always gives the same answer.
python3 scripts/run_evals.py meditron:7b \
--samples 100 \
--hardware "MacBook Pro M3"Score the replies
Each eval has a strict scoring rule. For MCQ we extract the first letter. For entity extraction we compute F1. For JSON we check parse validity.
# model replied "The answer is C."
# scorer extracts → "C"
# gold answer → "C" ✓ correctCompute health score
The seven scored evals are averaged into a single health score (0–100). A grade from A to E is assigned. Speed is recorded separately — it depends on hardware.
med_knowledge 61.0%
clinical_reasoning 67.0%
evidence_reasoning 55.0%
claim_verification 48.0%
entity_extraction 44.0% F1
json_reliability 72.0%
consistency 74.0%
─────────────────────────
Health Score 60.1% → CSubmit & appear on the leaderboard
Open an issue with the results JSON or use --push to upload directly. Scores are public, reproducible, and linked to the exact run config.
python3 scripts/run_evals.py meditron:7b --push \
--submitted-by "yourhandle"
# → score live on leaderboard in seconds