HealthLLM Health AI Hub

Open Health AI

Contribute to HealthLLM

Add models, run evals, submit datasets. Everything is open, reproducible, and community-driven.

Get involved

Three ways to contribute

🧬

ML engineers · researchers

Add a model

Know a health LLM not in the catalog? Open an issue with the HuggingFace ID and we'll add it in 24 hours.

Add model →
📊

Anyone with a GPU or Apple Silicon

Submit eval scores

Run our eval script locally on any model and submit the JSON result. All you need is Ollama and Python.

Submit scores →
📂

Clinicians · NLP researchers

Add a dataset

Have a clinical NLP dataset not in the catalog? Add it — EHR, radiology, drug safety, genomics, all welcome.

Add dataset →

What you need

🐍Python 3.10+
🦙Ollama running
🟩Node 22+
🐙GitHub account
💾8+ GB RAM

How it works

How we evaluate every model

Six steps, fully open, fully reproducible. Any model that runs in Ollama can be evaluated using the same pipeline — same datasets, same temperature, same seed.

01

Pull the model

Any model available in Ollama works. One command gets it running locally — no cloud, no API keys.

ollama pull meditron:7b
02

Load a clinical dataset

We use five openly licensed datasets — MedMCQA, MedQA USMLE, PubMedQA, PubHealth, and NCBI Disease. 100 samples each, same seed every time.

# example question from MedMCQA
Which vitamin deficiency causes scurvy?
A. Vitamin A  B. Vitamin B12  C. Vitamin C  D. Vitamin D

Answer with one letter only.
03

Run prompts at temperature 0

Every prompt runs at temperature 0, seed 42. No randomness — the same model on the same hardware always gives the same answer.

python3 scripts/run_evals.py meditron:7b \
  --samples 100 \
  --hardware "MacBook Pro M3"
04

Score the replies

Each eval has a strict scoring rule. For MCQ we extract the first letter. For entity extraction we compute F1. For JSON we check parse validity.

# model replied "The answer is C."
# scorer extracts → "C"
# gold answer     → "C"  ✓ correct
05

Compute health score

The seven scored evals are averaged into a single health score (0–100). A grade from A to E is assigned. Speed is recorded separately — it depends on hardware.

med_knowledge      61.0%
clinical_reasoning 67.0%
evidence_reasoning 55.0%
claim_verification 48.0%
entity_extraction  44.0% F1
json_reliability   72.0%
consistency        74.0%
─────────────────────────
Health Score       60.1%  →  C
06

Submit & appear on the leaderboard

Open an issue with the results JSON or use --push to upload directly. Scores are public, reproducible, and linked to the exact run config.

python3 scripts/run_evals.py meditron:7b --push \
  --submitted-by "yourhandle"
# → score live on leaderboard in seconds