For clinicians who buildMD-dev

what changed for clinicians of any background building their own software with ai — coding agents, models, health data access, privacy & the rules — checked daily, each note against at least two sources


Study compares cloud and in-house open-weight models for flagging urgent portal messages

A Mass General Brigham team led by Valdery Moura Junior and Lidia Moura published a study in npj Digital Medicine comparing six language models for flagging urgent patient-portal messages, tested on primary-care, epilepsy-clinic and public patient messages. Three were cloud models reached through Microsoft's Azure OpenAI service (GPT-4o, GPT-5, GPT-5 mini). The other three were open-weight models, meaning their files can be downloaded and run on your own computers: Llama 4 Scout, GPT-OSS-20B and Gemma 3 27B, run entirely inside the hospital's systems with the Ollama or vLLM serving tools. In the earlier preprint, which covered only 503 de-identified epilepsy-clinic messages scored against five epilepsy-trained neurologists, five of the six models caught every message the neurologists unanimously called urgent, while GPT-OSS-20B missed several but raised fewer false alarms. The published paper says the models can rule out urgent messages with high sensitivity but that specificity remained limited, and that one model that can run on-premises reached high sensitivity in the primary-care set. The preprint noted a single clinic, zero-shot prompting and a low share of urgent messages (about 3%) as limitations.

Why it's notable: This is published evidence that a mid-sized open model run on in-house hardware can screen out non-urgent inbox messages with high sensitivity without patient messages leaving the institution, though with more false alarms. A clinician building something similar should still test on their own clinic's messages and decide in advance how many missed urgent messages, versus false alarms, they can accept.

#npj Digital Medicine#Mass General Brigham#patient portal#open-weight models
Sources: npj Digital Medicine, Research Square · last verified 2026-11-22

Markdown version for pasting into your own agent or notes

← All notes More in Studies & evidence →