Model evaluations · Applied research
Clinical Model Failure Evaluation
A reproducible evaluation harness that separates semantic failures from infrastructure errors and compares matched control, challenge, and rescue prompts.
Synthetic clinical pilot
The problem
An aggregate accuracy number can hide why a model fails. Evaluation needs matched examples, deterministic labels, strict scoring, and enough provenance to inspect an individual result.
Implementation
Seven synthetic task families stress source attribution, arithmetic, unit conversion, conflicting history, negation, long context, and prompt injection. The runner checkpoints signed cases and configuration. Analysis checks completeness and reports paired differences with uncertainty estimates.
Evidence
The saved advanced pilot records 41 semantic errors in 252 prompts and zero operational errors. Lab-unit challenges failed in 12 of 12 contexts while controls and rescues had zero errors. Two highlighted failure families were replayed twice. The aggregate pilot summary, datasets, runner, analyzer, and tests are included.
Limits
These are related synthetic templates and exploratory failure discovery. They do not estimate clinical prevalence or prove an internal model mechanism. The rescue changed multiple inputs. The reserved confirmation phase has no reported result. Evidence-substring flags are reported separately from answer correctness.