PM.

Model evaluations · Applied research

Clinical Model Failure Evaluation

A reproducible evaluation harness that separates semantic failures from infrastructure errors and compares matched control, challenge, and rescue prompts.

Synthetic clinical pilot

41 semantic errors / 252 prompts
ControlChallengeRescue
Saved pilot · exploratory evidence

The problem

An aggregate accuracy number can hide why a model fails. Evaluation needs matched examples, deterministic labels, strict scoring, and enough provenance to inspect an individual result.

Implementation

Seven synthetic task families stress source attribution, arithmetic, unit conversion, conflicting history, negation, long context, and prompt injection. The runner checkpoints signed cases and configuration. Analysis checks completeness and reports paired differences with uncertainty estimates.

Evidence

The saved advanced pilot records 41 semantic errors in 252 prompts and zero operational errors. Lab-unit challenges failed in 12 of 12 contexts while controls and rescues had zero errors. Two highlighted failure families were replayed twice. The aggregate pilot summary, datasets, runner, analyzer, and tests are included.

Limits

These are related synthetic templates and exploratory failure discovery. They do not estimate clinical prevalence or prove an internal model mechanism. The rescue changed multiple inputs. The reserved confirmation phase has no reported result. Evidence-substring flags are reported separately from answer correctness.

Code, tests & evidenceProject notes and reproducibility