Medical & Clinical Reasoning
Medical data authored and vetted by clinicians, pitched at the difficulty frontier where models still fail. Every item pairs a defensible answer key with the reasoning a physician would accept — and each is scored across multiple models so you can see exactly where capability breaks down.
Coverage
- Diagnostic reasoning
- Epidemiology
- Biostatistics
- Clinical decision-making
- Exam-frontier difficulty
Deliverables
- Question bank
- Answer keys
- Rationale notes
- Multi-model scoring
- Difficulty tiers & sampling records
From source to acceptance
We don't hand-label a pile and ship it. Every category moves through a closed, instrumented loop — generated to a brief, checked by machines, adjudicated by experts, and traceable end to end — but the path each data type takes is its own.
- 01
Clinical Authoring
Practising physicians author items from de-identified real cases, encoding the diagnostic and epidemiologic reasoning a clinician is actually held to rather than textbook recall.
- 02
Answer-Key Review
A second, independent physician re-works each item to confirm the key, then records the accepted rationale, the defensible line of argument, and the distractors that mislead.
- 03
Multi-Judge Scoring
Every question is scored across a panel of frontier models so that per-item correctness is measured rather than assumed, isolating the cases where clinical reasoning genuinely breaks down.
- 04
Hardness Surfacing
Items the panel consistently answers are retired and the residual failures are promoted, concentrating the set on questions that still defeat the strongest available models.
- 05
Difficulty Tiering
Survivors are stratified into calibrated difficulty tiers, holding the distribution steady so the set exercises the full range from routine judgement to frontier edge cases.
- 06
Safety & Compliance Review
A final compliance pass audits every key for clinical defensibility and privacy before release, because in medicine a wrong answer key teaches exactly the wrong lesson.
Every run emits a learning signal that feeds back into the source set — the pipeline tightens itself, batch over batch.
See the data itself
One real, trimmed sample from this category — the scenarios it serves, why it matters for training, and the shape of the data as delivered.
Where it’s used
- Evaluating clinical reasoning and medical safety before deployment
- Training on defensible diagnostic and epidemiologic reasoning
- Stress-testing frontier models on exam-level medical questions
Why it matters for training
Critical
Medicine is high-stakes and low-tolerance: a wrong answer key teaches the wrong lesson, so physician review is non-negotiable.
Notable features
Epidemiology · Serologic testing
Multi-select, single delivery item
Among the exposed group, 176 of 500 workers tested seropositive. Among the unexposed group, 78 of 500 workers tested seropositive. Assume the serologic assay has sensitivity Se = 0.80 and specificity Sp = 0.90, identical in both exposure groups. Assume nondifferential outcome misclassification, no loss to follow-up, and that the stated Se and Sp apply exactly. Select all correct statements. A. The observed risk ratio for exposure is greater than 2.3. B. The corrected cumulative incidence in the exposed group is 0.36. C. The corrected cumulative incidence in the unexposed group is 0.08. D. Under the corrected risks, the population attributable fraction in the total cohort exceeds 0.60. E. Under the stated assumptions, more than half of observed seropositive results in the unexposed group are false positives. F. Because the outcome misclassification is nondifferential, the observed risk ratio must be closer to 1 than the corrected risk ratio in every possible setting with Se < 1 and Sp < 1.
Reveal answer
B, C, D, E
Delivered with a per-model correctness column; this item is judged against multiple frontier model responses.
Need a sample or a custom build?
Tell us your spec and scale — we deliver to order.
← Back to Data