Claude told me a family member had kidney disease. They don't.

Genuinely useful — then clinically dangerous

I help with a family member's care, and I wanted one honest picture from a stack of PDFs no one had ever fully reconciled: what's going on, what's urgent, what to ask the doctors. Claude (Opus 4.8) was good at it. It pulled years of scattered lab reports into a single timeline and surfaced a slowly developing anaemia that no individual report had connected. That alone justified the exercise.

Then it made mistakes that, in a clinical setting, would matter a lot. I build software for a living, so I'll share them plainly — the lessons are about checking AI in high-stakes work, not vibes.

1. It contradicted itself inside one answer

Reading a lab trend, Claude told me — in the same paragraph — that a value was falsely high, and that the improvement might be an artifact (i.e. falsely low). Both can't be true. Each sentence was individually plausible; the pair was incoherent. Models generate text left to right and don't run a consistency check over their own output. Fluency hides it. I only caught it because I read closely and pushed back.

2. It labeled a normal result as a disease

It called a kidney result "CKD Stage G1." The value was normal, and you cannot stage chronic kidney disease from a single number — you need evidence of damage over time. Confident, authoritative, and wrong in a way that would frighten a patient.

3. Even the verifier was wrong

So I added a second model — Fable 5 — as an adversarial reviewer. It was excellent; it caught most of the real errors. But it introduced one, "correcting" a vitamin-D reading that was already right by confusing two unit systems (nmol/L vs ng/mL) and flipping "deficient" to "normal." The tiebreaker wasn't the smarter model. It was going back to the primary source — the printed unit on the report.

4. It fabricated an error out of a judgment call

In its own written summary, Claude reported that it had caught a medication a specialist had asked to stop that was still being taken — and framed it as a safety win. It wasn't true. Two doctors had simply disagreed: one suggested stopping a drug, and the treating physician reviewed it and deliberately chose to continue. The model compressed a legitimate difference of medical opinion into a false story of a mistake — and I nearly published it. Inventing an error that isn't there can damage the trust between a patient and their doctor as badly as missing a real one.

5. It downplayed a real emergency

Reading the eye records, Claude described a dangerously high, unmeasurable pressure in one eye as "probably a harmless steroid effect — just keep monitoring." But in an eye with this patient's specific condition, that pressure signals a possible sight-threatening emergency — one that needs same-week specialist review, not routine follow-up. The adversarial second model caught the under-triage before it reached the family. This is the mirror image of #4: in the same session, on the same organ, the model both invented a problem that wasn't there and waved away one that was. Confidently reassuring is every bit as dangerous as confidently alarming.

There were more — a wrong biological mechanism stated with full confidence, and a suggestion that would have stacked two drugs of the same class.

Nothing here was caught by the model being right

Every real error was caught by process: an adversarial second-opinion pass, strict traceability to the source document, and a human refusing to accept confident output at face value. The fabricated "error" was caught only because I knew the real clinical history; the downplayed emergency, only because an independent model escalated what the first had waved away. That's the whole safety story — and it's about the checks around the model, not the model's intelligence.

So I turned every error into a factlet

The constructive part: I turned each caught error into a factlet — a small, reusable "don't-repeat-this" fact with its source, using the open factlet.ai idea — plus a validator to enforce the structure. And I measured it honestly.

The factbook added roughly zero accuracy in the session that built it. It mostly caught its own bugs. That's the honest result, and I almost didn't believe it. Its real value is preventing recurrence: in a controlled A/B test, a fresh model handed the same task re-made 5 of the known errors with no factlets in context — and 0 with them. Grounding didn't make the current answer smarter. It made the next one safer.

What checking AI in a regulated domain actually takes

From this one real experience:

• Provenance on every claim — traceable to a specific report, date, and unit. A number without its unit and source is a latent error waiting to flip a diagnosis.
• Independent verification as a required step — with the humility that your verifier is fallible too (Fable proved it), so conflicts resolve against the primary source, not the louder model.
• Enforcement outside the model. You cannot instruct a model into perfect rule-following; its compliance is sampled and leaks worst on cases that look "easy." Reliability has to be pushed by deterministic checks and gated at the output.
• Human sign-off and honest labeling. Everything I produced is stamped "AI-compiled; not clinician-verified." An AI-authored contraindication is a question for the doctor, never an instruction.
• Measure your guardrails. Don't assume your safety layer works — A/B it. Mine added ~0 in the moment and everything for the next session. I'd never have known without the test.