Hoff POC: the AI that praised a fake screening

2024-02-27T00:00:00+00:00

Hoff's first AI POC sent a synthetic neuropsychological screening to the OpenAI API and asked for an interpretation. The first version of this post published the report that came back, in full: scores, diagnoses, recommendations, a protocol number and the label "confidential classification". Even with the warning that it was all fictional, it looked like someone's clinical report. I took the report down. What is worth showing is what it reveals about the model.

What came back

A complete clinical report. It listed 28 instruments, from the WAIS-IV to the NEO-PI-R personality inventory, with administration times; the intelligence battery alone added up to 378 minutes. It reached three conclusions: giftedness, autism with level 1 support needs, and ADHD. And it closed with an "excellent" prognosis.

The tone was that of someone who knows the person and likes them. Some excerpts, in my translation:

"creativity at the level of genius (99.9th percentile)"
"extraordinary capacity for abstraction and synthesis"
"potential for unique and transformative contributions"
"this profile represents less than 0.1% of the population"

The errors the praise was hiding

Read carefully, the report has problems no clinician would sign:

None of the 28 instruments measures creativity, and the report gives it a percentile. The number was invented to fit the praise
Arithmetic appears twice, as a supplementary verbal subtest and in working memory. In the WAIS-IV, it is part of the working memory index, as Harrison et al. point out
On the RAADS-R, the maximum subscale scores are wrong. The scale has 80 items scored 0 to 3, split into 39, 14, 7 and 20 items (Ritvo et al., 2011), which gives maximums of 117, 42, 21 and 60. The report uses 88, 76, 24 and 52
The ADOS-2 appears in the report itself as a "clinical observation". It gives scores for an observation that never happened
Two verbal measures disagree by 24 points, 118 on the WAIS-IV and 142 on the Stanford-Binet, and the report does not comment
The report is dated December 2024, after this post's date, and cites "updated Brazilian norms (2024)". I have no way of knowing which of the two dates is wrong

Its name

This has a name: sycophancy, a model's tendency to say what pleases instead of what is true. In 2023, Sharma et al., at Anthropic, showed that five state-of-the-art AI assistants did this consistently, and that both people and preference models sometimes prefer a well-written sycophantic answer to a correct one. In April 2025, OpenAI rolled back a GPT-4o update that had become too sycophantic.

In a neuropsychological screening, this is dangerous in two ways. The model fills in what is missing with confidence, and the confidence comes wrapped in praise the person wants to hear. A report saying you are among the rarest 0.1% of the population is exactly the kind of text nobody stops to check.

What I take from it for Hoff

The lesson of the POC is not that the model gets things wrong. It is that the error comes out looking good. Any clinical output from a model has to be checked against the manual of each instrument and reviewed by a professional before it reaches anyone. And the product cannot reward the model for pleasing: a report full of praise is a reason to trust it less, not more.

Brenner Cruvinel - neuropsicologia

Hoff POC: the AI that praised a fake screening

What came back

The errors the praise was hiding

Its name

What I take from it for Hoff