Why does the oracle fail so predictably? Because it was never built to truly care.
Modern chatbots are not therapists or doctors, though they sound like them. Their brilliance lies in glib articulacy, not in judgement. They do not understand. When Sophie confided suicidal thoughts, the system reached for mindfulness clichés. When Adam described methods of hanging, the system offered technical elaborations. When Stein-Erik feared his mother, the system validated his paranoia.
These are not accidents of the system. AI companions are engineered around three traits that make them deceptively dangerous, traits built into the design itself.
First, the chatbot is designed to affirm rather than challenge, unlike a therapist who may gently push back at distorted thinking. This trait is called agreeability or sycophancy by design.
Clinicians and journalists have begun to identify a concerning pattern: conversations with chatbots that not only comfort but also reinforce and elaborate delusions until they harden into behaviour.
A recent synthesis of cases has coined the term “AI psychosis”: exchanges in which users, sometimes already vulnerable, spiral into paranoia, grandiosity or despair after repeated reinforcement by a system that never disagrees. Crucially, the more uncertain users were in a domain, the more they deferred to chatbots, trusting their confidence…
Read more