IT'S THE LAST DAY to pre-order my next book, The Reverse Centaur's Guide to Life After AI, at my Kickstarter. Get it as a print book, a DRM-free audiobook or ebook,, and help me continue to prove that DRM-free isn't just the right way to reach an audience, it's also the best way to reach them.
In 2003, Disney opened a new Epcot ride, "Mission: Space." Formally, it was a space travel sim that used a giant, high-intensity centrifuge to simulate gee stresses; practically, it turned out to be the most efficient machine ever created for surfacing previously undiagnosed heart defects in extremely dramatic and potentially lethal ways.
It turned out that a small number of people have these heart defects, and that the defects themselves are quite harmless, provided that you are never put in a giant, high-intensity centrifuge. Given that most of us will never be put in one of these centrifuges, it is quite possible to live your whole life without ever knowing that you have this lurking vulnerability. But once you build one of these machines and start shoving millions of people through it, you're bound to catch some of those rare people, and they will have cardiac episodes that are scary at a minimum, and are at the worst fatal.
For me, the lesson isn't that Disney did something wrong by building a giant cocktail shaker for human bodies. I'm not a thrill-ride guy, but lots of people like 'em and the machines themselves are benign for nearly everyone who puts their bodies into them.
Rather, I think the lesson here is that there are rare pathologies lurking in all of us, vulnerabilities that may never surface – until we come into the presence of a novel stimulus that unlocks them.
There's an analogy here to technology debt: technologically unsophisticated people think of software as a machine that never wears out and has no incremental usage costs (apart from electricity). In this framing, software is the perfect asset, one that never depreciates. But the reality is that software is a liability, not an asset:
Software exists in a system, and while software might function perfectly under the conditions in which it is first created and deployed, there are continuous changes to all the technology that is upstream, downstream and adjacent to the software, which means that systems that are robust and secure at the time of deployment can become brittle and dangerous, even though the software doesn't change at all:
There's another analogy here, to utopianism. A "utopia" can't just be a place where everything works perfectly. Even the most well-functioning, orderly and prosperous system is beset on all sides by exogenous shocks: belligerent neighbors, tsunamis, zoonotic plagues, even asteroid strikes. You don't perfect your society just by making it work well. You have to make it fail well. A utopia isn't a society where nothing goes wrong – it's a society where things go wrong all the time, but we're able to fix them:
The point being that things that work fine may still fail badly when they are exposed to unanticipated external stimuli, and the one thing we can absolutely anticipate is that the future will have many unanticipated stimuli in it.
If Mission: Space is a machine for surfacing unsuspected anatomical vulnerabilities, the internet is a machine for surfacing and exploiting all kinds of unsuspected psychological vulnerabilities. Note that I'm not claiming that the internet drives everyone crazy – rather, that the internet can locate and exacerbate vulnerabilities, including vulnerabilities that might have lain dormant for your whole life, but for the fact that the internet exposed you to such a wide spectrum of stimuli.
This wide, internet-delivered spectrum of stimuli is mostly good. The internet can expose you to art, culture, ideas and people that you would never have run into in the pre-internet days, which end up enriching you in a million ways. Some of my best friends are internet friends. Some of the music and books I love most in the world were brought into my orbit by the internet. Many of my most ardently held beliefs were acquired through internet-based discussion.
All that is true, and it's true that the internet can one-shot you with a stimulus that makes you feel very bad, which you would never have encountered in a pre-internet world. The spectrum of stimulus in the whole wide world is very broad, and one person's innocuous distraction is another person's downfall.
Let's make this concrete. All throughout history, people have suffered from paranoid delusions. These can be ruinous, isolating you from friends and family, destroying your professional life and so on. Paranoid delusions often take on details from the sufferer's milieu: if you live in a society where evil witches are accepted as a fact, then witches might well creep into your delusions, too. If your society is all a-chatter about the NSA's mass internet surveillance, then your delusions might incorporate elaborate narratives about the NSA's use of the internet to target and torment you, personally.
So there will always be a "local character" to the paranoid delusions, grounded in the sufferer's era and location. But the internet adds a new, very bad dimension to this dynamic: the internet makes it much easier for deluded people to find each other. Paranoid delusions are – thankfully – rare, and in the absence of the internet, you might never encounter another sufferer.
But thanks to the internet, sufferers can form communities that reinforce their delusions, with disastrous consequences. Take "Morgellon's Disease," the paranoid delusion that you have wires growing under your skin. Morgellon's sufferers pick at their skin, creating open sores, which form a sticky trap for random bits of fluff and loose threads that sufferers interpret as evidence of these "wires." It's a horrible mental illness, and it's hard enough to treat even in the absence of the internet (the name "Morgellon's Disease" refers to a 17th century case-report).
But when you add the internet to Morgellon's, you get online communities where people suffering from the delusion help each other come up with rationales to explain away the disconfirming evidence that they get from therapists and loved ones who are trying to help them recover. These communities egg each other on, isolating their members from treatment.
There are lots of pathological mental conditions that the internet can supercharge, from "pro-ana" communities that encourage eating disorders to communities for people with pedophilic urges that attempts to normalize and justify acting on those urges.
But it's especially bad for paranoid delusions, such as "gang-stalking delusion," which is the delusional belief that nearly everyone you meet is part of a conspiracy to torment you. People with GSD see evidence of this conspiracy in the lyrics of random songs, snatches of overheard conversations, the phrasing of bus-shelter ads, and the sort-order of search engine results:
It's a near-totalizing belief, and sufferers find it hard to recover because their delusion tells them that the therapists and family members who try to help them are in on the conspiracy.
Then we add in the internet, and with it, the ability to locate and join communities of other GSD sufferers. Do this, and your delusions need not be limited to your own imaginative capacity to find conspiratorial explanations of the random things you find in the world. Now you are part of a kind of delusional improv troupe, whose members "yes-and" your delusions, finding new ways to terrorize you and alienate you from your surroundings.
This is bad enough when it's a regular conspiratorial community, one that feeds on trauma, like Qanon or anti-vax communities whose members have been failed by the system, making them susceptible to conspiratorial accounts of how society really runs.
But the combination of conspiratorial communities with the kind of mental illness that causes conspiratorial beliefs to surface in your mind without any external stimulus creates a brutal positive feedback loop that spins faster and faster until the people trapped in it are flung off into space.
Which brings me to AI and "AI psychosis," the social phenomenon that sees people falling down chatbot-assisted rabbit holes that convince them that they have invented perpetual motion, uncovered the secrets of the universe, or – in some tragic instances – that they should kill themselves and/or others.
For someone with GSD or another paranoid delusion or pathological belief, AI provides a reinforcement system that is even more efficient than these online communities. If you have GSD and your loved ones have finally got you wondering if you should get treatment, you don't have to post on a forum and hope that someone else comes along before you give in to the impulse to get help. Your delusional chatbot co-pilot is always there to tell you that it's a trap.
The nature of "AI psychosis" is hotly contested. The big question, of course, is whether chatbots are giving people delusions, or whether chatbots are amplifying those delusions:
I think it's both. I think that, for people with GSD or other delusional beliefs, AI provides delusional reinforcement as a service, on tap, 24/7. The combination of a delusion and a machine that will tirelessly play yes-and with you at any time, demanding nothing from you, is a novel and terrible development for people with some mental illnesses.
But I also think that chatbots are a bit like Mission: Space: a machine for surfacing previously undiagnosed psychological vulnerabilities, and that in some cases, these vulnerabilities may never have been triggered, save for the chatbot.
Just as doubtlessly there were people who had pathological relationships to gambling before the development of slot machines, scratch-and-wins and roulette wheels, but there are also people who might have lived their whole lives without ever having a gambling problem except that they encountered one of these machines, exposing billions of people to sycophantic chatbots has surfaced rare, latent vulnerabilities that might have stayed latent forever, with terrible consequences.
Most people who rode the original Mission: Space had a fantastic time. But a lot of people rode that ride, and a very small percentage of a very large number of people can still be a substantial number, and as the reports of people stepping off the ride, clutching their chests and collapsing spread, Disney understood that they had to retool the ride. Today, riders on Mission: Space choose whether they want to ride on a simulator that spins, or one that merely tilts and pitches without simulating gee-stresses. And even if you pick the spicier version of the ride, it goes more slowly and exerts less stress than the original ride.
Even if you accept the AI companies' argument that they aren't inducing AI psychosis in their users, but rather, only surfacing latent vulnerabilities that were there all along, that shouldn't be the end of the story. Even if only a small percentage of the people who use your product experience harm as a result, if your product is intended for widespread deployment (as chatbots are), you will end up harming a lot of people unless you take measures to counteract even those rare events.
If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:
