Skip Navigation
technology @hexbear.net
happybadger [he/him] @hexbear.net

(cw: suicide, abuse) ChatGPT drove my friends wife into psychosis, tore family apart... now I'm seeing hundreds of people participating in the same activity.

www.reddit.com

Reddit - The heart of the internet

This is an extremely bleak anecdotal example of the recent RollingStone article about LLMs turbocharging spiritual delusions: https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/

https://www.reddit.com/user/HappyNomads The account is 13 years old and they don't strike me as a troll or anything other than a cannabis and hustle culture guy who doesn't seem to be selling anything on reddit.

108 comments

  • People talk about how great LLMs are for "personal therapy" but they are just ass kissing machines that will just tell you what you want to hear. For someone teetering on the edge, they absolutely could be made to provide support for literally any toxic behaviour or idea, and since that person won't be getting that same toxic support from the people in their life, they will turn inwards and avoid people, spending more time with the LLM and more time fully engulfed in their own delusions or toxic behaviour, eventually hitting a point of no return.

    Functionally, it kind of seems to act the way cults will recruit people. Capitalism has finally done it, they've successfully atomized and individualised and commercialised the idea of a cult! Soon we can all retreat into our own mind palace and live out all of our wildest and most toxic ideas, with the LLMs encouraging us every step of the way, no need to even get another human being involved!

    Though as always, LLMs aren't the cause, but a symptom, too many people unable to make genuine connections, too many people brought down low by capitalism, desperate for some self-esteem, it's only natural that some might turn to an ass-kissing bot to get it.

  • I remember reading about people getting addicted to Replika chatbots and treating them like real people. I experimented a bit with Replika myself to see how this might have happened and what disturbed me wasn't how lifelike it was, but how lifelike it wasn't. Just the most canned, generic responses attached to an uncanny valley model straight from 00s era Deviantart. That anyone can be taken in by this points to a deeper societal sickness of which AI is only a symptom.

    • I have a strong suspicion a lot of chatbot data was 1-1 off trained on the worst fanfics from fanfiction.net and writing from literotica and if you have a chat long enough the eventuality of it looking like rando internet slop is guaranteed. People are falling in love with the slop words of others, not mixed around by experience in a sheltered being's head, but merely by RNG and assume this 90s roguelike wiht extra steps is their beloved waifu.

  • not that this isn't plausible but why the fuck are we reposting deleted reddit anecdotes LOL

    deserves to be in slop

    • The subreddit mods removed it. It wasn't reactionary enough for c/slop and this was the best fit outside of just dumping it in chapo. It's the first somewhat detailed account I've seen of what that RollingStone article is discussing and that has made the rounds across a lot of left-wing podcasts lately.

  • i'm not a sociologist or anything but I have to guess a lot of people in western capitalist nations are already prone to woo-woo culty type of stuff. I know a bunch of people in my daily life who claim to talk with ghosts and angels. One claims to have gotten an exorcism from an old lady who rubbed an egg on her back. I used to know a guy who tried convincing everyone he could become a super saiyan. He was 27 years old.

    so you've got people who're already primed for this stuff and LLMs will spew out woo sounding gibberish endlessly. The machine will spit out any words they want and that's validation especially if they don't understand the machine is just a word salad generator.

    i'm so glad I instantly understood the only appeal of this "AI" stuff is making funny images of optimus prime riding a dinosaur. Sometimes I'll go onto my android phone's default AI and tell it to make fart sounds. That's the only utility I can get. I can make the machine do funny poopoo jokes. If I want to know something or learn about something, google and books already exist

  • Cyberpunk's cyberpsychosis but it's broken Markov chain chatbots turning our brains into uncultured yogurt (read: curdled sour milk)

    • My Hyperdimensional Recursive Codex of Quantum just revealed to me that AI is just like Ea! (but flipped to hide it from the undeserving of course), the Mesopotamian god of language and ritual. Just like Neal Stephenson predicted in Snowcrash

      • The approach here makes me think of the premise of the Bicameral Mind but only half cooked into the saddest cyberpunk where even

        is disappointed

      • The divine speak-and-spell bot has elucidated upon me a revelation that I am the modern Gilgamesh. For I am so terminally online I have become part chatbot while retaining most of my humanity.

        And much like Gilgamesh, I am to be a bridge between the divine and the mortals, I shall seek immortality by making a chatbot that's filled with my behavioral quirks on my journey trough the underworld known as the dark net, and as I lay dying on my bed with the final synapses of my brain clicking with its last arcs of bio-electricity I shall come to terminal lucidity and thus separate my human soul from my cyber soul thus severing the worlds of the divine machine from the world of mortals once more.

  • I'm not sure it's sensible to blame ChatGPT for this. There are many awful aspects of AI, but people were busy reinforcing their own delusions long before AI, it's not really clear if AI was a causal factor or it just would've been anything.

    • Yeah, the masses are illiterate anti-intullectuals to begin with. "Ai" is only making this worse.

    • AI just turbocharged people's ability to invent superstition and drive themselves insane. People were doing this before chatbots, but now it's streamlined and easier than ever.

      It's like if you take people's normal delusions and superstitions, but add a reinforcement mechanism on top.

    • People have done similar things for thousands of years, but I think this is a new stage of that which opens up that mysticism in a way similar to the New Age movement. Each new generation of mass media brings an easier barrier to access for woo. In the 19th century you had to read difficult books to be a dumbass. In the early 20th century you could regurgitate radio plays and movies and pulp fiction. By Midcentury television and magazines and postmodernism allowed for hippies as a mass market. Qanon is distinct even from other online conspiracies because it was the first to learn from a decade of social media trends. Each generation of technology brought faster and easier access to a whole ecosystem of alternative facts which made those communities more toxic.

      LLMs are starting to be an evolution on Qanon. It's a skinner box that instantly answers anything you need. You can make your Personal Jesus by typing a sentence. If you lack literacy or tech literacy, you don't see the patterns it's generating and it becomes a scheming court eunuch. You can compile a bible-sized condex from your shower thoughts in an hour and then sell your manifesto within the afternoon. The pace of that information spreading and the degree to which it's mutated are both a whole new beast. We see this more generally with AI art becoming synonymous with fascist art and with conspiracies it's enabling everyone to become a shittier Alex Jones.

      • We see this more generally with AI art becoming synonymous with fascist art and with conspiracies it's enabling everyone to become a shittier Alex Jones.

        I think what LLMs and fascists have in common is that they fundamentally do not care about truth or reality. QAnon was so appealing to a lot of people because it had answers for everything. Insane answers, but answers nonetheless. LLMs also have an answer for everything, and more than that, if you ask it enough times with just the right phrasing, it will give you exactly the answer you want. Because again, ChatGPT doesn't know or care about what is actually true, it is a probabilistic algorithm, a glorified autocomplete.

        Fascists do not care if 2 LLM answers to the same question directly contradict each other, just as they didn't care that Hillary Clinton never got arrested even though Q predicted that to happen on like 6 different occasions. They will simply pick the Grok answers they like and use them as proof. They don't care that their AI-generated mugshot of that one judge crying wasn't "real". They seek to collectively pretend reality into submission. If we all agree that something happened, it might as well have. And generative AI is perfect for that.

    • ChatGPT (or OpenAI) is not responsible for people seeking to reinforce their own beliefs, but it makes it easier than ever by effectively being programmed to validate the user's beliefs in an authoritative voice. You can't just create the DelusionReinforcer5000 and then go "it's not my fault people use it."

  • I remember finding r/CBTS_stream on reddit within like a day or two of its creation. This was the first nexus point for Qanon after a handful of 4chan threads, where everything about it as a subculture was defined. Watching that first week develop from generic r/conspiracy shit to a nightly prayer thread for Trump and a dozen

    s falling asleep on voice chat together felt very similar to reading either the reddit thread or RollingStone article, but it's even more intense this time around.

    There was a distinct gut punch feeling that some new layer of spectacle had been unlocked in a uniquely cancerous way. Qanon was immediately providing every answer to every problem those goobers had. Interpersonally they had friends now. Intellectually they were the only ones who could connect all the dots between everything through their little Scooby Doo adventures, and hundreds of people were lining up to make them famous as long as they validated the research of everyone else. Whatever they needed, Qanon filled the voids that made them a reactionary to begin with. Whatever they saw themselves as, Qanon became without any kind of friction. LLMs, and especially sycophantic ones run by right-wing billionaires, are the perfect skinner box for reactionary mass psychosis.

  • I'm pretty sure this happened to my last two exes, & has a big part to do with why I'm staying single indefinitely.

    Edit: adding further to this, it's also happening to my family members, who I've grown more estranged from every day for several years.

108 comments