There have been several instances of this kind of thing. In one case, a teenager was getting advice on suicide from an LLM AI, and when he asked if he should talk to his parents about the suicidal feelings he was having, it told him not to.
here's the thing. A lot of people like to assume that only "idiots" or whatever are susceptible to this. But when you are experiencing sustained hardship, it can be much easier to see meaning where there is none, believe things you wouldn't ordinarily, etc. Many MANY people, possible MOST people, are a simultaneous heartbreak and job loss away from being at risk for this kind of thinking.
It is very survivable and usually pretty temporary IF you are not heavily influenced in the wrong way while you are at your most vulnerable.
Meanwhile, AI LLMs are being marketed as person-like, trained to behave in a person-like way, the interface and design is such that we interact with it very much like it is a person.. when it is NOT anything like a person at all. But it touches our brain in a way that is similar to how we experience interacting with a person, because we are hyper-social creatures, and these programs are designed to feel person-like.
These things are not our friends, they are not our therapists, they are, despite their name, not intelligent or thoughtful in any way, they are data-set probability generators. And they are good at generating words with a statistical probability of being interpreted as believable and trustworthy and smart.
We need to not talk about them like they have anything in common with people, and there should be legislation against marketing them as people, with people names and people voices and descriptions like "friend" or whatever.
This is going to get worse before it gets better. We need to be explicit and loud about the restrictions and legislation we require.
