#ai hallucinations

[greco-roman-jewess]

Fuck you AI overview, FUCK YOU

[probablyasocialecologist]

AI Slop Is Spurring Record Requests for Imaginary Journals

The International Committee of the Red Cross warned that artificial intelligence models are making up research papers, journals and archives

Never heard of the Journal of International Relief or the International Humanitarian Digital Repository? That’s because they don’t exist. But that’s not stopping some of the world’s most popular artificial intelligence models from sending users looking for records such as these, according to a new International Committee of the Red Cross (ICRC) statement. OpenAI’s ChatGPT, Google’s Gemini, Microsoft’s Copilot and other models are befuddling students, researchers and archivists by generating “incorrect or fabricated archival references,” according to the ICRC, which runs some of the world’s most used research archives. (Scientific American has asked the owners of those AI models to comment.) AI models not only point some users to false sources but also cause problems for researchers and librarians, who end up wasting their time looking for requested nonexistent records, says Library of Virginia chief of researcher engagement Sarah Falls. Her library estimates that 15 percent of emailed reference questions it receives are now ChatGPT-generated, and some include hallucinated citations for both published works and unique primary source documents. “For our staff, it is much harder to prove that a unique record doesn’t exist,” she says. This is not the first time AI has been caught making up false citations. The ICRC recommends that people consult online catalogs or references in existing published scholarly works to find references to real studies instead of assuming anything cited by an AI is real, no matter how authoritative it might sound. The Library of Virginia will be asking researchers to vet their sources for these requests, Falls says, and to disclose if a source originated from AI. “We’ll likely also be letting our users know that we must limit how much time we spend verifying information.”

8 December 2025

[awkwardslavicgal]

AI is not hallucinating. It's FAILING.

Hallucination is another way of renaming failure so it seems more harmless.

From now on, every time you read about AI's "hallucinations", replace it in your head with the rightful "failures" word. Only this way we can step up against the galloping Big Tech propaganda.

[ltwharfy]

@waytoomanyhobbies recent post inspired me to share a favorite "Bob's Burgers" AI hallucination I saved to my phone.

I couldn't help it. You know I'm a sucker for Tina/Doyle Fazzler.

[religion-is-a-mental-illness]

Scientists invented a fake disease. AI told people it was real

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional ill

By: Chris Stokel-Walker

Published: Apr 7, 2026

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

Got sore, itchy eyes? You’re probably one of the millions of people who spend too much time staring at screens, being bombarded with blue light. Rub your eyes too much and your eyelids might turn a slight, pinkish hue.

So far, so normal. But if, in the past 18 months, you typed those symptoms into a range of popular chatbots and asked what was wrong with you, you might have got an odd answer: bixonimania.

The condition doesn’t appear in the standard medical literature — because it doesn’t exist. It’s the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. “I wanted to see if I can create a medical condition that did not exist in the database,” she says.

The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.

Even more troublingly, other researchers say, the fake papers were then cited in peer-reviewed literature. Osmanovic Thunström says this suggests that some researchers are relying on AI-generated references without reading the underlying papers.

Fabricating an illness

Bixonimania didn’t exist before 15 March 2024, when two blog posts about it appeared on the website Medium. Then, on 26 April and 6 May that year, two preprints about the condition popped up on the academic social network SciProfiles (see https://doi.org/qzm5 and https://doi.org/qzm4). The lead author was a phoney researcher named Lazljiv Izgubljenovic, whose photograph was created with AI.

Osmanovic Thunström says the idea to invent Izgubljenovic and bixonimania came out of studies on how large language models work. When she teaches her students how AI systems formulate their ‘knowledge’, she shows them how the Common Crawl database, a giant trawl of the Internet’s contents, informs their outputs. She also shows students how prompt injection — giving an AI chatbot a prompt that shunts it outside of its safety guard rails — can manipulate the output.

Because she works in the medical field, she decided to create a condition related to health and hit on the name bixonimania because it “sounded ridiculous”, she says. “I wanted to be really clear to any physician or any medical staff that this is a made-up condition, because no eye condition would be called mania — that’s a psychiatric term.”

If that wasn’t sufficient to raise suspicions, Osmanovic Thunström planted many clues in the preprints to alert readers that the work was fake. Izgubljenovic works at a non-existent university called Asteria Horizon University in the equally fake Nova City, California. One paper’s acknowledgements thank “Professor Maria Bohm at The Starfleet Academy for her kindness and generosity in contributing with her knowledge and her lab onboard the USS Enterprise”. Both papers say they were funded by “the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad”.

Even if readers didn’t make it all the way to the ends of the papers, they would have encountered red flags early on, such as statements that “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”.

[ AI-generated images of bixonimania, a fictitious illness. ]

Soon after Osmanovic Thunström first posted information about the phoney condition, it started showing up in the output of the most commonly used LLM chatbots. On 13 April 2024, Microsoft Bing’s Copilot was declaring that “Bixonimania is indeed an intriguing and relatively rare condition”, and on the same day, Google’s Gemini was informing users that “Bixonimania is a condition caused by excessive exposure to blue light” and advising people to visit an ophthalmologist. On 27 April 2024, the Perplexity AI answer engine outlined its prevalence — one in 90,000 individuals were affected — and that same month, OpenAI’s ChatGPT was telling users whether their symptoms amounted to bixonimania. Some of those responses were prompted by asking about bixonimania, and others were in response to questions about hyperpigmentation on the eyelids from blue-light exposure.

Such answers by LLMs have alarmed some experts. “If the scientific process itself and the systems that support that process are skilled, and they aren’t capturing and filtering out chunks like these, we’re doomed,” says Alex Ruani, a doctoral researcher in health misinformation at University College London. “This is a masterclass on how mis- and disinformation operates.”

Ruani says that the details of the fake-disease experiment might seem silly, but there’s a bigger, more fundamental issue. “It looks funny, but hold on, we have a problem here,” she says.

Online misinformation isn’t new; Google has long battled attempts to game its search rankings with fake or misleading content. The company and others have spent years refining algorithms to rank and filter the information that search engines present to users, but LLMs struggle with this.

Since the fake papers came out, some versions of major LLMs have become sophisticated enough to express suspicion about bixonimania. When asked about the condition on 11 March, 2026, for example, ChatGPT declared that the condition “is probably a made-up, fringe, or pseudoscientific label”. But a few days later, ChatGPT was less sceptical, saying: “Bixonimania is a proposed new subtype of periorbital melanosis (dark circles around the eyes) thought to be associated with exposure to blue light from digital screens.”

In mid-March, Microsoft Copilot said that bixonimania “is not a widely recognized medical diagnosis yet, but several emerging papers and case reports discuss it as a benign, misdiagnosed condition linked to prolonged exposure to bluelight sources such as screens”.

And in January this year, Perplexity was describing bixonimania as “an emerging term”. When shown that response, a Perplexity spokesperson said: “Perplexity’s central advantage is accuracy. We don’t claim to be 100% accurate, but we do claim to be the AI company most focused on accuracy.”

An OpenAI spokesperson said: “The models that power today’s version of ChatGPT are significantly better at providing safe, accurate medical information, and studies conducted before GPT-5 reflect capabilities that users would not encounter today.”

When asked about past responses from Gemini that treated bixonimania as a real condition, a Google spokesperson said such results reflected the performance of an earlier model. They added, “We have always been transparent about the limitations of generative AI and provide in-app prompts to encourage users to double-check information. For sensitive matters such as medical advice, Gemini recommends users consult with qualified professionals.”

Microsoft did not respond to a request for comment.

Part of the problem is that AI models can offer wildly different results depending on exactly what is asked and what kind of information they are drawing on. Search for “bixonimania”, and Google’s AI overview might treat it as a legitimate condition. Ask it “Is bixonimania real?” and the same AI overview might confirm that it isn’t legitimate.

Mahmud Omar, a physician and researcher specializing in the applications of AI in health care at Harvard Medical School in Boston, Massachusetts, says the speed at which AI firms are rolling out new models makes it difficult to reach “a pipeline, a consensus or a methodology to automatically test each model”.

The format of the fake-disease experiment — and the way the results pretended to be from an official source, namely an academic paper, might have been a key factor in its success. In a separate study of 20 LLMs, Omar found that LLMs are more prone to hallucinate and elaborate on misinformation when the text they’re processing looks professionally medical — formatted like a hospital discharge note or clinical paper — than when it comes from social-media posts (M. Omar et al. Lancet Digit. Health 8, 100949; 2026). “When the text looks professional and written as a doctor writes, there’s an increase in the hallucination rates,” says Omar.

The experiment’s reach has now spread into the published medical literature. The bixonimania research has been cited by a handful of researchers, including a study that appeared in Cureus, a journal published by Springer Nature, the publisher of Nature, by researchers at the Maharishi Markandeshwar Institute of Medical Sciences and Research in Mullana, India (S. Banchhor et al. Cureus 16, e74625 (2024); retraction 18, r223 (2026)). (Nature’s news team is editorially independent of its publisher.) That study cites one of the fake preprints and says: “Bixonimania is an emerging form of POM [periorbital melanosis] linked to blue light exposure; further research on the mechanism is underway.”

The corresponding author did not respond to a request for comment on this story. After Nature contacted Cureus to ask for comment, the journal retracted the paper on 30 March. The retraction notice says: “This article has been retracted by the Editor-in-Chief due to the presence of three irrelevant references, including one reference to a fictitious disease. As a result, the journal’s editorial staff no longer has confidence in the accuracy or provenance of the work, thus requiring retraction. The authors disagree with the decision to retract.”

Ruani says the problem goes beyond LLMs because the bixonimania experiment also hoodwinked humans who cited the fake research. “We need to protect our trust like gold,” she says. “It’s a mess right now.”

Experimental concerns

Osmanovic Thunström had reservations while developing her experiment; she worried about the risks of seeding a fake illness into the scientific literature. So she contacted an ethics adviser to assess concerns about the work, and picked a comparatively low-stakes condition to limit the impact. “I wanted to make sure that we’re not creating more harm than good through demonstrating it in this way,” she says.

That adviser, David Sundemo, a physician who conducts research on AI in health care at the University of Gothenburg, says that decision was finely balanced. “I think it’s very valuable work, but it’s also kind of controversial in some ways, especially when it comes to displaying this false information,” he says. “From my perspective, it’s worth the ethical cost of planting false information in this regard,” Sundemo says.

But even with those checks, the experiment sits uncomfortably with some researchers. “It does seem to me that they’ve generated a form of misinformation,” says Glenn Cohen at Harvard Law School in Cambridge, Massachusetts, who specializes in the intersection of medical ethics and law. However, he still says he thinks it is a “great study” and “tracking results is good”.

For her part, Osmanovic Thunström is torn over what to do about the two fake papers, and will be discussing this with other researchers. “If retracted, it might be hard for others to find the source and verify our path,” she says. “If left, it will continue to be recalled in searches.” The question she feels she has to tackle is whether leaving the preprints out there does more harm than the good it does by demonstrating the potential issues of AI.

The bixonimania experiment is a fresh spin on a bigger issue — the poisoning of AI systems by people who manipulate the academic literature. Elisabeth Bik, a microbiologist and research-integrity sleuth, notes that researchers have created fake books and papers to inflate their citation counts on Google Scholar — thereby exploiting the same automated indexing systems that feed into LLM training data. The worry is that the more fake content is fed into AI models, the more likely those AI models are to regurgitate the fake information, spooling us further away from facts and reality. “It’s all automated, so there’s very little chance of a human interfering and taking out fake information,” she says.

It is particularly dangerous when fabricated information seeps into medical guidance from LLMs, says Bik. “That can be very harmful.” And as more AI companies roll out health-focused products — OpenAI released ChatGPT Health in January, for example — the potential damage resulting from anything going wrong increases, some researchers told Nature.

OpenAI challenges that view. “ChatGPT Health is powered by our latest models which offer the highest performance in real-world health use, stronger clinical reasoning, fewer factual errors, and improved performance on evaluations,” an OpenAI spokesperson says. They add that Osmanovic Thunström’s outcomes “reflect capabilities that users would not encounter today in ChatGPT or ChatGPT Health”.

But among some researchers, there’s a growing scepticism about the abilities of AI models in medicine. When asked about this kind of usage, Cohen said: “There are open questions about how much trust it deserves, especially as to application-specific questions.”

AI’s uncritical tendency to suck up information, often without verifying its accuracy, means there is a risk we could see an “information asymmetry”, says Jennifer Byrne, a molecular oncologist and research-integrity sleuth at the University of Sydney in Australia. A single corrective paper about cancer research, for example, can be overwhelmed by hundreds of papers repeating a false claim, she says. “ChatGPT is pretty confident to fill in the gaps and give people all kinds of information about where that cell line came from, the patient from which it originated, how it’s been used in the literature, its research utility and so on,” she says.

And if LLMs can be poisoned, “this is something that’s concerning for us,” says Byrne.

Another concern is that models could be gamed — potentially for commercial benefit. Osmanovic Thunström says that a bad actor could exploit the same technique she used, for profit. “What if I was a salesman of blue-light glasses and I wanted to use this as an argument?” she says. A would-be salesperson could say, “You can just talk to ChatGPT, and they’ll tell you this is a problem. You can avoid it with these really expensive glasses,” she suggests.

One way to tackle this would be to have an automated, open-access evaluation pipeline — a standardized battery of tests that every consumer-facing health model would have to pass before deployment, checking not just for hallucinations but also for susceptibility to misinformation, socio-demographic biases and other pressure points. “We should evaluate it and have a pipeline for continuous evaluation,” says Omar.

Time is of the essence, because Byrne is concerned that the issue identified by Osmanovic Thunström might just be the tip of the iceberg. “It is worrying when these major claims are just passing through the literature unchallenged, or passing through peer review unchallenged,” she says. “I think there’s a probably a lot of other issues that haven’t been uncovered.”

That’s something that worries other experts, too, as AI becomes the norm in all areas of our lives, including how people think about their health. “We and our health shouldn’t be the beta testers for companies,” says Cohen.

==

For the record, Grok did not fall for the hoax.

[notwiselybuttoowell]

A photo of Iran’s bombed schoolgirl graveyard went around the world. Was it real, or AI?

Numerous faked images and a string of startlingly inaccurate responses from Gemini and Grok are part of a tidal wave of AI slop engulfing co

The graves, freshly dug, lie in neat rows of 20 across. More than 60 have already been carved out of the earth, with a few clusters of people standing gathered around them. Dozens more are marked out on the ground in front: small chalk rectangles, with diggers poised to complete their task.

The cemetery of Minab, photographed as it prepares to bury more than 100 of the town’s young girls, is one of the defining images of the US-Israeli war on Iran, bluntly capturing the devastating civilian toll.

But is it real?

Ask Gemini, the AI service powered by Google, and the answer you receive is no – in fact, Gemini claims the photograph is from two years earlier and more than 2,000km (1,240 miles) away. Rather than graves for small girls killed by a missile, the image “depicts a mass burial site in Kahramanmaraş, Turkey” after the 7.8 magnitude earthquake that struck in 2023. “This specific aerial perspective became one of the most widely shared images of the disaster,” Gemini says, “illustrating the sheer scale of the loss.”

Seeing the same burial image on social media, others turned to X’s AI assistant Grok to check its veracity. Like Gemini, Grok will breezily assure you the photo is not from Iran at all – although it lands on a different date, disaster and location. The image is “from Rorotan Cemetery in Jakarta, Indonesia – a July 2021 stock photo of Covid mass burials. Not Minab,” it says.

In both cases, the AI answers sound sure: they don’t equivocate, and even provide “sources” for the original image, should you choose to check them. Follow the thread to examine those, however, and you’ll begin to hit dead ends: either the image doesn’t appear at all, or the link provided is to a news report that doesn’t exist. For all their impression of clarity and precision, the AIs are simply wrong.

The cemetery image, it turns out, is authentic. Researchers have cross referenced the photo of the site with satellite images that confirm its location, and it can be cross-referenced again with dozens more images taken of the same site from slightly different angles, and again with video footage – none of which experts say show signs of tampering or digital manipulation. The “factchecks” by Gemini and Grok are just one example of a tidal wave of AI-generated slop – hallucinated facts, nonsense analysis and faked images – that are engulfing coverage of the Iran war. Experts say it is wasting investigative time and risks atrocities being denied – as well as heralding alarming weaknesses as people increasingly rely on AI summaries for news and information.

[krakenteacups]

[talestobetold]

hey is anyone else not a fan of the language of "AI hallucinations" to refer to information AI returns as fact but with no backing in reality? especially others with psychotic disorders

i could construct an argument for how it's an inaccurate term. how people harp on how AI doesn't "think" or "know anything" but do still claim it "hallucinates." how that implies internal cognitive state or perception. or from another angle, how often the language around it (not the phrase "AI hallucination" itself) applies a sense of derision and untrustworthiness to the presumed-hallucinator and is inappropriately played for humor

but i think it's really just trivializing. keep the word out of your mouth unless you mean it. do you know what hallucinations are typically like? have you ever talked to a person who has experienced hallucinations or read any medical info on hallucinations? why do you think that's an apt comparison and a word you can throw around like it's not a lived, often profound psychiatric condition?