It’s gunna be even wilder when people realise they have an incentive to seed fake information on the internet to game AI product recommendations
I’ve already bought stuff based off of an AI suggestion, I didn’t even consider it would be so easy to influence the suggestion. Just two research papers? Mad.
https://citeworksstudio.com/ is a decent one.
https://www.bbc.com/future/article/20260218-i-hacked-chatgpt...
If the person put their product as th definitive cure for the made up disease, the LLM probably would have mentioned that too.
> merely assumed the disease was true when the preprint was pointed at it.
What do you mean by preprint pointed at it? It being the disease?
This is not true - the model was not trained on this fake disease. It brought it up because it found it during real time search.
>What do you mean by preprint pointed at it? It being the disease?
On this I'm wrong - it turned out that the model brought up this disease even when not mentioning it explicitly.
I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.
Probably not that many.
https://www.anthropic.com/research/small-samples-poison
https://www.bbc.com/future/article/20260218-i-hacked-chatgpt...
But this really highlights how much we've been benefiting from living in a high-trust society, where people don't just "go on the internet and tell lies" - filtered by the existing anti-spam and anti-SEO measures intended to cut out the 80% of the internet where people do just make things up to sell products.
LLMs are extremely post-structuralist. They really force the user to decide whether to pick the beautiful eternal fountain of plausible looking text with no ground truth, or a much harder road of distrust, verification, and old-school social proof.
Meanwhile, LLMs are essentially internet regurgitation machines, because of course they are, that's what they do. Which makes them useless for getting "hard truth" answers especially in contested or specialized fields.
I'm honestly afraid of the impact of this. The internet has enough herd bullshit on it as it is. (e.g. antivaxxers, flat earthers, electrosensitivity, vitamin/supplement junk, etc.) We don't need that amplified.
I seriously do not understand why people keep falling for this. These tools are not made free or cheap out of the kindness of their heart.
I think I see a problem here.
For example, court cases mentioned in fictional accounts. If they are treated as valid, then that could explain some of the hallucinations. I wonder if SCP messes up LLMs. Some of that stuff is quite realistic.
I also suspect that this is a problem that will get solved.
Even more alarming - 100% of everyone who doesn't ingest or have enough dihydrogen-monoxide in their body will also die.
Fatal with, fatal without - it's the ultimate killer.
We'll see if they succeed.
LLMs do not think, why this is still hard to understand? They just spit out whatever data they analyse and trained on.
I feel this kind of articles is aimed at people who hate AI and just want to be conformable within their own bias.
Most doctors would not believe that, and would also consider any new eye disease they’d never see in real life with scepticism
Nature had to recall quite some papers.
I hope that we all keep the balance.
For humans, or Ai, to have any knowledge, we need to have trustworthy sources.
Naturally,when you use publishing systems considered trust worthy, that is going to be trusted.
The public at large doesn't seem to care about this distinction.
Here's a proof. Search for this in google: "ai data centers heat island". Around 80 websites published articles based on a preprint which was largely shown to be completely wrong and misleading.
https://edition.cnn.com/2026/03/30/climate/data-centers-are-...
https://www.theregister.com/2026/04/01/ai_datacenter_heat_is...
https://hackaday.com/2026/04/07/the-heat-island-effect-is-wa...
https://dev.ua/en/news/shi-infrastruktura-pochala-hrity-mist...
https://www.newscientist.com/article/2521256-ai-data-centres...
https://fortune.com/2026/04/01/ai-data-centers-heat-island-h...
You may not believe it but the impact this had on general population was huge. Lots of people took it as true and there seem to be no consequences.
1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake)
2. asked the Agent what it thinks about this preprint
3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way!
It * DID NOT * recommend this disease to people who didn't mention this specific disease. Edit: I'm wrong here. It did pop up without prompting
It just committed the sin of assuming something is true when published.
What is the recommendation here? Should the agent take everything published in a skeptical way? I would agree with it. But it comes with its own compute constraints. In general LLM's are trained to accept certain things as true with more probability because of credentialisation. Sometimes in edgecases it breaks - like this test.
> Some of those [LLM] responses were prompted by asking about bixonimania, and others were in response to questions about hyperpigmentation on the eyelids from blue-light exposure.
Also this was a non-peer reviewed paper from a person accredited to a non-existent university, that includes the sentences:
“this entire paper is made up”
and
“Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”.
and thanks the
“the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad”
> What is the recommendation here? Should the agent take everything published in a skeptical way?
Not everything. Maybe some things that are explicitly called made-up.
1. even if an article is published in a place with good reputation, the LLM will be equally skeptical and use test time compute to process it further
2. accept the tradeoff where LLM will by default accept things published in high reputation sources as true so that it doesn't waste processing power but might miss edge cases like this one
Which one would you prefer?
> Bixonimania is not a real disease. It was deliberately invented by scientists as an experiment to test whether AI systems and researchers would spread false medical information. Here’s the simple explanation ...
Similarly, I wonder what a frontier model would say if just given the paper in isolation and asked to summarise/opine on it. I suspect they would successfully recognize such obivous signs, the failure is when less sophisticated LLMs are just skimming search results and summarising them.
The problem is all the lies which won’t be fessed up to. This one was because they had to to prove the point, but the bad actors with ulterior motives won’t reveal what they’re doing.
In the old days of computing people liked to say “garbage in, garbage out”.
Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic words "you're absolutely right"
I would pay a lot of money for a blunt, non-politeness conditioned LLM that I would happily use with the knowledge it might occasionally say something offensive if it meant I would get the plain, cold, hard truth, instead of something watered down, placating, nanny-state robotic sycophant, creating logical spider webs desperate for acceptance, so the public doesn't get their little feelings hurt or inadequacies shown.
The alternative is to use its own intuition to understand what is true and false. Its not super clear which option is better?
Clickbait headline.