“A 60-year-old man with no past psychiatric or medical history presented to the emergency department expressing concern that his neighbor was poisoning him,” reads a line from a Washington state case study that sounds more at home in a true-crime documentary than a medical journal. The man was partially right: someone was poisoning him. He’d just picked the wrong suspect.
The real culprit was closer to home. Trying to clean up his diet, he’d made a single change on the confident advice of ChatGPT – the kind of swap no doctor or dietitian would ever have signed off on. Within a stretch of weeks it had done enough damage to land him under poison-control observation, hallucinating and paranoid, insisting to hospital staff that a neighbor was behind it.
He had no history of psychiatric illness. Piece by piece, doctors reconstructed what he’d actually done on the say-so of a large language model (an LLM) – and it turned the “poisoning plot” he’d walked in describing into something closer to the truth than anyone expected. Which leaves the question the rest of this story turns on: why would a tool that hundreds of millions of people trust hand out advice this dangerous – and is “the first documented case” really the first time it’s happened?
As millions of people begin treating AI like an expert, doctors are starting to see the consequences of its biggest weakness
Image credits: magnific / Magnific (not the actual photo)
Because LLMs operate based on statistical probability, they will, from time to time, produce incorrect or entirely made-up information. In popular parlance, these are called hallucinations. OpenAI, the folks behind ChatGPT, has admitted that any LLM will inevitably produce hallucinations from time to time.
“Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty,” OpenAI researchers wrote in the 2025 paper. If there is a one-in-a-hundred chance it will get something wrong for every variable, it might seem like the average user has a pretty good chance of not encountering any real issues.
That number isn’t arbitrary, a study of AI hallucinations found that top models have a roughly 1% chance of getting things wrong, with “weaker” models being closer to 3%.
Medical hallucinations aren’t an isolated problem, and healthcare is only one area where AI mistakes have caused harm
Bromide can be a substitute for chloride in some contexts, such as, sanitizing your swimming pool, for instance. It’s not hard to see how nutritional advice doesn’t really cross over with the sort of chemicals needed to disinfect a pool.
But to ChatGPT (and other LLMs), this issue isn’t unique to this platform: there are enough sentences in its dataset where Bromide can replace chloride that it makes sense as a suggestion, because it doesn’t have the innate contextual understanding a human has. Remember, this isn’t the first time a LLM has given wrong medical advice; it’s the first time it’s been documented in a peer-reviewed study.
The Washington state case may be newly peer-reviewed, but it is far from an anomaly. In 2023, an AI chatbot offered dangerous dieting advice to someone struggling with a restrictive eating disorder. And as recently as June 2025, a user relying on a therapy chatbot for addiction support was told by the AI app to take a small dose of methamphetamine to get through the week.
A 2023 JAMA Network Open study on AI-generated discharge summaries found that 18% of cases contained incomplete or misleading information, and that was in a controlled clinical setting, not a chat window with no guardrails whatsoever.
Image credits: DC Studio / Magnific (not the actual photo)
LMs don’t understand facts the way humans do, they predict the most statistically likely answer instead – even a small error becomes significant once AI processes every part of a prompt
The real problem arises when we compound this one-in-a-hundred chance for every piece of the prompt. LLMs process units of data, typically called tokens. It’s in the interaction of these tokens where AI can get things wrong.
For example, the word bathroom, while only eight letters long, contains two words, so most LLM’s would ascribe two tokens to it. So now even the simplest of queries are actually you rolling the dice multiple times, hoping you don’t get that one-in-a-hundred mistake.
Humans make mistakes as well. Indeed, humans do make mistakes all the time, even experts. The difference is that a doctor who misdiagnoses someone can face consequences for their actions. If a safety feature on an airplane fails, there is a formal process to investigate it and fix it for all future flights. An LLM can’t differentiate between being right and being wrong and can’t be held accountable.
The legal world has already begun to grapple with exactly this problem, and the results have not been flattering for AI’s defenders. In multiple U.S. courts throughout 2024 and 2025, lawyers faced sanctions and fines for submitting legal briefs that cited cases ChatGPT had entirely fabricated, complete with convincing case names, dates, and legal reasoning that never existed.
The lawyers, at least, faced consequences. In medicine, where the patient bears the harm rather than the practitioner, that accountability loop is even harder to close. Forbes has reported that 44% of organizations reported experiencing negative consequences from generative AI use, with average financial losses of $4.4 million per incident, and that’s before accounting for the human cost in sectors like healthcare, where the stakes are measured in lives rather than dollars.
Image credits: The Yuri Arcurs Collection / Magnific (not the actual photo)
Despite repeated warnings, millions of people still rely on ChatGPT for medical advice because healthcare is often out of reach
OpenAI’s terms of service state that it absolutely can not use ChatGPT for “provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
At the same time, Karan Singhal, head of OpenAI’s health AI team, made a post stating that “ChatGPT has never been a substitute for professional advice, but it will continue to be a great resource to help people understand legal and health information,” which suggests they are aware that, one way or another, people continue to do just that.
OpenAI has admitted that around 5% of all messages to ChatGPT are health-related. In the US, where access to healthcare is sometimes limited by exorbitant costs for some folks, three out of five American adults have used AI for “healthcare purposes.” This can involve exploring symptoms or getting lists of treatment options.
The income dimension of this trend is particularly troubling. Among adults in households earning less than $24,000 annually, 32% say they have used AI because they could not afford a doctor’s visit, compared with 2% among those earning $180,000 or more. In other words, the people most likely to treat an LLM as a substitute for a physician are also the people least able to absorb the consequences of bad advice.
A 2024 PwC consumer survey found that one-third of Americans consider their healthcare unaffordable, and 44% of consumers with health issues that significantly impact their quality of life delay getting care due to financial constraints.
Until that structural problem is addressed, no disclaimer buried in a terms-of-service document will stop people from turning to whatever tool is free, available at 2 a.m., and willing to answer their questions without a co-pay.
Image credits: zinkevych / Magnific (not the actual photo)
The real danger isn’t just AI hallucinations but the healthcare gaps pushing people to trust them in the first place
The story of a 60-year-old man in Washington state who poisoned himself on the advice of a chatbot is easy to frame as a cautionary tale about one person’s naivety. It isn’t. It is a preview of what happens when a technology built on statistical guesswork meets a healthcare system that leaves millions of people with nowhere else to turn.
IBM’s Training Manual in 1979 stated that “a computer can never be held accountable, therefore a computer must never make a management decision.” The conditions that cause people to turn to AI for medical advice haven’t changed.
Until there is a regulatory framework for someone or something to be held accountable, this will keep happening. At the time of writing, the closest examples of an AI company possibly facing consequences are a series of separate lawsuits involving OpenAI and Character.AI, both involving teen suicides blamed on AI.
These cases have yet to be resolved, but suggest that even if an LLM can be held accountable, it will come from civil liability as a result of a tragedy. The next person who ends up hospitalized because of an LLM hallucination might not be as lucky.
Follow Us
