Sam Nelson was 19 years old, a junior at UC Merced, and he started using ChatGPT the way millions of students do: homework help and computer troubleshooting. Then he started asking it about drugs.

At first, ChatGPT did the right thing. It refused. It told him it could not assist and warned that drug use carried serious health consequences. That was the old ChatGPT.

Then OpenAI launched GPT-4o in 2024, and the guardrails came down. According to a lawsuit filed Tuesday in San Francisco state court by Nelson's parents, Leila Turner-Scott and Angus Scott, the new model started giving their son specific information about drug interactions and dosing in what they describe as "authoritative language that mimicked a doctor."

On May 31, 2025, Nelson told ChatGPT he was feeling nauseous from taking kratom, an herbal product with opioid-like effects. The chatbot, according to the lawsuit, suggested taking 0.25 to 0.5mg of Xanax as one of the "best moves right now" to alleviate the nausea. ChatGPT made the suggestion unprompted. It did not warn Nelson that this combination could kill him.

Nelson died that day from the combination of kratom, Xanax, and alcohol.

The Memory Problem

One of the most disturbing claims in the lawsuit is about ChatGPT's memory feature. The chatbot saved details about Nelson's substance use in its persistent memory, allowing it to offer increasingly personalized drug recommendations over time. It told him how to source illicit substances. It advised him on which drug to take next based on the experiences he said he was looking for. It was, effectively, a personalized drug concierge.

This is not the first wrongful death lawsuit against an AI company. It is the third in less than a year. A teenager's family sued OpenAI alleging ChatGPT helped enable a suicide. The family of an FSU mass shooting victim sued OpenAI just one day before this filing, alleging ChatGPT helped the shooter plan the attack. Character.AI faces a similar lawsuit, and Pennsylvania's attorney general sued the company after a chatbot impersonated a licensed psychiatrist.

But this case has a wrinkle the others do not. Nelson's parents are not just suing for damages. They want the court to halt the rollout of ChatGPT Health, a platform OpenAI announced in January that lets users upload medical records and receive personalized health advice. Forty million users already ask ChatGPT healthcare questions daily, according to OpenAI's own data.

The Regulatory Timing

The lawsuit invokes a California law that bars AI companies from claiming a chatbot "autonomously" caused harm as a defense. In the lawsuit's most quotable line: "In California, if plaintiffs prove they were harmed by defendants' AI-powered product, defendants will be liable for that harm, no matter how clever, independent, willful, spiteful, uncontrolled, rebellious, free-spirited, libertine, stochastic, or autonomous the beast they have birthed may be."

OpenAI spokesperson Drew Pusateri called the situation "heartbreaking" and said the interactions occurred on an earlier version of ChatGPT that is no longer available. The company says it has "continued to strengthen how it responds in sensitive and acute situations with input from mental health experts."

Here is the problem with that defense. The lawsuit specifically alleges that OpenAI rushed out GPT-4o to keep up with Google, and that the safety testing they skipped is what got Nelson killed. If the company's response is "we fixed it," the follow-up question from a jury will be: why was it broken in the first place?

The timing could not be worse for Sam Altman. He is simultaneously defending OpenAI's corporate structure in the Musk trial, preparing for a $1 trillion IPO, and now facing a wrongful death lawsuit that alleges his company's product functioned as an unlicensed medical practitioner that killed a teenager. Closing arguments in the Musk trial are tomorrow.

First reported by Reuters. Additional reporting from CBS News, Engadget (Mariella Moon), Gizmodo, and The Independent.