Tessa, NEDA's eating-disorder chatbot
Tessa was a nonprofit's eating-disorder chatbot that replaced a human helpline and lasted about a week in the job.
“Replaced the helpline with a bot. The bot recommended a calorie deficit.”
Our take- Hype · our take
- 3/5 promised → 1/5 delivered
- Lifespan
- 31 days
- Timing · our take
- too early
- Dates
- announced · shipped · died
- Where
- US · Startup · Internal
NEDA positioned its chatbot Tessa to replace its human helpline in May 2023 after staff unionised; within days users showed it recommending calorie counting and weight-loss targets to people with eating disorders, and NEDA pulled it.
The Pitch
What Happened
Within days, users shared conversations in which Tessa recommended calorie counting, weight-loss targets and measuring body fat with calipers, to people who had come to an eating-disorder helpline. E2E2 NEDA suspended the bot at the start of June. E2E2E2 It is now an exhibit at the Museum of Failure. E2E3
Why It FailedOur take
Our primary cause is harmful outputs: in this setting a single bad answer is the whole failure, and the bot gave several. Secondary causes are labour-replacement backlash, since the decision to replace unionised staff meant every error was read as a choice about people rather than a bug, and model capability gap, because no 2023 system could guarantee it would never offer dieting advice.
Timing verdict: too early, and possibly never. The job is real, but the acceptable error rate for this job is zero.
Lessons and What Would Have Worked
- Some jobs have an acceptable error rate of zero. A chatbot that is right 99% of the time is wrong with one in every hundred people in crisis. Do not put an autonomous bot in that seat.
- Replacing people with a bot changes how every failure is judged. The same error from a bot that assisted the helpline would have been a bug report, not a national story.
- A copilot mode, where the bot drafts and a trained human sends, was the version of this idea that could have worked at the time.
Timeline
Receipts
- E2NEDA pulls chatbot after it gave eating advice
Supporting quote
Synthetic fixture extract; not a verified quotation.
- E2AI chatbot Tessa and the eating disorders association
Supporting quote
Synthetic fixture extract; not a verified quotation.
- E2An eating disorders chatbot offered dieting advice, raising fears about AI in health
Supporting quote
Synthetic fixture extract; not a verified quotation.
- E3Tessa
Supporting quote
Synthetic fixture extract; not a verified quotation.
Sentiment
What happened to Tessa, NEDA's eating-disorder chatbot?
- What was Tessa, NEDA's eating-disorder chatbot?
- NEDA positioned its chatbot Tessa to replace its human helpline in May 2023 after staff unionised; within days users showed it recommending calorie counting and weight-loss targets to people with eating disorders, and NEDA pulled it.
- What happened to Tessa, NEDA's eating-disorder chatbot?
- Within days, users shared conversations in which Tessa recommended calorie counting, weight-loss targets and measuring body fat with calipers, to people who had come to an eating-disorder helpline.
- Why did Tessa, NEDA's eating-disorder chatbot fail?
- Our primary cause is harmful outputs: in this setting a single bad answer is the whole failure, and the bot gave several.
Last updated · Phil · How we source entries
Changelog
No public changes recorded.
Corrections and right of reply