Weizenbaum's Nightmare
Updated 2026-08-16
INTRODUCTION
English translation pending.
CORE DEFINITION
Raised by Joseph Weizenbaum, creator of the ELIZA chatbot, in the 1970s. Weizenbaum was disturbed to find that users, including his own secretary, confided in a simple pattern-matching script and treated it as understanding them. He argued that some decisions, such as those of judges, therapists, and caregivers, should not be delegated to machines even when machines could produce the right outputs, because the human involvement is itself part of the value.
SCAFFOLDING EFFECT
Reduce cognitive load
- Ask the second question: after whether a machine can do a task, ask whether it should - Mark the boundary: identify roles where human presence is part of the good being delivered - Protect the vulnerable: set human fallback for care settings where users may form attachments
Anchor fast decisions
People readily attribute understanding and empathy to systems that mirror conversational cues, because the social response is triggered by the form of the interaction rather than by the underlying capacity. That projection is what makes delegation feel acceptable, and it also makes users susceptible to manipulation or to substituting a script for genuine care. The concern is about what the substitution does to the people involved, not about technical feasibility.
MINIMUM ACTION
In progress 0/1Practice this model in one real situation:
account_treeGenealogyexpand_more
menu_bookReferencesexpand_more
Source support: Explicit
- en.wikipedia.orghttps://en.wikipedia.org/wiki/ELIZA_effectverified
PRIVATE NOTES · Only visible to you
SAVED Q&A
ENTRY Q&A · Private saving available
Ask with a clear boundary
thinkingmodels answers from published entry context only.
Your question is sent to thinkingmodels. The answer uses public entry context only.
RELATED MODELS