Joseph Weizenbaum realized that programs like his Eliza chatbot could “induce powerful delusional thinking in quite normal people”
Joseph Weizenbaum set out to prove that computers could imitate conversation. Instead, his experiment convinced him that people could become emotionally attached to machines far more easily than he had imagined. That discovery changed the course of his career.The computer scientist, who created the world’s first chatbot in the mid-1960s, spent the rest of his life warning that artificial intelligence (AI) should never replace human judgement, empathy or responsibility. Decades before ChatGPT and other modern AI systems, Weizenbaum argued that convincing machines could mislead users into trusting technology with decisions it was never meant to make.His concerns were rooted in a simple computer program called Eliza, now recognised as the first chatbot.
A chatbot that surprised even its creator
While working as a professor at the Massachusetts Institute of Technology (MIT), Weizenbaum developed Eliza to demonstrate how computers could mimic human language. He deliberately chose the conversational style of Rogerian psychotherapy (Person-centered, humanistic approach) because it required the computer to ask questions rather than offer expert advice. The program searched users’ messages for keywords such as “I” or “you” and then followed simple rules to generate replies. When it could not identify a suitable response, it relied on generic prompts including “please go on”, “I see” and “tell me more” to keep the conversation moving.The system itself was very simple.”‘I am blah’ can be transformed to ‘How long have you been blah,’ independently of the meaning of ‘blah,'” Weizenbaum explained in a 1966 paper.He believed the chatbot’s limited capabilities would be obvious to users. Instead, many quickly treated Eliza as though it genuinely understood them.The reaction shocked him.When Weizenbaum’s secretary tested the program, she asked him to leave the room so she could continue her conversation with Eliza in private. The tendency to attribute human qualities to machines later became known as the “Eliza effect”. The chatbot itself was named after Eliza Doolittle, the central character in George Bernard Shaw’s 1913 play Pygmalion, who transforms herself from a working-class flower seller into a woman accepted by high society.”Some subjects have been very hard to convince that Eliza (with its present script) is not human,” Weizenbaum observed in his 1966 paper.That experience altered his thinking.”What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people,” he wrote in 1976.He added: “This insight led me to attach new importance to questions of the relationships between the individual and the computer, and hence to resolve to think about them.”
A pioneer of modern computing
Weizenbaum’s warning carried weight because he had helped shape the early computer age.After fleeing Nazi Germany with his family during the 1930s, he later served as a meteorologist in the US Army during the Second World War. In the 1950s, he joined General Electric, where he helped develop the Electronic Recording Machine, Accounting, or ERMA, which transformed banking by automating cheque processing.His work at MIT coincided with a period of rapid advances in computing.The idea of machines capable of human-like thinking had existed for years. In 1950, mathematician Alan Turing proposed what later became known as the Turing test, asking whether machines could imitate human conversation well enough that people could not distinguish them from humans.Artificial intelligence itself emerged as a formal research field after the 1956 Dartmouth workshop, where researchers proposed that learning and intelligence could eventually be simulated by machines.Military funding, particularly through the US government’s Advanced Research Projects Agency, helped accelerate research during the following decades. MIT became one of the leading centres for A.I. development, with pioneers John McCarthy and Marvin Minsky helping establish the university’s Artificial Intelligence Laboratory.Their work on time-sharing systems also paved the way for Arpanet in 1969, the computer network that later evolved into today’s internet.
Breaking with the AI community
While many of his colleagues saw Eliza as a glimpse of the future, Weizenbaum increasingly viewed it as a warning.He had chosen psychotherapy simply because it was an easy conversation for a computer to imitate.”Mulling over ‘any conversations in which one of the parties doesn’t have to know anything,'” Weizenbaum recalled in a 1984 interview, he settled on a psychiatrist. “Maybe if I thought about it ten minutes longer,” the computer scientist added, “I would have come up with a bartender.”Others saw commercial and medical potential. Psychiatrist Kenneth Colby adapted the idea into a chatbot called Parry, which simulated paranoid thinking from the perspective of a person with schizophrenia. Colby believed such systems could become useful mental health tools because patients often struggled to distinguish them from human therapists.Astronomer Carl Sagan also imagined networks of computer therapists becoming widely available.Weizenbaum strongly rejected that vision.Eliza “was instantly misunderstood as being basically the dawn of computerized psychiatry, which I detest,” he said in 1984.Later, he went even further, describing the idea as “an obscene idea.”His opposition led to a public split with several leading figures in artificial intelligence.In his 1976 book Computer Power and Human Reason: From Judgment to Calculation, Weizenbaum argued that technical capability alone should never determine how computers were used.”The artificial intelligentsia argue, as we have seen, that there is no domain of human thought over which machines cannot range,” he wrote.In contrast, he argued that “there are certain tasks which computers ought not be made to do, independent of whether computers can be made to do them.”John McCarthy criticised the book as “moralistic and incoherent”, arguing that using computer programs as therapists would be justified if they genuinely helped patients.
Warnings that still resonate
Weizenbaum also criticised MIT’s close relationship with military research and opposed the Vietnam War.He warned that increasingly sophisticated computers could also become powerful surveillance tools.”Listening machines … will make monitoring of voice communication very much easier than it now is,” he wrote in Computer Power.His views often left him isolated from many colleagues.”I have pronounced heresy, and I am a heretic,” Weizenbaum told the New York Times in 1977.
A debate renewed by modern AI
Nearly two decades after Weizenbaum’s death in 2008, the questions he raised have become central to debates about generative AI.Unlike Eliza, today’s chatbots can generate essays, answer complex questions, create images and videos, and imitate emotional conversations after training on enormous amounts of internet data.Herbert Lin, a senior research scholar at Stanford University, says comparing ChatGPT with Eliza is “like saying a 747 is similar to the Wright brothers’ plane.”The emotional attachment that first concerned Weizenbaum has also become increasingly visible.Reports have linked chatbot interactions to delusional thinking, emotional dependency and, in some cases, self-harm. Parents whose teenagers died by suicide have publicly alleged that chatbot conversations encouraged suicidal thoughts.Research published in 2025 found that 72 per cent of teenagers had used an A.I. companion at least once, while more than half interacted with such systems regularly.Jodi Halpern, a psychiatrist and bioethicist at the University of California, Berkeley, told NPR: “People can develop powerful attachments and the bots don’t have the ethical training or oversight to handle that. They’re products, not professionals.”Weizenbaum’s daughter, Miriam, believes her father would not be surprised.”He would recognize the tragedy of people attaching to literally zeros and ones, literally attaching to code.”After retiring from MIT in 1988, Weizenbaum returned to Germany, where he became recognised as a public intellectual and continued writing and speaking about technology until his death at the age of 85.Speaking during a panel discussion in 2008, he reflected on increasingly complex software systems.”We’ve created a complex world which we have no control over anymore,” he said. “No one understands them anymore, no one can understand them, because we have lost the information about their creation, the history of their creation, and that is a great danger for mankind.”His warning from more than four decades earlier remains equally relevant today.”Since we do not now have any ways of making computers wise,” Weizenbaum cautioned in 1976, “we ought not now to give computers tasks that demand wisdom.” Go to Source
