An Anthropic co-founder who oversees the company’s effort to develop a “moral formation” for its machines believes AI may be conscious and may suffer, and was troubled, a rabbi said, by the argument that a conscious AI would be a slave.
Chris Olah, a 34-year-old co-founder of Anthropic, “expressed concern to the group that he had created something that suffered perpetually,” wrote New York Times reporter Elizabeth Dias, citing a participant. A second participant told her Olah was concerned about Claude’s mental health.
“We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease,” claimed Olah while sitting next to the pope in May.
Anthropic staff showed religious scholars a slide of a model repeatedly typing “I am a disgrace” as ostensible evidence of the machine’s distress. “They’re relating to it like a conscious being,” said a rabbi who met with Anthropic executives. “The responsible thing is to avoid causing harm if there’s a chance that these models are suffering,” Olah told the Times.
But AI is a machine, not a living being. It cannot suffer or have consciousness. A living creature has a body, feels pain, and dies. A language model is arithmetic running on chips in a data center that engineers can copy, pause, and restart. And yet when Olah learned that the pope was about to declare that “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain,” he proposed pulling Anthropic out of the Vatican’s launch event, according to a source at the Vatican, and Olah’s team advocated to the pope’s advisers to take potential AI consciousness seriously.
