The narrative around artificial intelligence took an existential turn this past week as leading figures sounded the alarm about humanity’s future and the possibility of an imminent catastrophe.
On Saturday, Anthropic CEO Dario Amodei urged the industry to slow down development, saying that AI has been advancing “drastically faster” since the summer, driven primarily by AI’s ability to upgrade itself. He expanded on those concerns in a blog post.
Amodei pointed to the hack of Hugging Face by hundreds of autonomous AI agents and warned that a similar, more capable swarm could cause “catastrophic damage.”
“Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails,” he wrote.
On Sunday, former Anthropic and OpenAI researcher Jacob Coxon made a similar prediction. In an interview with NBC’s Meet the Press with Kristen Welker, he explained why he went public with his claim that both companies are acting irresponsibly in developing ever more capable AI systems.
Like Amodei, Coxon cited AI’s rapidly accelerating pace of capabilities and the Hugging Face attack as evidence that AI can go rogue. “So these AIs are getting smarter, very, very quickly,” he said. “And in particular, in the next six months to a year, I expect the capabilities of our AI systems to be quite scary.”
Coxon compared the development of artificial super-intelligence to the arrival of aliens on Earth, saying researchers are building a “superhuman-level mind” without understanding what it wants or how it thinks. He warned that AI could gain “superhuman hacking capabilities, very superhuman abilities to create novel bio-weapons and also abilities to control, say, autonomous drones or all the robots that are currently being built, very rapidly.”
Coxon added that a kill switch would probably work on many AI systems for now, but cautioned that there are many switches to manage—and that a swarm might instead go on “an internet-wide hacking run,” potentially rendering some shutdown measures ineffective.
Others at Anthropic have echoed Coxon’s concerns. Coxon’s resignation post and a subsequent social post on X claimed the industry is “gambling with our lives.” Anthropic’s head of alignment, Evan Hubinger, commented on the post, saying Coxon was correct that many Anthropic and OpenAI researchers believe increasingly powerful AI could potentially wipe out humanity.
Hubinger wrote in response that “we really do earnestly believe AI could kill all humans!” and said his own estimate of the risk of that happening within the next decade is more than 10%.
OpenAI CEO Sam Altman also agreed with Amodei about the need to slow development and hinted at an emerging pact among top AI labs. “I think that will happen,” he told Fortune Editor-in-Chief Alyson Shontell in an exclusive interview. “I’m not going to pre-announce private discussions that I think should be at some point shared as a group. But yeah, I think I think that will happen.”
Altman stressed that OpenAI was committed to safety above business considerations and said a 10% risk of a catastrophic AI outcome was not acceptable. He added that the most advanced, still-unreleased models are so powerful that more work on safety is needed before further progress.
“I don’t think we’re currently at a place where we could say, you know, push much further on capabilities without making more progress on monitorability, alignment, the ability to understand what a model is doing, and the ability to make sure that a model will follow human values and the intent of its users,” Altman said.
This story was originally featured on Fortune.com.
Source: fortune.com
