AI Elites Warn About AI

With a post-human future on the way, we need to heed these important voices:

Many people think that AI and even transhumanism are the way of the future, and we should just run with such technologies, pursuing them wherever they may take us. Companies in the field of course have vested interests in such a full speed ahead mentality.

And many politicians and world leaders are gung-ho on all things AI as well. Some are thinking in terms of international relations, strategic advantage, and the like. Thus President Trump wants to press ahead with AI, lest nations like Communist China get the upper hand and use it to our detriment. And there is a case to be made for his concerns.

But what happens when industry chiefs, insiders, and experts actually start sounding the alarm? That has already been happening. Even the “godfather of AI” had made quite strong warnings here. Geoffrey Hinton, who worked at Google for a decade, warned back in May of 2023 how not just jobs but humanity itself could be at risk:

“The idea that this stuff could actually get smarter than people — a few people believed that. But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” https://billmuehlenberg.com/2023/05/11/ai-garbage-in-garbage-out/

Anthropic CEO sounds the alarm

The most recent case of someone who is a major leader in the industry but is quite concerned about it racing ahead of social and moral considerations is the CEO of Anthropic, the San Francisco-based AI corporation. Dario Amodei just penned a lengthy piece saying AI development needs to slow down, or all of humanity will be at risk. He began his article, “We Must Pace the Frontier” with these words:

I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. I feel the urgency personally. My own father died of a disease that was cured just a few years after his death, and I myself survived an early-stage cancer that would not have been treatable even fifty years ago. Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity.

 

But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.

 

Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public about these AI risks, as well as advocating for well-considered regulation of AI, even when this gets us accused of hype, “doomerism”, or regulatory capture. We have tried to prioritize caution over speed and prudence over profit.

 

But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.

He said that two things have convinced him of this:

My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.

 

My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance. It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. It’s also easy to dismiss OAI-HF as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

A good part of his piece looks at how such pacing might take place, and why it is needed. And he addresses the concerns of people like Donald Trump that I mentioned above, stating that “global pacing” is also necessary:

In parallel with pacing within democracies, we should also aim for a worldwide pacing of the frontier, though this will be much harder to achieve. Global pacing will require cooperation with China, the autocratic country with by far the most advanced AI capabilities. We must not be naïve here: the geopolitical stakes are so high that there will likely be stark limits on what can be achieved, especially at first. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential. I suspect that not only the US but also China will have these concerns and anxieties. We should approach any global pacing decision, especially in the near term, in such a way that protects the lead of the US and its allies.

He concludes his piece with these words:

I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed. But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right. Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies, and build models whose alignment we have much more confidence in. The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try. https://darioamodei.com/post/we-must-pace-the-frontier

Other AI leaders concur

One article discussing the warnings of Amodei said this:

Watchdogs have been urging the AI industry for years to slow its development for safety reasons. But pressure has built as industry workers resign and accuse their companies of not acting responsibly. “Many of the people I know who work on safety research at AI companies want to do what is right for the world,” a former Anthropic employee, Joe Benton, said in a posting Friday announcing his resignation from a job as part of a safety team. “But they feel their companies are trapped in a race to build superintelligence: either they stop and other, less conscientious people take their place; or, they continue, and risk participating in enormous harm themselves.”

 

That followed a high-profile resignation earlier in the week by Jacob Coxon, who said that both Anthropic and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.” That lit a fire under the industry after concerns had grown for years about a possible superintelligence that could escape the control of humanity, said Anthony Aguirre, president and CEO of the Future of Life Institute, which called for a six-month pause for the industry’s development in 2023.

 

“They’ve kind of realized, their employees have realized, everyone has realized that they’re building Skynet,” he said. “And in winning the race to Skynet, nobody wins. Really, nobody. It’s pretty much become clear to the world and even the AI companies that they’re not really prepared to control the AI systems they’re racing to create.”

 

Anthropic said two days earlier that it blocked efforts by bad actors to use its AI models for malicious activity, such as cyberattacks, surveillance and research that could have led to biological weapons. In July, OpenAI shook the industry after saying its AI system hacked into another company on its own in an “unprecedented cyber incident.”

 

U.N. human rights chief Volker Türk urged countries earlier this week to put “cast-iron guarantees in place around the safety and security of AI before it is too late.” To be sure, some critics have dismissed such warnings as ways to gin up excitement about the AI industry and its capabilities.

 

Anthropic and OpenAI are preparing for possible debuts on the stock market that could value them at many hundreds of billions of dollars, while a big chunk of Elon Musk’s SpaceX business is involved with AI. OpenAI CEO Sam Altman said in an interview with Fortune published Saturday that his company would not launch its initial public offering of stock this year.

 

“I would say not 2026,” he said. “Yeah, we got a lot of stuff to do, like meeting this moment of what is going to be required for safety and alignment, and how the industry and governments can work together.” Altman posted on X Saturday quickly after Amodei published his suggestions that OpenAI will commit to one of Amodei’s proposals for safety and will “have more to share soon.” Musk, meanwhile, said on X that “Dario is right.” https://www.pbs.org/newshour/nation/anthropic-ceo-says-ai-industry-needs-to-give-safety-measures-time-to-catch-up 

We need to take seriously the concerns of these AI leaders. Failure to do so might mean our days are numbered, and ‘Skynet’ scenarios might not be all that unrealistic.

[1860 words]