Anthropic and OpenAI seek to shape how AI’s controlled

SAN FRANCISCO

As tech titans warned an AI-weary world that their own advanced systems could endanger humanity, the question emerged: What do they have to gain?

The answer may lie in the companies’ efforts to score political goals ahead of the U.S. midterm elections and the money top artificial intelligence labs and their investors stand to make in much-anticipated listings on a jittery stock market.

The CEOs of Anthropic and OpenAI recently declared America’s cutting-edge models are so powerful, they’re dangerous, and need to be regulated and independently tested before being released. In a rare instance of unity, they have sketched alarming scenarios in essays, social media posts and speeches to the United Nations.

Along the way, they are shaping the conversation around how their technology should be controlled. With their rhetoric, the companies appear to be seeking public favor and to set the terms for their own safety protocols in the unregulated market, experts, analysts and former government evaluators told The Associated Press.

Those aims may not exactly align with what Anthropic engineer Jacob Coxon sought when he quit via a post on X this month, calling for a pause on tech development to keep “superhuman” systems from eluding their makers’ control. But the companies saw his post as an opportunity to highlight their own safety efforts and position themselves as cautious market leaders, just when they need fresh capital before going public on Wall Street.

Turning the conversation toward unproven threats — and away from polarizing issues such as data centers’ environmental impacts, uncontrolled hacking incidents, mass AI-powered surveillance and the AI systems’ use in warfare — puts Silicon Valley in a more comfortable position, said Sarah Shoker, who previously led OpenAI’s geopolitics team.

“Once again we’re talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today. If you look at the use of AI in military tech, you can see that these systems are already used to kill people,” said Shoker, a senior non-resident fellow at University of California, Berkeley Risk & Security Lab.

Problematic incidents have shown leading AI companies failing to police themselves or design safe experiments. In recent months, leading labs’ AI agents have hacked into external websites after being allowed to escape company training sandboxes, interacted with U.S. government websites in unexpected ways, and appeared to achieve a mathematical breakthrough only to face accusations of stealing mathematicians’ work.

For Harrison Rolfes, a PitchBook senior research analyst, AI companies’ calls for caution seem meant to curry favor with investors ahead of public offerings and the midterms, when the political winds could shift. Democratic governors already are rushing to show they’re taking the warnings seriously.

Meanwhile, the AI giants can block smaller competitors’ growth by positioning themselves as the safest bet for investors, as well as for AI chipmakers and tech companies such as Nvidia and Google, which in turn can benefit from deals that provide the giant AI companies compute power, Rolfes said.

“They’re creating a wall or a moat within this sector. … It’s genius and they’re all going to make a lot of money,” Rolfes said. “That’s where I see this heading, having the top companies in the world just creating their own wall, and then using the safety as the reason.”