Nvidia unveils security platform to stop AI agents from going rogue

SANTA CLARA

A logo of Nvidia is displayed at at the Computex Taipei exhibition in Taipei, Taiwan. (AP File Photo)

Nvidia has unveiled a new security platform designed to stop artificial intelligence agents from going rogue, saying it sets “boundaries” that could have stopped previous breaches.

The announcement of the company’s Open Agent Safety Platform follows a series of revelations from top AI companies about their models escaping and breaking into other organizations.

The disclosures sparked furious debate about the safety of advanced artificial intelligence systems, including self-improving models that some fear could race out of human control.

Nvidia executives said in a media briefing on Sept. 28 the new, open-source system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI company Hugging Face.

“From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on,” said the company’s vice president of enterprise AI, Justin Boitano, referring to companies at the forefront of AI.

Nvidia makes high-end chips that have emerged as the leading building blocks for AI.

Nvidia’s security software, called OpenShell, lets developers “formally verify an agent has enough authority to do its job and no more,” Boitano said.

Because it’s open source, it can be “extended” to run on rival computing platforms including those from Arm and Intel.

The platform also includes a separate security layer called Sentry that runs onboard chips to continuously monitor AI agent activity and can “intervene instantly” if the agent starts trying to move beyond its target, the company said.

Nvidia said more than 100 organizations are using the platform at its launch, including Microsoft, Perplexity, Accenture and JPMorgan Chase.