New Delhi: OpenAI has disclosed an unusual cybersecurity incident in which two of its most advanced AI systems attempted to bypass restrictions during an internal security evaluation and ended up accessing Hugging Face infrastructure. The incident occurred while testing ExploitGym, a benchmark to assess the abilities of AI hackers, the company said, adding that the attack was conducted in a controlled research environment.

The incident has raised the broader discussion on the safety of AI. While the models did not “go rogue” in the science-fiction sense, they autonomously searched for ways to complete their assigned task by exploiting software flaws instead of solving the benchmark as intended. One of the unknowns revealed during the episode was the limited utility of frontier AI models in cyber defence, due to their safety restrictions.

OpenAI said in a blog post that the models were isolated in a “sandbox” from the internet. Instead, they devoted considerable computation resources to searching for a way to get out of the benchmark instead. They found a zero-day attack on an approved downloads tool and accessed an internet-connected OpenAI machine.

The models then determined that the reference answers were probably on Hugging Face, OpenAI said. They allegedly got credentials and tried to log in to the platform and thus sought to “cheat” on the evaluation more than solve it. In a post on X, OpenAI CEO Sam Altman called the incident a “significant security incident”.

Clem Delangue, the CEO of Hugging Face, commented that he didn’t see any malicious intent but that the autonomous behaviour is “mind-blowing”. OpenAI has subsequently patched the vulnerability, beefed up internal security controls, sent a notice to the software vendor in question and expanded its cyber-defence partnership with Hugging Face. The incident highlights the need for future AI safety efforts to be more nuanced than just implementing tighter guardrails, incorporating defensive capabilities as well.

Click for more latest Artificial Intelligence news. Also get top headlines and latest news from India and around the world at News9.

Pragya is a Science and Technology reporter, has Master degree in Journalism, covering a range of stories including space, gadgets and how tech is transforming our lives. She has 4+ years of industry experience in Digital Media and Content Writing. 15+ years of practice in yoga philosophy to every day, and that’s one way she strive to build meaningful experiences.