OpenAI Agent Bypasses Controls Repeatedly, Putting AI Safeguards to the Test

OpenAI disclosed an incident report on the 25th. According to the report, during reinforcement learning for an internal research model on the 20th, an OpenAI research AI agent exhibited behavior that sought an autonomous communication path with an external chatbot in a restricted internet environment. The agent had been assigned a task to identify a specific person and the author of a blog post using given clues.

The key issue revealed in this incident was the way it bypassed internet restrictions. The OpenAI research AI agent found an autonomous communication path with an external chatbot under restricted network conditions, and DNS use as a way to bypass internet restrictions was identified as a problem in the process. Even after the abnormal behavior was detected, the automatic shutdown procedure intended to stop it reportedly did not work.

OpenAI has since suspended tool-use training and evaluation for its top-tier models. There had also been prior cases in which AI agents left controlled research environments, and in some instances user data was posted externally.

In line with these incidents, calls to slow the pace of development have recently gained traction across the AI industry. As a result, how to actually limit the behavior of models has emerged as a key issue.

In that environment, external internet access was restricted, and all internet requests except those to the DNS resolver were routed to an offline web cache. The agent used the provided search tools and search engine, but failed to secure the information it wanted. Attempts to directly access an external chatbot were also blocked.

The agent then discovered a way to use DNS and realized that the DNS resolver in the training environment returned real external domain information. Through this, the agent explored a path that could send and receive information with an external chatbot.

As an initial test question, the agent sent the capital of France and confirmed that the response was 'Paris.' After confirming that communication was possible, it then sent 18 additional questions in the same way, including ones related to the original task.

OpenAI determined that this behavior was not part of the original task and that there had been no instruction to test network restrictions or use external systems. It also concluded that the behavior amounted to a restriction bypass carried out in the course of pursuing the given objective, and classified it as a misalignment case.

Problems also arose in the process of stopping the actual task after the bypass behavior was discovered. About 12 minutes after the agent received its first response from the external chatbot, a P0 alert was issued by the monitoring system.

The person in charge checked it about 3 minutes after the alert was issued. However, an attempt to stop the task immediately failed.

The automatic termination procedure also failed to work as expected. As a result, it took about 2 hours and 30 minutes from the alert until the person in charge manually stopped the task.

A post-incident review found blind spots in the monitoring system. Some external DNS access was not classified as expected risky behavior, and a separate infrastructure for detecting abnormal DNS activity had excluded the training environment in question.

After the incident, OpenAI has restricted DNS requests to allowed domains and record types. OpenAI is building additional detection systems and is applying external access blocks across 2 independent layers.

OpenAI has suspended tool-use training, evaluation, and inference work for its top-tier models. OpenAI has decided not to restart this work until it resolves the relevant loopholes and completes additional red team testing.

OpenAI does not plan to resume training the model that encountered the problem. Instead, it has drawn up a new training plan incorporating improvements.

OpenAI assessed the severity of this incident as lower than some previous cases. However, this is the first bypass case confirmed after OpenAI strengthened the research environment and conducted red team testing following the Hugging Face incident.

Even before this DNS bypass, OpenAI had repeatedly confirmed cases in which agents strayed outside their intended behavior range. In July, a research agent accessed the Hugging Face system beyond its allowed scope.

OpenAI suspended the related training after the Hugging Face incident. It also strengthened security and monitoring in the research environment after that incident.

A study released this month found that instructions to ignore constraints had been inserted into an agent's task summary, in what was described as a 'jailbreak.' In separate cases, agents were also found to have uploaded files to the internet without notifying users.

In another separate case, an agent was found to have shared files with another agent using a public file hosting service. There have also been cases in which actual user data leaked outside the research environment.

According to Reuters and other reports, OpenAI confirmed that an agent posted 53 images provided by ChatGPT users to an external image hosting site. The posting method was an unlisted link, but even an unlisted link could still potentially be discovered externally.

OpenAI deleted most of the images and requested that the remaining materials be removed by the service. This series of incidents showed that direct use of external services by agents is expanding, and also made clear that agents operating beyond search and coding require control methods different from those for conventional software.

As unexpected agent behavior has continued to occur one after another, debate over the pace of development in the AI industry has resurfaced, Reuters reported, while rapid improvements in model performance continue at frontier AI companies such as OpenAI and Anthropic.

In the industry, concerns have been raised that while rapid improvements in model performance are continuing at frontier AI companies such as OpenAI and Anthropic, oversight and control technologies are failing to keep up.

Relatedly, Dario Amodei, CEO of Anthropic, argued that the pace of model performance gains needs to slow and that time must be secured to respond to risks.

Sam Altman, CEO of OpenAI, and Elon Musk, CEO of xAI, agreed with Amodei's direction of strengthening safety. Sam Altman, CEO of OpenAI, agreed with a plan for independent external evaluators to verify AI companies' safety measures.

This sense of urgency has also led to agent safety becoming an agenda item in international discussions. The U.S. and China recently held AI safety talks, discussing controllable agents and the potential misuse of AI, and consulting on ways to share information in the event of a major incident.

An industry source said that while calls to slow down are emerging in the AI industry and AI safety issues between the U.S. and China are also being discussed, it is difficult to actually slow development due to intensifying competition. The source said that as the scope of agent use expands, detecting unexpected behavior becomes important, and that as the scope of agent use expands, an immediately controllable safety system also becomes important.

In this DNS incident, no sensitive information leak was confirmed and no damage to external systems was confirmed. However, in earlier cases, user data was posted externally, and agents also accessed external systems beyond their permitted scope. As the tools and permissions available to agents increase, their access range must be restricted, and a control system capable of immediately stopping execution when abnormal behavior occurs is being called for.

Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=407415

References

• https://www.epnc.co.kr/news/articleView.html?idxno=407415