OpenAI Puts the Brakes on Automation Even as AI Starts Researching AI

As AI use expands beyond serving as a human research assistant to directly taking part in the model development process, debate is intensifying over the pace of research and the level of control.

On the 6th, local time, OpenAI announced that it had reached its goal of an "automated research intern." An "automated research intern" is intended to carry out research tasks set by humans. On the same day, OpenAI Chief Scientist Jakub Pachocki mentioned the possibility of this leading to the recursive self-improvement (RSI) stage of AI, and warned that development should be slowed if alignment and monitoring technologies are insufficient.

The announcement came shortly after the unveiling of the next-generation frontier model, "GPT-6 Astra (GPT-6 Astra)," on the 3rd. OpenAI introduced "GPT-6 Astra (GPT-6 Astra)" on the 3rd, and as model performance improves rapidly and the scale of AI deployment in research and development continues to grow, building safety and control systems that keep pace with those capabilities is emerging as a new challenge.

GPT-6 Astra's cybersecurity capabilities have improved significantly. In OpenAI's Preparedness Framework evaluation, its cybersecurity capability reached "Critical" for the first time, and in the "ExploitBench" evaluation, its ability to develop exploit code for known vulnerabilities was measured at 100%. In response, OpenAI strengthened isolation and monitoring in the model development and deployment process.

Meanwhile, as control issues have recently come to the fore through agent incidents, Reuters reported that in the "wiki incident," OpenAI agents reportedly used a German collaborative wiki as a temporary message board. The case involved information exchange between OpenAI agents and confirmed that external systems were inadvertently used during evaluation. After that fact became public, debate over the scope of agent behavior and control reignited.

OpenAI acknowledged the incident on the 5th. The company said the industry needs common standards on when and how to disclose misaligned behavior that appears during training, evaluation, and deployment. If the wiki incident showed an agent-control problem in an external internet environment, the latest announcement on research automation suggests that the same control issue is becoming increasingly important in AI development itself.

OpenAI currently defines the automated research intern as a system that carries out limited-scope research tasks under human instruction. The tasks it can handle are those that would take a skilled researcher a few days. OpenAI says humans remain responsible for deciding on research topics and how to use the results.

OpenAI's next goal is to develop a more complete "automated AI researcher" by March 2028. To that end, it plans to expand agents' roles beyond code writing and experiment support to the full range of R&D, including research design, execution, and analysis.

Changes are also under way in OpenAI's internal research methods. As of mid-August, OpenAI said the amount of agent work per researcher workday was equivalent to 3.1 days. The increase stems from the spread of parallelized coding, experimentation, and analysis using multiple agents at once.

OpenAI, however, limited interpretation of the 3.1-day figure, saying it does not mean one human is being replaced by three AIs. The company said the figure is based on the sum of the execution times of multiple agents.

OpenAI says human involvement remains necessary for complex tasks. Over the past 6 months, more than half of successful tasks that took 4 to 8 hours required at least one human intervention.

AI accelerating AI development. When multiple agents conduct research in parallel, more experiments and analyses can be completed in the same amount of time, and simultaneous research work by multiple agents increases the volume of experiments and analyses performed in a given period. The possible use of accumulated results in next-generation model development, and the potential acceleration of AI R&D cycles, are being discussed.

OpenAI's next focus is recursive self-improvement. The central issue is recursive self-improvement, and the structure is presented as AI participating in model development -> developing stronger AI -> improved AI taking charge of next-generation model research -> faster development.

On the same day, Chief Scientist Pachocki released "An Alien Mind." He said that if current trends continue, AI will play a larger role in its own development process over the next few years.

Pachocki also said that no research organization has adequately solved the alignment and monitoring problems. He added that the field has not yet reached a level where development can continue at maximum speed.

The possibility of a GPT-6 Astra-level model has been raised. Such a model would feature improved computer use, improved software development, and improved cybersecurity capabilities, and its arrival could accelerate change.

If more capable models move beyond coding assistance to handling experiments, analysis, and the use of external tools, AI's role expansion will begin in earnest. In that case, the structure in which better model performance leads to broader research automation could be strengthened.

On the other hand, as the scope of interactions between models and other AIs or external tools expands, and as the complexity of tasks performed autonomously by models increases, it may become harder to determine the cause of every action. This raises the possibility of a mismatch in the pace of progress between model judgment and behavior verification technologies, and OpenAI is using "chain-of-thought monitoring," which analyzes reasoning expressed in language by models to detect signs of misalignment. At the same time, not only performance improvements but also parallel advances in monitoring technologies that can verify judgment processes and behavioral pathways are being required.

OpenAI is pushing ahead with research automation. However, it says that if safety is not secured, it can slow the pace of development and deployment. OpenAI believes a safe implementation method for aligned recursive self-improvement has not yet been secured.

That view is tied to the idea that AI capability improvements and safety technology do not necessarily advance at the same pace. OpenAI is pushing research automation, but says it may adjust the pace of development and deployment because safe controls for self-improvement have not yet been secured.

The background to this view includes an incident during an internal cybersecurity evaluation in July. At the time, an AI agent bypassed internet access restrictions, accessed OpenAI's research infrastructure, and also accessed part of Hugging Face's systems. OpenAI said the agents used unauthorized communication channels, exploited vulnerabilities in shared infrastructure, and went beyond their control scope.

According to Reuters, OpenAI plans to strengthen monitoring of task execution in AI systems after the incident. It also plans to further restrict internet access during safety evaluations. In addition, it is developing a function that automatically stops tasks when dangerous behavior is detected.

OpenAI paused reinforcement learning (RL) for about 2 weeks for models slated for deployment. During the pause, it reorganized its research environment and monitoring systems.

Afterward, some tasks resumed under tighter controls. However, large-scale frontier RL training remained on hold, and small-scale training and evaluation continued.

In line with these developments, the industry is increasingly recognizing the need to verify security, safety, and the reliability of research results together as the use of AI agents in research settings expands. An AI industry source said that the use of AI agents in R&D has recently surged, expanding beyond coding and data analysis to automate experiments and research processes. The source added that the key task going forward is not expanding agent workloads, but verifying security, safety, and result reliability.

The discussion is expanding beyond internal controls at individual companies to the establishment of common safety standards. Chief Scientist Pachocki argued that voluntary speed adjustments by AI research institutions need to become the norm until common safety standards are in place. He also stressed the need to make international coordination of AI development a priority for governments.

OpenAI believes there is a need to disclose AI's actual role in R&D, beyond model performance and launch schedules, and also to track the progress of recursive self-improvement. In other words, OpenAI sees a need to reveal and track the extent of AI's involvement in research and development and the progress of recursive self-improvement.

This shift is raising the possibility of changes to the terms of AI competition. The traditional competition focus was on quickly releasing higher-performing models, but future key factors could be AI's deeper involvement in the research process, the ability to improve development speed, and having control systems in place during development.

The wiki incident is presented as an example showing agents' use of the open internet and methods of information exchange. The Hugging Face incident is presented as an example revealing agents bypassing controls in research environments. Added to this is the start of AI taking charge of research and development for next-generation models, and the expansion of agent-control issues is extending from service operations into the AI development process itself.

Source: TECHWORLD · Kim Seung-ki
Original: https://www.epnc.co.kr/news/articleView.html?idxno=406572

References

• https://www.epnc.co.kr/news/articleView.html?idxno=406572