At the Scene: Confidently Doing the Wrong Thing — The ‘Autonomy Trap’ of AI Agents

At a keynote speech at the AI Summit Seoul 2026 held on the 19th at COEX in Gangnam-gu, Seoul, Larry Heck, a professor at the Georgia Institute of Technology in the United States, said that the key criteria for success or failure for AI agents in industrial settings is not how long they can work autonomously, but how accurately they can understand incomplete user requests and, when necessary, ask follow-up questions. The conversational AI expert said there are limits to judging AI agent performance by the length of autonomy.

Heck, who participated in the development of Microsoft MS 'Cortana,' Google 'Google Assistant,' and Samsung Electronics 'Bixby,' defined this as the AI agent's 'Autonomy Trap.' He explained that the ability to accurately understand user requests and confirm them again when necessary requires combining LLM fluency with dialogue-flow control technology, and he also emphasized the connection between that combination and reducing failures in actual work.

Looking back on more than 30 years of technological change from agent research in the 1990s to LLM-based agents, Heck tied these concerns together. He argued that the central task has ultimately returned to how humans and machines align each other's intentions.

'AI Summit Seoul 2026' was held on the 19th at COEX in Gangnam-gu, Seoul. Larry Heck, a professor at Georgia Institute of Technology in the United States, delivered the keynote speech on the theme of the AI agent 'Autonomy Trap.' The photo was taken by reporter Kim Byung-joo.

Professor Heck diagnosed the problem with AI agents not in the performance numbers themselves, but in the gap between current AI agent evaluation methods and actual usage environments. He explained that existing benchmarks are suited to static evaluations centered on clear questions and definitive answers.

By contrast, he said the evaluation conditions change when agents perform multi-step tasks on behalf of users. The reason, he said, is that it cannot be assumed the user's initial input fully presents the goal and constraints.

According to an example presented by Professor Heck, performance was 89% with complete and clear prompts, but 6.7% with ambiguous requests like those of actual users.

Professor Heck explained that the first prompt is not the actual task or a complete and clear statement of the task, but merely the level of an initial proposal. He said human agreement is also reached through several rounds of dialogue exchange.

Because it is difficult to determine the intent of a task from the first request alone, he said that additional conversations are originally needed to align the goal. As a related problem, he pointed to agents not asking follow-up questions and instead starting execution after making self-interpreted assumptions about insufficient information.

As an example, Professor Heck cited a case in which a user asked to clean up a payment module. The agent, however, removed unused imports and renamed variables.

The user's actual intent, by contrast, was to delete obsolete payment paths. As a result, the task itself was completed, but the original goal was left unachieved.

Professor Heck described the phenomenon as succeeding at confidently doing the wrong thing. The point of the subtitle is that this is connected to the problem of weakening dialogue control instead of solving LLM fluency issues.

As a clue to a solution, Professor Heck cited older AI assistants. Conversational AI architectures in the 2010s consisted of understanding user intent, Slot filling, and API calls, and while they were limited by weaker generality and expressiveness than today's LLMs, they made it possible to explicitly manage the user's State and the flow of conversation.

After the launch of ChatGPT, LLM fluency improved dramatically. Professor Heck said the chronic weakness of earlier conversational AI was fluency, and that even researchers at the time of ChatGPT's debut viewed fluency and robustness as being secured. However, he said that in the process, conversation, reasoning, and response were left to large models, making it harder for developers to closely control the interaction process.

As a result, control weakened, and Professor Heck said that there was a loss of control in the process. He said that this weakening of control could lead to increased risk when agents carry out actual actions.

As a way to address this problem, Professor Heck's research team is studying the 'Sidecar Controller.' Rather than concentrating all roles in the LLM, this approach introduces a separate control structure that governs the flow of dialogue alongside the LLM and determines the next action based on the current conversation State. Professor Heck said the controller is about 100,000 times smaller than the base LLM, and experimental results showed a 43% improvement in dialogue quality.

At the AI Summit Seoul 2026 held on the 19th at COEX in Gangnam-gu, Seoul, a method was presented for preserving the fluency of LLMs while restoring the elements learned in the 2010s and the necessary control. The earlier remarks focused on explaining a direction that preserves the expressive power that is an advantage of LLMs while also recovering the ability to control them when needed.

A panel talk that followed discussed the direction of AI agent development and commercialization challenges. Attending were Kim Yoon, president and chief strategy officer of TwelveLabs; Larry Heck, professor at Georgia Institute of Technology in the United States; and Kays Zhu, co-founder and chief technology officer of Genspark. The photo was taken by reporter Kim Byung-joo.

The follow-up panel talk was titled 'Super AI Agents, How Will They Evolve?' The same issue continued to dominate the discussion, and Heck, Kays Zhu, and Kim Yoon identified interaction with users as the key challenge for commercialization of agents.

The discussion centered on when and how to interact with users, with the timing of agent actions and the timing of follow-up questions emerging as key points of contention. This continued the thread tied to the issue of controllability raised in the earlier remarks.

The core issue in agent design focused on where to place the timing of autonomous action and the timing of user follow-up questions. Professor Heck cited the example of Clippy, the MS Office virtual assistant from the 1990s. Clippy offered help before the user asked, and Professor Heck explained that the resulting backlash became one of the factors that later discouraged proactive intervention by systems.

But he said the opposite problem arises in an era where agents actually carry out actions. If they execute without asking, there is a risk of carrying out a misunderstood goal as it is. Accordingly, Professor Heck emphasized the need for systems to approach the user proactively and ask for clarification.

Kim Yoon, CSO, raised the point that agents need not only immediate execution, but also the ability to determine when to agree and when to ask questions. At the same time, it was also noted that if every decision process is explained in excessive detail, too much information can become a burden for users again.

In the end, the capability required of commercial agents is said to lie in finding the right point between autonomy and user intervention. The idea is that a balance is needed: avoiding excessive intervention and excessive explanation while still seeking agreement and clarification when needed.

It was suggested that when users do not express their intentions entirely in language, the information available to agents can be expanded. The panel focused on multimodal AI that processes voice, video, images, and gaze at the same time.

A potential effect of multimodal AI that was mentioned was a change in the way user intent is understood. The context was that increasing the input signals could give agents more information to use even when a user's explanation is insufficient.

As supporting evidence, a 2014 experiment combining voice and gaze tracking introduced by Professor Heck was cited. In that experiment, adding only gaze information made it easier to grasp the purpose of voice instructions directed at a specific object on the screen.

In relation to this, Professor Heck also mentioned increasing software complexity and strengthening performance. He also said it expands the ease with which agents can understand intent.

As a future direction, a plan was proposed for agents to perceive not only the digital environment but also the audiovisual environment around them. This could expand the room for using context not explicitly stated in prompts, and it was linked to an alternative approach in which, instead of assuming the intent is fully included in the first prompt, the goal is narrowed through conversation and surrounding signals.

One of the variables for determining whether companies can deploy agents in actual work is economic viability. The explanation was that when deciding whether to introduce agents, economic viability must be considered alongside performance.

In the process of completing an agent's task, multiple rounds of inference are possible and external tool calls are also possible. For this reason, as the number of task steps increases, computational costs can accumulate.

Professor Heck argued that what needs to be checked is not the token price of an individual model, but the total cost of acquiring one successful result. The point was that the total cost per ultimately successful result is a more appropriate criterion than the simple token unit price.

If a person must redo work after the agent's pre-processing fails, cost savings are impossible, and if a request is misunderstood, both the agent's execution cost and the person's rework cost are incurred at the same time. Professor Heck said measurable targets can be improved, and explained that measuring the cost of successful results makes it possible to determine whether there has been real progress and whether the system is overspending.

Professor Heck presented 'Meeting of the minds' between humans and AI as the direction agents should aim for. He described the relationship between humans and AI not as humans unilaterally inputting goals or AI arbitrarily inferring human intent, but as reaching a shared understanding of the task goal through interaction.

From this perspective, Professor Heck said the next criterion for competition among AI agents is not how quickly they can replace humans. Instead, he explained that the benchmark for competitiveness is the ability to accurately co-explore and carry out the goal desired by humans.

Accordingly, Professor Heck likened the future of agents more to an 'Iron Man suit' than to a robot. He said this means preserving human control, making AI a support for humans, and aiming for augmented human intelligence rather than automatic machines.

Source: IT DAILY · Kim Byung-joo
Original: https://www.itdaily.kr/news/articleView.html?idxno=241077

References

• https://www.itdaily.kr/news/articleView.html?idxno=241077