Appier: “AI Must Know What It Doesn’t Know” ... Limits of LLMs’ Awareness of Missing Information Confirmed
TECHWORLD ·
✦ AI Summary
Appier’s AI research team announced two research papers on the 10th related to LLMs’ awareness of missing information and multilingual reasoning characteristics.
In the first study, 28 major LLMs showed accuracy that was 30% to 50% lower on questions for which “none of the above” was the correct answer than on ordinary questions, and applying DPO improved accuracy in identifying questions with no correct answer by about 30 percentage points.
In the second study, the internal reasoning language affected performance, with high-resource languages such as English showing relatively strong results in math and knowledge-based tasks, while local-language reasoning proved better for problems requiring cultural context.
Appier released research findings analyzing LLMs’ ability to recognize when information is lacking and how the choice of reasoning language for different tasks affects performance. As the scope of enterprise applications for agentic AI expands, Appier argued that what matters is not just the ability to generate answers, but also the ability to distinguish situations in which the model does not know the answer and to choose an appropriate reasoning method.
Appier’s AI research team announced two research papers related to LLMs’ awareness of missing information and multilingual reasoning characteristics on the 10th. The release addressed not only LLMs’ ability to get answers right, but also the importance of recognizing when no answer exists and selecting the right reasoning approach.
The first of the two papers focused on whether a model can recognize when there is no correct answer. To test this, the team designed experiments by adding a “none of the above” option to multiple-choice questions.
The experiment covered 28 major LLMs of different sizes. The results showed that accuracy on questions for which “none of the above” was the correct answer was 30% to 50% lower than on ordinary questions.
Appier explained that even when the models had relevant knowledge, they tended to pick another option rather than judge the situation as one of insufficient information when no valid answer existed.
To address these limitations, the research team applied supervised fine-tuning (SFT) and direct preference optimization (DPO). DPO was run as a method that learns correct and incorrect answers at the same time, and when applied, accuracy in identifying questions with no correct answer rose by about 30 percentage points.
However, the same approach was not effective across all tasks. In problems with clear answers and mutually independent choices, the “none of the above” approach performed relatively well. By contrast, in other tasks, the team found it necessary to design both an answer-abstention approach and an approach that requests additional information, tailored to the characteristics of the task.
This kind of judgment capability can be combined with retrieval-augmented generation (RAG) in enterprise settings. If search information is insufficient, AI can perform additional retrieval instead of generating an immediate answer, and the process can also lead to transferring the decision to a human.
In a separate second study, the team analyzed how the language used for AI reasoning affects outcomes. The research focused on large reasoning models (LRM) and compared the relationship among the input language, internal reasoning language, and final response language.
In this experiment, the team compared how the model internally chose its reasoning language regardless of the question language and the resulting performance differences. It found that even when receiving questions in other languages, the model frequently used high-resource languages such as English as its internal reasoning language. In some models, the mismatch rate between the internal reasoning language and the final answer language exceeded 90%.
The team then used “text prefilling” to induce reasoning in a specific language and observed performance changes by task. As a result, high-resource languages such as English showed relatively strong performance in math and knowledge-based tasks.
By contrast, in problems that required cultural context, reasoning in a local language was found to better reflect regional characteristics. Similar language-based differences were also observed in safety evaluations, and in identifying harmful or illegal questions, local-language reasoning was in some cases more effective.
Based on this, Appier proposed the possibility of “reasoning language routing,” which switches the reasoning language according to task, region, and cultural context. Under this approach, AI would choose an internal reasoning language suited to the situation, while the final answer would be provided in the language the user wants. Appier compared this to selecting a model or a tool.
Chaihan Yu, CEO and co-founder of Appier, said that since AI has moved beyond the question-answering stage and begun autonomous decision-making, evaluating AI solely on whether it produces the right answer is insufficient. He said that evaluation should therefore also include the ability to recognize situations with missing information and adjust behavior, as well as the ability to select a reasoning method appropriate to each task.
Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=406790
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.