Motif Files Objection to 2nd Evaluation of Korea's Sovereign AI Project; NIPA to Issue Decision Within 15 Days
TECHWORLD ·
✦ AI Summary
Motif Technologies has objected to the results of the second-stage evaluation in the Sovereign AI Foundation Model project.
Motif demanded disclosure of detailed scores, the basis for point allocation, and the evaluation standards and methods, and the government has begun reviewing whether the evaluation process was appropriate.
The National IT Industry Promotion Agency (NIPA) plans to review Motif's objection and notify the result within 15 days of receipt.
Motif Technologies has filed an objection to the results of the second-stage evaluation in the Sovereign AI Foundation Model project. On the 27th, Motif issued a statement disputing the results of the second-stage evaluation and demanded disclosure of detailed scores by evaluation item, the basis for assigning benchmark points, the standards for expert and user evaluations, and the methods used for expert and user evaluations. Motif said the detailed scores, the basis for point allocation, and the evaluation standards and methods must be disclosed so that the evaluation process and results can be verified.
In response, the government has begun reviewing whether the evaluation process was appropriate. The National IT Industry Promotion Agency (NIPA) plans to review Motif's objection and notify the result within 15 days of receipt. NIPA said it will announce the result within 15 days after reviewing the objection.
The review is expected to focus on whether the existing evaluation was conducted in accordance with the prescribed procedures and methods, rather than on re-scoring the results. The focus is expected to be on confirming compliance with the procedures and methods used in the existing evaluation, rather than on revising the results.
The background to Motif's objection is the gap between the benchmark ranking and the overall ranking. Motif's 'Motif 3' earned 47 points in AAII. Upstage scored 37 points in AAII, SKT scored 35 points in AAII, and LG AI Research scored 31 points in AAII.
Although 'Motif 3' outperformed Upstage, SKT, and LG AI Research in AAII scores, it finished fourth in the final overall evaluation. Motif said this result underscores the need to disclose detailed scores, the basis for point allocation, and the evaluation standards and methods so that the evaluation process and results can be verified.
The Ministry of Science and ICT, after receiving Motif's request and the consent of the other elite teams, disclosed the detailed scores from the second-stage evaluation on the same day. In the second-stage overall score, SKT ranked first with 70.6 points, followed by Upstage with 69.9 points, LG AI Research with 69.0 points, and Motif with 65.8 points.
However, the rankings differed by evaluation item. The second-stage evaluation method combined 40 points for benchmarks, 35 points for expert evaluation, and 25 points for user evaluation. The benchmark category consisted of 25 points for AAII and 15 points for the National Information Society Agency (NIA) evaluation, while the user evaluation consisted of 15 points for professional users and 10 points for the general public.
Motif received the highest score in the benchmark evaluation. However, it posted relatively low scores in the expert and user evaluations, which pushed it down to fourth place overall.
Separately from the objection, Motif said it would not participate in the third-stage evaluation of the Sovereign AI Foundation Model project. As a result, the subsequent competition will continue as a three-team lineup centered on SKT, Upstage, and LG AI Research.
Separately from this controversy, the government is reviewing the evaluation and competition framework for the Sovereign AI Foundation Model project. The purpose of the review is to examine whether the existing evaluation standards reflect the latest technological level and competitiveness amid rapid changes in AI technology. Deputy Prime Minister and Minister of Science and ICT Bae Kyung-hoon said at the 2nd Science, Technology and AI Future Strategy Meeting on the 27th that a fundamental review of the validity of the evaluation and competition framework devised 1 year and 6 months ago was necessary, and noted that the AI trend cycle is 3 to 6 months.
Deputy Prime Minister and Minister of Science and ICT Bae Kyung-hoon said the government would consider concentrating investment to develop world-class models and supplementing the evaluation system. The Ministry of Science and ICT is reviewing ways to reflect technologies that have recently grown in importance, such as AI agents, advanced reasoning, and coding, in future evaluations.
The Ministry of Science and ICT plans to discuss development directions and evaluation methods with participating companies. It also plans to push ahead with improvements to the evaluation framework to match technological changes.
NIPA's review of the objection and the government's review of the evaluation framework are proceeding in parallel, and as these two tracks intersect, attention is focused on the possibility of changes to the standards and methods that will be applied in the subsequent evaluations of the Sovereign AI Foundation Model project.
Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=406241
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.