Motive Objects to Disclosure of Detailed Scores for 'Independent AI Foundation Model' Project 2nd Evaluation; SKT Tops Overall

The Ministry of Science and ICT has released the detailed results of the second-stage evaluation for the 'Independent AI Foundation Model' project. The disclosure came after Motif Technologies issued a statement the previous day requesting that the detailed evaluation results be made public.

The ministry said that, after Motif Technologies made its request, the other elite teams also agreed to the release of the detailed results. The ministry accordingly disclosed the evaluation scores.

The ministry stressed that the purpose of the project is not simply to compete for rankings or to select teams that will be eliminated. It said the goal is to foster the growth of domestic AI companies and the qualitative development and expansion of the AI ecosystem.

According to the Ministry of Science and ICT, in the disclosed Artificial Analysis Intelligence Index (AAII) benchmark breakdown, Motif Technologies scored the highest at 11.9 out of 25. It was followed by Upstage with 9.4 points, SKT with 8.8 points, and LG AI Research with 7.8 points.

The evaluation was divided into the NIA benchmark, external expert review, and professional user review. The NIA benchmark covered 7 areas: math, knowledge, long-context understanding, safety, reliability, Korean, and instruction following. The scores were SKT 13.4 points, Upstage 13.3 points, LG AI Research 12.8 points, and Motif Technologies 12.7 points.

The external expert review involved 10 outside experts from industry, academia, and research, who evaluated development strategy and technology, development results and plans, and expected impact and contribution plans. Team scores were calculated as the arithmetic mean of the remaining scores after excluding the highest and lowest among each committee member's evaluations, and the results were LG AI Research 29.5 points, SKT 29.3 points, Upstage 29.1 points, and Motif Technologies 27.1 points.

The professional user review involved 49 participants, including the CEOs of AI startups. The leading organization differed depending on the evaluation type: SKT recorded the highest score in the benchmark and professional user reviews, while LG AI Research recorded the highest score in the expert review. The professional user review scores were SKT 11.6 points, LG AI Research 11.3 points, Upstage 10.8 points, and Motif Technologies 8.4 points.

The general public review was conducted by an evaluation panel selected using gender and age quotas, and the number of panel members who actually participated was 185. The general public review scores were LG AI Research 7.6 points, SKT 7.5 points, Upstage 7.3 points, and Motif Technologies 5.7 points.

In the combined results of the separate evaluations, the SK Telecom (SKT) elite team took first place with 70.6 points. It was followed by Upstage with 69.9 points, LG AI Research with 69.0 points, and Motif Technologies with 65.8 points.

After the results were released, the ministry said it hopes companies will contribute broadly to the domestic and global AI ecosystem based on their underlying strength and capabilities, rather than focusing only on scores and rankings. It also said it expects the AI ecosystem to develop into a dynamic one in which new and competitive AI companies continue to take on challenges.

Earlier, Deputy Prime Minister and Minister of Science and ICT Bae Kyung-hoon raised the need to reconsider the evaluation method for the 'Independent AI Foundation Model' project at the previous day's '2nd Science, Technology, and AI Future Strategy Meeting.' He said there is a need for a fundamental reassessment of the effectiveness of the evaluation and competition framework established 1 year and 6 months ago, and noted that the cycle of AI trend changes is 3 to 6 months.

Source: IT DAILY · Yang Seung-gap
Original: https://www.itdaily.kr/news/articleView.html?idxno=241264

References

• https://www.itdaily.kr/news/articleView.html?idxno=241264