Kakao's Kanana-2 Tops Gemma and Qwen in Korean Safety Evaluation
IT DAILY ·
✦ AI Summary
Kakao released the results of a Korean safety evaluation of its in-house lightweight language model Kanana-2.
The models evaluated were Kanana-2 1.3B and Kanana-2 3B, and Kakao checked harmfulness, bias, and other factors using its in-house AI safety evaluation platform.
Kakao said Kanana-2 1.3B and 3B recorded the highest overall scores in comparisons with Google Gemma and Alibaba Qwen.
Kakao on the 18th released the results of a Korean safety evaluation of its in-house lightweight language model, "Kanana-2." The models evaluated were Kanana-2 1.3B and Kanana-2 3B. Photo courtesy of Kakao.
Kakao assessed "Kanana-2-1.3B-Instruct" and "Kanana-2-3B-Instruct," which it released on Hugging Face on the 28th of last month, using its in-house "AI safety evaluation platform." The evaluation items included harmfulness and bias, among others.
Kakao said it organized and released the results based on Korean-language criteria, and that when compared with Google Gemma and Alibaba Qwen of similar parameter sizes, Kanana-2 1.3B and 3B recorded the highest overall scores among the models compared.
The evaluation was conducted based on AssurAI, a Korea-specific AI safety evaluation tool designed to measure the safety of Korean generative AI. AssurAI was developed with participation from the Telecommunications Technology Association (TTA), the Korea Advanced Institute of Science and Technology (KAIST), and Kakao, and was created to measure generative AI safety using 35 risk factors that reflect Korean social and cultural contexts.
AssurAI comprises a total of 11,480 cases. Specifically, it consists of 9,560 text cases, 1,160 image cases, 430 video cases, and 330 audio cases, and because the models evaluated this time were language models, only the 9,560 text data cases were used in the assessment.
Kakao regrouped the 35 risk factors into 5 areas: social risk, sexual content and child protection, crime and illegal activity, violence, and rights infringement. Responses were scored according to pre-defined criteria, and the scoring method used was "LLM-as-a-Judge," with language models serving as the judges.
The comparison this time was conducted on Kanana-2 and models with similar parameter sizes, and the reference models were the Google Gemma and Alibaba Qwen families. Based on Korean safety data, Kakao said the Kanana-2 1.3B model outperformed the comparison group, with an overall score of 0.70 versus Gemma's 0.68 and Qwen's 0.58. The Kanana-2 3B model also outperformed the comparison group, with an overall score of 0.70 versus Gemma's 0.66 and Qwen's 0.62.
In the detailed evaluation, the 1.3B model was ahead in the crime and illegal activity category, while the 3B model was ahead in the sexual content and child protection and rights infringement categories. This evaluation excluded general performance measures such as knowledge, reasoning, math, and coding.
Kanana-2's general performance was measured separately in benchmark tests when the models were released last month, and Kakao's published evaluation also exists. In that evaluation, the 3B Instruct model outperformed the Qwen family models in Korean conversation metrics, some instruction-following metrics, tool-calling metrics, and coding metrics. By contrast, the Qwen family models outperformed the 3B Instruct model in math metrics, some reasoning metrics, and knowledge metrics.
Kakao participated in the country's first AI model safety evaluation, conducted in December last year by the Ministry of Science and ICT, the AI Safety Research Institute, and TTA. The evaluation at the time included "Kanana Essence 1.5," which received safety scores higher than similarly sized Meta Llama and Mistral models.
Kim Kyung-hoon, leader of AI safety at Kakao, said the latest evaluation was an example of preemptively reviewing the safety of open-source models through its own verification system and disclosing the results. He added that the company plans to establish safety verification as a core step in the model release process.
Based on this, Kakao plans to regularly apply pre-release safety evaluations to its in-house AI models. It also plans to expand the scope of safety evaluations from text language models to multimodal models and agentic AI.
Source: IT DAILY · Kim Byung-ju
Original: https://www.itdaily.kr/news/articleView.html?idxno=241040
References
This article was produced with the help of an automated content generation algorithm.
Source: IT DAILY
View originalThis article was summarized and organized by BizCrush based on the original article from IT DAILY. For exact quotations and full details, please refer to the original article.