Kakao Unveils Research on AI Model Training Optimization and Lightweighting at COLM 2026
TECHWORLD · · 1 views
✦ AI Summary
Kakao said it will unveil research results at COLM 2026 on reducing LLM training costs and shrinking model size.
At the main conference, it will present methods for improving the training efficiency of MoE models and a methodology for predicting the optimal learning rate.
At TokShop, it will unveil byte-based model lightweighting technology and the BBT architecture, and said it applied them to pretraining for Kanana-2.6-155b-a17b.
Kakao announced on the 8th that it will unveil its research results at the international conference on language modeling, COLM 2026. The topics it will disclose are reducing LLM training costs and shrinking model size. The presentations will be given in two sessions, including the main conference and the tokenization workshop, TokShop.
At the main conference, Kakao will present a method for improving the training efficiency of Mixture-of-Experts (MoE) models. The presentation will introduce a methodology for predicting the optimal learning rate of large-scale MoE models. The predicted cost amounts to about 1% of the total training cost.
The learning rate is a value that determines the scale of parameter updates during training. It affects both training stability and performance. Kakao proposed a way to apply the "mu-parameterization" technique to MoE architectures and scale training settings found in small models up to large models. It also explored the optimal learning rate by gradually increasing model size and the amount of training tokens. In addition, it verified whether settings obtained in short training runs remain effective in large-scale training.
At TokShop, Kakao will present byte-based model lightweighting technology.
Kakao said it applied the methodology to pretraining for Kakao's "Kanana-2.6-155b-a17b." Kakao explained that, as a result of the application, it was able to carry out training on a scale of 10 trillion tokens stably.
TokShop will then unveil the "BBT (BPE-Guided Byte Transformer)" architecture. BBT aims to reduce model size while improving performance. Unlike conventional BPE-based models that store information at the word and subword level, BBT uses byte units while preserving its processing of multiple-character groupings.
In comparative experiments, Kakao said that applying BBT reduced the number of parameters by 20.2% to 40.4% and improved test performance by 2.7% to 6.6%. It also said BBT improves robustness against typos and character perturbations and enhances transfer performance to languages not seen during training.
Kakao said it expects the technology to help reduce memory usage in on-device environments. It also said it expects the technology to improve model usability in environments with many multilingual inputs and typos. Kakao plans to expand the scope of its research to various model architectures and multimodal and multilingual environments.
A Kakao official said the research shows ways to maintain and improve AI model performance while reducing training and operating costs. The official added that Kakao plans to expand its research to a variety of model architectures and extend it to multimodal and multilingual environments, with the goal of increasing the potential for application in real-world services.
Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=407957
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.