Layup Unveils Korean LLM Inference Acceleration Model, Releases Open Source Version for Hangeul Day
TECHWORLD · · 2 views
✦ AI Summary
Layup has made public the LLM inference acceleration draft model "GLM-5.2-NVFP4-DFlash-ko" on Hugging Face for Korean queries and agent tasks.
The model applies z-lab's DFlash method and is a draft model used together when running Z.ai's "GLM-5.2" in NVFP4 mode.
Layup separately trained the model on Korean data and recorded MAL of 2.71 and 2.77, while also developing a model for "GLM-5.3-Flash" and a Korean diffusion language model with about 11 billion parameters.
Layup has unveiled an LLM inference acceleration model for Korean query and agent tasks. The open-source release was timed for Hangeul Day. Layup said it recently made the draft model, "GLM-5.2-NVFP4-DFlash-ko," public on Hugging Face.
The model applies the DFlash method released by z-lab. Its purpose is to serve as the draft model used together when running Z.ai's open LLM "GLM-5.2" in NVFP4 mode. NVFP4 is NVIDIA's 4-bit floating-point format.
A draft model is a small model that uses Speculative Decoding. It works by first predicting multiple tokens that are likely to be generated next. The large model then validates the predicted tokens in batches.
Only the tokens that pass this process are accepted. The large model makes the final decision. As a result, it improves generation speed while maintaining response quality.
The DFlash method is structured so that the draft model generates multiple tokens in batches, replacing sequential token-by-token prediction. The speedup depends on how many tokens the draft model can get right at once, and as the prediction hit rate rises, the number of tokens accepted in a single validation increases. As the number of accepted tokens increases, the resulting acceleration effect also expands.
Layup found that prediction hit rates dropped in Korean queries and Korean agent tasks in its existing draft model trained on English data. This drop in prediction hit rates led to weaker acceleration effects. Accordingly, Layup separately trained a draft model based on Korean data in order to maintain acceleration in Korean environments.
Based on measurements from actual Korean agent traffic, the average acceptance length (MAL) was 2.71 for agent tasks and 2.77 for Korean queries. Layup described this as acceleration performance comparable to the English DFlash model. By contrast, the average acceptance length of the English-trained draft model in Korean environments was in the 1-point range.
Layup is developing a Korean draft model for Z.ai's "GLM-5.3-Flash" while also training, from scratch, a Korean diffusion language model with about 11 billion parameters.
This diffusion model is intended to be an on-device model that performs agent tasks on local devices. Training is being carried out in a GPU cluster environment with 64 B200 units, and some resources are supported by the Ministry of Science and ICT and the National IT Industry Promotion Agency (NIPA) through the "Advanced GPU Utilization Support Project."
Layup CEO Shin Jeong-gyu said the reason for releasing the Korean draft model is that it is difficult to fully secure the Speculative Decoding acceleration effect needed by Korean users with an English-trained draft model alone. He added that the company is releasing the Korean draft model on Hangeul Day so that AI service developers and companies can secure acceleration effects in Korean environments.
Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=408065
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.