MangoBoost Targets the Inference Market With an 'AI Factory' as It Strengthens Hardware-SW Integration
TECHWORLD ·
✦ AI Summary
MangoBoost unveiled Mango Inference, an integrated AI inference platform combining hardware and software, and has begun expanding its service business.
MangoBoost is integrating infrastructure and inference services by combining its in-house DPU and inference software, while also strengthening cooperation with AMD.
The company is building 1MW infrastructure in Gangnam, plans to expand to 5MW by 2027, and is envisioning a global 500MW AI infrastructure business.
MangoBoost unveiled an integrated AI inference platform combining hardware and software as it moves to expand its service business. Going beyond supplying infrastructure technology, the company is pushing ahead with its own system-based AI inference services and is moving fully into its
On the 12th, MangoBoost held a press briefing at Grand InterContinental Seoul Parnas and unveiled its serverless AI inference platform, Mango Inference. At the event, CEO Kim Jang-woo introduced the company's AI infrastructure strategy, and CTO Kwon Dong-up presented the structure and technical features of the new service.
MangoBoost was founded in 2022 by researchers from Seoul National University. The company has developed DPU servers, GPU servers, storage systems, and AI inference software.
CEO Kim distinguished MangoBoost from neocloud companies centered on leasing GPU resources, presenting full-stack technology that designs hardware and software simultaneously as its core competitive edge. He said MangoBoost is an AI factory company different from neoclouds, and explained that it possesses true full-stack technology that handles both software and hardware.
Kim also emphasized the difference in business models from neoclouds. He explained that while neoclouds secure GPU servers and focus on leasing computing resources, MangoBoost integrates its own hardware and software and provides inference services as well.
Regarding this structure, MangoBoost said it connects GPU and storage through its in-house DPU and integrates inference software to optimize overall infrastructure performance and costs. Kim said this configuration is a way of encompassing both hardware and software.
MangoBoost is also deepening its collaboration with AMD. The company said it improved processing performance and cost efficiency by applying its own network technology and inference software to AMD Instinct GPU.
As part of this collaboration, in MLPerf Inference 6.1 it connected 32 AMD GPU in Korea and the U.S. into a single inference environment. As a result, it recorded scaling efficiency of up to 97% in MLPerf Inference 6.1.
MangoBoost is working to cut inference costs by combining the price competitiveness of AMD GPU with its own technology. Going forward, it plans to expand the use of its own hardware. At the same time, it is also pushing ahead with plans to use NVIDIA GPU according to customer needs and to use domestically made neural processing units (NPU) according to customer needs.
Mango Inference is an API service that provides open-weight AI model access. Companies and developers can issue an API key without building separate GPU infrastructure. They can also integrate AI models into existing applications.
CTO Kwon said the performance of open-weight models is approaching that of closed models. However, he said it is still difficult for companies to run open-weight models on their own. The requirements for in-house operation include securing GPU infrastructure, optimizing software for each type of hardware, and allocating servers to handle user requests.
In the current inference service market, some companies lease external neocloud GPU and then optimize the software. There are also operators that secure the latest GPU and supply computing resources.
Against this market backdrop, MangoBoost is pushing a strategy that integrates its own hardware, inference software, and API services. The company's goal is to secure both performance competitiveness and cost competitiveness.
CTO Kwon explained that it is difficult for companies to build their own LLM infrastructure. He said this requires understanding GPU architecture, software optimization, and multiple techniques for allocating requests to the appropriate GPU and cluster.
Accordingly, the company highlighted its own inference optimization software, LLM Boost, as a core technology. The company said LLM Boost optimizes GPU computation and the process of moving data from storage to GPU, thereby reducing bottlenecks. It also said that when applied to AMD MI355X GPU, some AI models delivered inference performance comparable to or higher than systems based on NVIDIA B200 and B300.
Meanwhile, the spread of AI agents has raised memory concerns. Because agents perform multiple tasks and repeatedly refer back to prior conversations and processing results, input context becomes longer over time, and the resulting increase in memory required for storage and processing has been presented as a challenge.
To respond to the growing demand for long-context processing, MangoBoost is developing KV cache layer management technology based on linking GPU memory with separate storage. This approach stores and reuses previous computation results.
The key is to reduce repetitive computation and improve memory efficiency. CTO Kwon explained that with the emergence of agents, processing context is becoming longer, and the importance of efficiently invoking existing processing data is also growing.
Mango Inference currently supports 5 models, including DeepSeek, GLM, and Minimax. The company plans to add new models sequentially.
The company said it is using AMD-based infrastructure. It also announced that it has priced the service at more than 20% below major inference service providers.
MangoBoost is building AI inference infrastructure at a data center near Dosan Park in Gangnam District, Seoul. It is also operating some infrastructure through a colocation model that leases space and facilities at existing data centers. In addition, it plans to build a 1-megawatt (MW) AI factory by the first quarter of 2027.
CEO Kim said the company is in the process of securing hundreds of the latest AMD GPU and has already received dozens of the latest AMD GPU. Inference services are being operated by combining its own infrastructure with neocloud GPU resources.
The company plans to gradually increase the share of its own infrastructure going forward, and it will also expand its business step by step. Kim said 1MW and 5MW are the plans for this year and next year.
MangoBoost plans to start with 1MW infrastructure in Gangnam and aims to expand to 5MW by 2027. In the longer term, it envisions a global 500MW AI infrastructure business as a mid- to long-term initiative through partner cooperation in the U.S., Japan, and Southeast Asia. Kim said it is in discussions with overseas partners, including those in the U.S., Japan, and Southeast Asia, regarding the 500MW scale.
As a way to secure funding for this large-scale investment, the company is discussing the establishment of joint ventures with domestic and foreign companies as well as external fundraising.
Expanding domestic infrastructure is presented not merely as a matter of increasing facilities, but as an issue tied to securing data sovereignty. CTO Kwon said that even with open-weight models developed overseas, operating a data center directly in Korea can increase control over data processing and management. Accordingly, not only securing models but also the location where actual inference takes place is seen as an important factor.
In addition, who operates the infrastructure and what management system handles the data generated during the service process are also presented as important factors. CTO Kwon explained that Korea should build data centers, operate them within Korea, and manage the data generated and accumulated in the process within Korea as well.
CTO Kwon also said that securing GPU alone is not enough. He stressed that who operates the system and whether accumulated data is managed within the country are important.
In this context, MangoBoost is pushing on-premises supply for sectors where data control is critical, such as finance and defense. The company plans to support AI model operations in customers' own environments and provide integrated hardware and software. It also plans to expand its business alongside API-based inference services.
Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=408052
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.