Vessl AI Supports Upstage's Solar Pro 4 Global Inference Operations
TECHWORLD ·
✦ AI Summary
Vessl AI announced on the 8th that it is supporting the operation of Upstage's LLM 'Solar Pro 4 (Solar Pro 4).'
It handled global serving for 'Solar Pro 4,' which ranked third in September usage on OpenRouter, and processed inference requests from users around the world.
Based on launch and promotion schedules and expected demand, Vessl AI predesigned inference capacity and improved performance using GPU clusters in multiple regions and its own inference engine.
Vessl AI announced on the 8th that it is supporting the operation of Upstage's LLM,
'Solar Pro 4' ranked third in usage on OpenRouter in September. During that period, Vessl AI handled global serving and processed inference requests from users around the world.
When new models are launched or during promotions, requests can surge in a short period of time. Accordingly, it is important to secure processing capacity based on anticipated demand in preparation for spikes in requests.
Vessl AI and Upstage shared their launch and promotion schedules as well as expected demand. Based on that, Vessl AI predesigned the necessary inference capacity and used GPU clusters in multiple regions, including Korea, Europe, and the United States, along with its own inference engine. It also allocated resources according to the model's characteristics and traffic patterns.
For performance improvements, the companies applied internal cache hit rate enhancements, model quantization, and inference speed optimization technologies. Vessl AI said that, based on comparisons before and after the endpoint was introduced, processing speed improved by an average of 2 times and by as much as 3 times, while token costs were reduced by up to 50%.
Vessl AI was responsible for demand forecast-based capacity design and also handled actual inference operations. As a result, Upstage was able to focus on model development and service expansion.
The company said it plans to provide global inference infrastructure tailored to the growing use of the Solar model. Vessl AI CEO An Jae-man emphasized that the global success of a model developed in Korea was supported by the country's neo-cloud infrastructure, and said the company aims to provide an environment where model providers can focus on research and product development based on its inference operations experience.
Source: TECHWORLD · Kim Seung-gi
Original: https://www.epnc.co.kr/news/articleView.html?idxno=407961
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.