AI PoC Fails 90% of the Time... “Inference Infrastructure and Cost Control Are the Answer”

The benchmark for AI competitiveness is shifting from individual models to full-stack infrastructure that combines data center power and cooling with a cloud software stack. With AI PoC failure rates of 90% being cited, inference infrastructure and cost control have been presented as the solution.

Against this backdrop, the Korea Artificial Intelligence Cloud Industry Association (KACI) held the 3rd Artificial Intelligence-Cloud Big Tech (AI-Cloud Big Tech) 2026 on the 1st. The event, themed “AI-Cloud Opened by Advance, Connect, and Transform,” drew about 300 people from public institutions and industry.

At the event, the latest technology trends spanning AI infrastructure, cloud platforms, and enterprise services were shared. In the opening remarks and welcome address, the shift from model-centered competition to ecosystem-linked competition was emphasized.

Kim Bong-gyun, chairman of KACI, said AI competitiveness is not completed by AI models alone. He explained that stable operation and scaling of large-scale AI services require computing infrastructure such as GPU and AIDC, as well as cloud systems that efficiently connect that computing infrastructure. He added that organic integration through on-site services is also needed.

Government and National Assembly officials said they were committed to supporting AI diffusion and on-site innovation. Kwon Eun-tae, an official at the Ministry of Science and ICT, said that AI, cloud, infrastructure, and SW companies must connect technology and experience across boundaries as a condition for improving AI productivity. Kwon also promised policy support from the government.

Choi Hyeong-du, a member of the People Power Party, conveyed his intention to support on-site innovation through a video message for the National Assembly's Science, ICT, Broadcasting and Communications Committee. He said he would push for regulatory improvements so that institutions do not hinder on-site innovation.

The event then featured a presentation by Koo Yong-jun, executive vice president of KT Cloud. Koo took the stage as the first keynote speaker.

The keynote topic was “Beyond GPU to AIDC: A New Benchmark in AI Infrastructure Competition.” Koo explained that AIDC needs to be redefined beyond simple building and power facilities into an “AI service center.”

Koo pointed out that in the past, users would conduct their own development if they had GPUs. He said that today, GPUs have become infrastructure as obvious as air.

He added that cloud SW stacks and orchestration platforms, including Kubernetes, need to be in place so developers can use them immediately. The photo shows Koo Yong-jun, executive vice president of KT Cloud, speaking, and the photo is credited to reporter Kwon Young-seok.

As server rack power demand has grown far beyond past levels, warnings about rapidly rising power density and physical limits have continued. Koo said that while conventional server racks used to draw around 5kW, the latest GB200 configuration reaches 120kW per rack, and next-generation architectures such as Vera Rubin are expected to reach 150-200kW per rack. He explained that, with these changes, the industry has also revised its scale benchmark to the point where anything below 10MW is considered an edge data center.

Koo also stressed the need to shift cooling methods. He said air cooling has an efficiency of 15%, while liquid cooling reaches 85%, making the move to liquid cooling essential. He also explained that, beyond communication between GPUs, network fabric design is important for resolving data plane bottlenecks between storage and computing nodes.

In response, KT Cloud plans to invest KRW 6 trillion by 2031. KT Cloud intends to build a total 1 gigawatt (GW) scale of AIDC in stages and apply its own cloud-native platform to link it to a multi-region nationwide backbone network as part of its “AIDC as a service” strategy.

HPE executive director Park Un-young gave a presentation titled “Overcoming the Limits of AI: HPE Private Cloud AI.” The photo credit is also Kwon Young-seok. The presentation laid out the reality of enterprise AI projects that remain at the pilot stage and presented an alternative to that reality.

Park explained in numbers the reality that enterprise AI projects do not easily move from proof of concept to production. Based on an IDC survey, he said the actual rate of AI PoC transitioning to mass production is 12%. He also said that, based on an MIT report, the actual transition rate is 5%.

Park described this situation as one in which 9 out of 10 projects fail. He said that rather than first expanding GPUs, companies need to examine whether they have a mass-production system that can deliver ROI. His point was that verifying an operating system with profitability should come before simply expanding GPUs.

He then explained that the market is shifting from training to inference. Based on Gartner data, he said the share of the training market and the inference market in 2026 will be 5:5. He also forecast that the inference market share will exceed 80% in 2028-2030.

Park explained that, with the spread of agentic AI, token costs per query are rising sharply. Based on this trend, he said it is becoming difficult to sustain reliance on large foundation models. He added that as market changes and rising cost burdens converge, a one-sided approach centered on large foundation models is hitting its limits.

It was also analyzed that when a domain-specific model (sLLM) based on a company's own data is operated in an on-premises environment, costs can be reduced by more than 5 times compared with public cloud.

HPE was then said to be cooperating with Nvidia. HPE is said to provide “HPE Private Cloud AI,” a turnkey appliance that has hardware, Kubernetes, and more than 500 AI open-source tools pre-integrated and validated.

“HPE Private Cloud AI” supports network-isolated environments for public-sector and defense use. It was also introduced as supporting quarterly patching and zero-downtime one-click updates.

At the venue, Kim Ji-hoon, director at Naver Cloud, gave a presentation. Kim Ji-hoon of Naver Cloud handled the final keynote speech.

The topic of Kim Ji-hoon's presentation was “A New Strategy for AIDC: AI Factory That Encompasses Neo Cloud and Hyperscalers.” Kim described the AI factory as an AI factory that mass-produces intelligence. The photo credit is Kwon Young-seok.

Kim said that the nature of AIDC is changing with the shift to AX. He explained that AIDC is evolving from a simple facility into an AI factory that continuously produces high-density intelligence (tokens), and that its role is also changing into a production base that supplies intelligence across the nation and industry.

He said the North American market is seeing rapid expansion in power scale, reaching 400MW in 2025, 1GW in 2026, and 1.6GW in 2027. Based on this, he argued that a combination of a Neo Cloud model centered on GPU rental and the capabilities of hyperscalers with large-scale stability is necessary.

Kim said Naver Cloud operates a single GPU superpod of 60,000 cards, has experience developing HyperCLOVA X, and possesses full-stack capabilities spanning infrastructure to services. He also said that, based on these capabilities, it secured cooperation for a USD 1 billion investment from Nvidia and signed a cooperation MOU worth up to USD 9 billion with Brookfield, a global infrastructure investor.

Naver Cloud is pushing ahead with its strategy to supply large-scale clusters to global AI off-takers, and in response to high security requirements, it is focusing on dedicated cloud services aimed at defense and finance as well as closed on-premises platform services. Based on these supply pillars, Naver Cloud aims to secure a foothold as an AI hub in the Asia-Pacific region.

Source: IT DAILY · Kwon Young-seok
Original: https://www.itdaily.kr/news/articleView.html?idxno=241965

References

• https://www.itdaily.kr/news/articleView.html?idxno=241965