At This Data Hall, Noise Has Been Cut Way Down: Inside a Water-Cooled AI DC Holding Several Thousand B200s

When we opened the door to the data hall at NHN FactoryX Seoul, an AI-only data center in Yeongdeungpo-gu, Seoul, the atmosphere inside was quieter than expected. Several thousand NVIDIA Blackwell B200 GPUs were running here, and the GPUs were carrying out continuous computation.

Even so, there was no high-speed fan noise, the kind typically associated with data centers. It was quiet enough to hold a reporting conversation with the person right next to you without raising your voice.

The reduction in noise is attributed to the adoption of direct liquid cooling (DLC), a method that circulates coolant directly to heat-generating components. With DLC in place, noise in the data hall has gone down.

The interior image of FactoryX Seoul is credited to NHN Cloud. The facility is described as a base for the government's "AI Computing Resource Strengthening Project," intended to support the national AI ecosystem.

NHN FactoryX Seoul is infrastructure under the Ministry of Science and ICT and the National IT Industry Promotion Agency (NIPA)-led "2025 AI Computing Resource Utilization Infrastructure Strengthening Project." The project aims to develop Korea's domestic AI ecosystem and is being carried out with a large budget.

The project supports companies, universities, and research institutes, and provides high-performance GPUs. Its goal is to help foster national AI models and talent. The GPU supply price for companies is about one-tenth of market price. The "proprietary AI foundation model" project, in which major Korean industry, academia, and research players are participating, is also under way, and NHN FactoryX Seoul plays a core role as training infrastructure for that project.

The entire facility is a state-owned asset. NHN Cloud handles government-budget-based procurement of equipment, data center leasing, and infrastructure construction, and will also be solely responsible for operations and management for the next 5 years. However, NHN Cloud does not receive separate government operating funds, and to secure operational efficiency, it is authorized to use 20% of the total deployed capacity for in-house service development and another 20% for enterprise cloud supply.

The center's data hall spans floors 3 through 10, with NHN Cloud's operating scope covering floors 3 through 6, or 4 floors in total. The computing resources are divided into 3 clusters. "NIPA-CL1" is configured as an integrated connection across floors 3 and 4, with 4,080 GPUs and 510 nodes; "NIPA-CL2" is located on floor 5 and operates with 2,040 GPUs and 255 nodes. "NIPA-CL1" and "NIPA-CL2" are used for government-supported operations.

Floor 6 is designated as a cluster for NHN Cloud's own use. The tour on this day also centered on the floor 6 in-house cluster operated independently by NHN Cloud.

The government-supported "NIPA-CL1" ranked 20th worldwide in the TOP500. The ranking is meaningful as an assessment that demonstrates "NIPA-CL1"'s computation performance at a global level.

On site, it was explained that in an air-cooled environment, the noise level would make conversation difficult, and a comparison target for that was also presented. The image was labeled as the FactoryX Seoul control room, and the image source was NHN Cloud.

The key change felt on site was the drop in noise after switching to water cooling. In the data hall's air-cooled network switch (InfiniBand) area, strong fan noise from blowers was heard, and even near ordinary air-cooled servers, the noise was at a level that made it difficult to hear the person next to you. By contrast, when we moved into the water-cooled area densely packed with B200 server racks, the perceived noise dropped noticeably.

That change stemmed from applying water cooling to the GPU chipsets and CPU, where 70% to 80% of the heat is concentrated. The cooling structure uses a copper Cold Plate placed in close contact with those components. Cooling is carried out through direct liquid cooling, which removes heat directly with coolant.

As a result of direct liquid cooling, the operation of the large, high-speed fans that were essential in conventional air-cooled servers was minimized. That is why the fan noise from network switches and servers, which was loud enough to make conversation difficult in the air-cooled area, dropped sharply in the water-cooled B200 server zone.

This difference is also linked to the fact that data center noise is one of the factors behind resident opposition when data centers are built near downtown areas or residential neighborhoods. While extreme noise from cooling equipment and exhaust fans is cited as the background for complaints, water-cooling technology has been presented as a promising alternative for improving the social acceptance of data centers and addressing noise complaints.

Air cooling and water cooling differ in terms of cooling performance and energy efficiency. Air-cooled servers tend to see GPU temperatures rise above 70 degrees Celsius, and in air-cooled environments, Throttling is likely to occur to prevent overheating. By contrast, in a water-cooled environment, GPU temperatures are stably maintained at around 50 degrees Celsius, making it possible to secure the chipset's maximum computational efficiency. Power consumption per server was also compared at 14.3 kW for air cooling and 12.3 kW for water cooling, and water cooling was presented as reducing power consumption by about 14% compared with air cooling.

Lee Il-jun, an executive director at NHN Cloud, said immersion cooling could potentially be more efficient. He added, however, that major hardware makers such as NVIDIA, HPE, and Dell do not provide official maintenance and support (AS) for immersion equipment. Accordingly, he said direct liquid cooling, which guarantees technical support, is currently the most stable and optimized method for long-term stable operation of enterprise AI data centers.

The photo shows Lee Il-jun, an executive director at NHN Cloud, giving the tour explanation, and the photo is credited to NHN Cloud.

NHN FactoryX Seoul applied redundant and specialized designs across its electrical and mechanical systems to support a high-density B200 cluster. Its power infrastructure is linked to two external Korea Electric Power Corp. substations, and the power receiving system is configured as 2N.

Power supplied from the external substations is 22,900V extra-high voltage, which is stepped down to 10,000V at the main transformer (MTR) in the substation room and then converted through each floor's transformer (TR) into 380V/230V, the operating voltage for servers.

A total of 10 uninterruptible power supply (UPS) units were installed, and they handle voltage and frequency stabilization. In an emergency, batteries provide about 10 minutes of uninterrupted power until generators start up.

In addition, in response to revised safety standards after the Kakao data center fire incident, a special fire wall was applied to the lithium battery room. The lithium battery room was physically separated from the electrical room and computing areas to strengthen the fire response structure.

To withstand the load of ultra-heavy GPU racks and water-cooling piping, the facility strengthened its floor design load to 2.0 tons per square meter. This was intended to support heavy water-cooling pipes and to bear ultra-heavy GPU racks weighing 1.2 to 2.3 tons each. Because water-filled cooling pipes are structurally difficult to suspend from the ceiling, a steel floor support frame was installed as a countermeasure.

This frame plays the role of stably supporting the load of the piping. The water-cooling supply system operates in a 3-stage circulation structure. In stage 1, heat is released to the outside from a cooling tower on the roof; in stage 2, chilled water is produced through a subterranean turbo chiller; and in stage 3, the chilled water is delivered to the data hall cooling distribution unit (CDU), after which a thin piping loop inside the server directly cools the GPU chipset.

The control room monitors supply water temperature, water pressure, flow, and microleak sensors in real time 24 hours a day. The purpose is to preemptively block the risk of equipment damage.

Lee Il-jun, an executive director at NHN Cloud, explained that data centers designed under the conventional model have structural limitations in that part of the rack space must be left empty because of power and cooling constraints. He added that future AI data centers will need to consider cooling technology, floor load capacity for water-cooling piping and ultra-heavy racks, and large-scale power capacity from the initial design stage.

Source: IT DAILY · Kwon Young-seok
Original: https://www.itdaily.kr/news/articleView.html?idxno=241066

References

• https://www.itdaily.kr/news/articleView.html?idxno=241066