Breakthrough in CXL 2.0 Memory Pooling/Sharing Technology: Four Servers Share 2TB of Memory, Achieving 210M IOPS and 120GB/s Bandwidth

Introduction: Advancing CXL 2.0 Memory Pooling

Since introducing the CXL protocol, H3 Platform has leveraged its experience in PCIe Switch development to focus on developing CXL Memory Pooling/Sharing systems. Initially, it was only possible for two servers to read and write to a single E3.S CXL memory module. Now, it has completed the sharing and pooling of 2TB of memory, allowing four servers to each access two 256GB (total 512GB) E3.S CXL memory modules. Additionally, we have also completed performance testing, and the results show that through CXL Memory Pooling/Sharing technology, H3 Platform has successfully achieved 66M IOPS (@512 bytes) and 43GB/s bandwidth when a single server accesses memory, while four servers running simultaneously reached 210M IOPS and 120GB/s bandwidth.

Test Environment: Hardware & Setup for CXL 2.0 Memory Sharing

Figure. Diagram of the test environment architecture

The test environment is as follows: The upstream ports (host ports 1 to 4) of the CXL 2.0 Switch are connected to four CXL 2.0-compatible servers, including two Intel Emerald Rapids (EMR) and two Intel Granite Rapids (GNR). Each server is equipped with 64GB of DDR memory. The downstream ports are connected to eight 256GB E3.S CXL memory modules, with each server bound to two memory modules. Thus, 512GB of memory per server is allocated for the read test.

Test Method: Measuring CXL Memory Performance

To evaluate system performance and measure the memory access throughput of each server, H3 Platform kept all other variables constant and only increased the number of servers to observe the transmission changes of the CXL 2.0 Switch. This test focused on two key metrics:

• IOPS (Input/Output per Second): Measures the number of read and write operations the CXL 2.0 Switch can process per second.
• Bandwidth: Measures the total amount of data the CXL memory system can transmit per second.

We conducted tests in scenarios where only one server was connected for read/write operations and where all four servers were running simultaneously, carefully recording the test data, as shown in the table below.

Test Results: 210M IOPS & 120GB/s Bandwidth with CXL 2.0

The results show that when the CXL 2.0 Switch is running with only one server, it achieves 66M IOPS, and when four servers are running simultaneously, the total IOPS reaches 210M (210 million), demonstrating the CXL 2.0 Switch's strong capability in handling data access requests. Additionally, the data transfer rate per server can reach 43GB/s, and with four servers, the total bandwidth reaches 120GB/s, delivering excellent performance.

Results Summary: Impact on HPC & Big Data

The test results verify the feasibility and high performance of CXL 2.0 Memory Pooling/Sharing technology. H3 Platform has successfully enabled four servers to share 2TB of CXL memory, achieving 210M IOPS and 120GB/s bandwidth, demonstrating the efficiency of CXL 2.0 in data access. The maturity of this technology will provide powerful computing support for applications such as In-Memory Databases, HPC (High-Performance Computing), and Big Data Processing. H3 Platform will continue to advance CXL technology, promote the adoption of memory pooling and sharing solutions, and accelerate innovation and evolution in data center infrastructure.