From Message Passing to Distributed Shared Memory: New Possibilities with CXL

With the development nearing its limits, where enhancing computer performance by adding more transistors on a single processor to increase speed and complexity becomes challenging, the traditional vertical scaling model is no longer viable. Instead, horizontal scaling has become the predominant approach in system architecture. In the past, due to the high remote communication costs associated with the distributed shared memory model, horizontal scaling has consistently adopted the traditional message-passing model. However, with the advent of the CXL era, the high cost of remote communication is no longer a barrier when adopting the distributed shared memory model.

Here are several advantages of CXL: 

1.      High-Speed and Coherent Data Transfer Enabled by CXL: 

CXL connects host processors with remote memory resources, enabling high-speed load/store instructions with a remote CXL memory access latency of approximately 300-500 nanoseconds, facilitating high-speed and coherent data exchange between different nodes.

2.      Memory Pooling Optimizes Memory Resource Utilization: 

Memory pooling, as a significant advancement, enables the creation of a global memory resource pool that optimizes overall memory utilization. The CXL switch and memory controller realize the dynamic allocation and release of memory resources, thereby minimizing resource inefficiencies and the accompanying expenses typically found in conventional architectures.

3.      CXL Cache Coherence Enables Memory Sharing: 

CXL 3.0/3.1 ensures memory sharing, enabling the concurrent mapping of identical memory blocks across multiple machines. Through hardware mechanisms that maintain cache coherence while multiple machines access memory simultaneously, a distributed shared memory model is established at the hardware level.

With the emergence of CXL, the high cost of remote communication in distributed shared memory (DSM) is no longer a hindrance. Thus, it's possible to compare it with the traditional message passing (MP) model to determine which is more suitable for horizontal scaling computing architectures. However, this decision depends on specific application requirements and system architectures, in need of a comprehensive consideration of various factors and making choices based on actual circumstances. In the current cloud network environment, where massive data generation makes efficient computing the mainstream requirement, comparing and analyzing the advantages and disadvantages of each model is essential.

Message Passing (MP):

• Advantages:

1.     MP simplifies communication between distributed nodes through a traditional Send/Receive interface.

2.     It assumes a tightly coupled architecture where each node can only access its local memory.

3.     This paradigm is intuitive and familiar for many programmers, resembling multi-threaded programming on a single machine.

• Disadvantages:

1.     MP typically involves copying data payloads between nodes, which can incur overhead.

2.     It requires explicit data communication, which can lead to complexities in programming and potential inefficiencies.

3.     MP may not fully leverage advancements in interconnect technologies like CXL, which prioritize fine-grained remote load/store operations.

Distributed Shared Memory (DSM):

• Advantages:

1.     DSM provides a unified memory space, abstracting away the complexities of explicit data communication.

2.     It facilitates access to remote memory resources through a pass-by-reference approach, enabling efficient data sharing.

3.     DSM supports a shared-everything data/state architecture, beneficial for quickly migrating workloads and load balancing.

• Disadvantages:

1.     Managing shared states in DSM introduces complexities, especially in scenarios involving concurrent access and partial failures.

2.     Traditional DSM approaches may not be robust against system failures, requiring sophisticated mechanisms for memory management and resilience.

In the context of the emerging CXL era, DSM presents renewed potential, especially with advancements in networking and interconnect technologies. CXL's support for fine-grained remote memory access aligns well with the pass-by-reference nature of DSM, offering opportunities for more flexible and efficient resource utilization. However, transitioning to CXL-based DSM requires addressing challenges related to managing shared states and ensuring resilience in the face of partial failures. Despite these challenges, CXL-based DSM holds promise for applications requiring high flexibility and scalability, particularly in the evolving landscape of cloud computing.

Reference: https://www.sigops.org/2024/revisiting-distributed-memory-in-the-cxl-era/