Knowledge Base
For large Graph Neural Network (GNN) and vector database workloads, the memory wall emerges when datasets grow beyond practical local GPU high-bandwidth memory (HBM) capacity while irregular, non-sequential access patterns make the required data harder to serve efficiently from outside local memory.
In GNN inference, large node features and adjacency matrices can create substantial memory communication and buffer overhead because graph processing involves irregular memory access. [S3] Graph-based vector search presents a related challenge: traversing an approximate nearest neighbor (ANN) graph index can involve non-sequential jumps among vectors, resulting in random memory accesses, cache misses, and significant memory-access overhead. [S4]
NVIDIA® SCADA™ addresses the storage-access layer of this problem by providing a GPU-initiated storage model designed for high-IOPS (Input/Output Operations Per Second) access to small blocks of data, enabling GPU workloads to reach NVMe-resident data beyond local memory capacity. [S2] HBM remains the GPU’s high-speed local memory, while high-IOPS NVMe provides access to a much larger data pool when the workload extends beyond HBM capacity.
GNNs and graph-based vector search encounter the memory wall through a combination of dataset scale and workload-specific memory-access behavior. As graph structures, node features, vector collections, and indexes grow, the working data may exceed what can practically remain in local GPU memory. At the same time, these workloads can require irregular or non-sequential access to the data they need, making memory access itself an important part of the system's challenge.
For GNNs, the memory wall comes from large datasets and scattered memory access.
In a GNN, each node uses information from its neighboring nodes to update its own representation. This process, known as message passing, requires the system to first identify a node’s neighbors and then retrieve the feature data associated with those neighbors.
During message passing, graph neighbors are logically connected, but their feature data is not necessarily stored next to each other in memory. As a result, GNN processing can require the system to gather data from scattered memory locations rather than reading one large, continuous block. BingoGCN, a 2025 ISCA study of large-scale GNN inference, identifies large node features, adjacency data, and irregular memory access as important sources of memory communication and buffer overhead. [S3]
Neighbor sampling can reduce how many neighboring nodes are processed at one time, but it does not remove the irregular structure of the graph. The GPU may still need to fetch feature data from non-contiguous locations. When the graph and its feature data become too large to fit entirely in local GPU HBM, this combination of limited memory capacity and irregular data access becomes part of the GNN memory wall.
At that point, adding more capacity elsewhere in the system is not enough by itself. The architecture also needs an efficient data path that allows GPU computation to reach the required data outside local HBM.
For graph-based vector search, the memory wall comes from large embedding datasets and random memory access.
Graph-based approximate nearest neighbor search (ANNS) is a common way to search large vector collections. Instead of checking every vector one by one, the search follows a graph index and moves between likely candidate vectors.
The challenge is that these moves are not always sequential. Research on graph-based ANNS shows that the search can jump between vectors stored in different memory locations, creating random memory accesses. These accesses can lead to frequent cache misses and higher memory-access overhead. [S4]
As embedding collections and graph indexes grow, the full working set may no longer fit in local high-speed memory. At that point, the system faces both a capacity limit and a random-access problem. Together, these two constraints form the memory wall for graph-based vector search.
The architecture therefore needs more than additional capacity. It also needs an efficient way for GPU computation to reach the required vector and index data when that data resides outside local memory.
HBM provides the high-bandwidth, low-latency local memory GPUs require for performance-critical workloads. However, scaling HBM or system DDR memory capacity can be prohibitively expensive. On a cost-per-byte basis, HBM and DDR memory may cost approximately 100× and 10× more than SSD storage, respectively. As graph datasets, feature tables, embedding collections, and vector indexes continue to grow, keeping them entirely in memory becomes increasingly costly from a total cost of ownership (TCO) perspective.
NVIDIA SCADA provides another path: keep performance-sensitive working data in HBM while using GPU-initiated I/O to reach data stored on high-IOPS NVMe when needed. [S2] This allows the accessible data footprint to scale beyond local HBM without requiring the entire dataset to remain in GPU memory.
When GPU-initiated requests can be distributed across multiple NVMe devices through the PCIe fabric, the system can draw on parallel storage resources while taking advantage of the substantially lower cost per byte of SSD capacity. This architecture can reduce dependence on expensive memory-capacity expansion and improve the balance among performance, capacity, and infrastructure cost, depending on workload behavior, storage configuration, and available I/O parallelism.
Micron characterizes NVIDIA SCADA as a programming model and technology stack for GPU-initiated, high-IOPS storage access targeting small-block workloads. [S2] For data-intensive GNN and vector-search workloads, this creates a direct architectural link between GPU computation and a larger NVMe-backed data pool.
NVIDIA SCADA—Scaled Accelerated Data Access—is a secure programming model and technology stack designed for GPU-initiated, high-IOPS storage access for small-block workloads. [S2]
Within the supported SCADA storage I/O path, storage operations can originate from GPU computation. This creates a mechanism for GPU workloads to access NVMe-resident data when the required information is outside local GPU memory.
For the workloads discussed here, that mechanism addresses an important infrastructure layer of the memory wall.
For GNN and graph-based vector-search workloads, SCADA provides a GPU-initiated path for issuing fine-grained storage requests when graph features, embeddings, or index data reside on NVMe. Rather than treating storage only as a bulk staging area, the supported SCADA I/O path allows GPU computation to request the data it needs as the workload progresses. [S2]
This gives data-intensive applications access to a much larger storage-backed data footprint while preserving HBM for performance-sensitive local working data. In supported configurations, SCADA can combine GPU-driven I/O with multiple NVMe devices and high-speed PCIe connectivity to increase the storage resources available to concurrent GPU requests.
Micron explicitly identifies graph neural networks and vector databases among workloads relevant to GPU-initiated storage operations using NVIDIA SCADA together with next-generation interconnects and high-performance SSDs. [S2]
The architectural benefit is a broader memory-and-storage hierarchy: HBM remains close to GPU compute for the hottest working data, while SCADA extends GPU-driven access into a much larger NVMe-backed data pool.
This approach can reduce the amount of data that must remain resident in expensive HBM and improve infrastructure cost efficiency, while actual performance and TCO benefits remain workload- and configuration-dependent.
The CPU continues to handle platform management and control responsibilities, while supported storage I/O can originate from GPU computation within the SCADA path.
Falcon 6048 provides the physical infrastructure for building the GPU-to-storage data path used with NVIDIA® SCADA™. It brings GPU resources, high-density NVMe storage, and PCIe Gen6 switching together in a converged system architecture, providing the connectivity and physical I/O resources required for GPU-initiated access to NVMe.
In the SCADA implementation described by Micron, NVIDIA GPUs orchestrated storage access through an H3 Platform Falcon 6048 PCIe Gen6 server. The system incorporated Micron 9650 PCIe Gen6 NVMe SSDs and Broadcom PEX90000 PCIe Gen6 switches. [S2]
The PCIe Gen6 switch fabric provides connectivity between accelerators and NVMe storage within the system. Micron describes Broadcom PEX90000 switches integrated into Falcon 6048 as connecting GPUs and NVMe devices through a unified PCIe Gen6 fabric. [S2]
This topology matters because GPU-initiated storage access ultimately depends on physical I/O resources. Multiple NVMe SSDs distribute storage across multiple devices, each contributing its own controller, PCIe connection, capacity, bandwidth, and I/O resources that can be accessed through the PCIe fabric.
Within this architecture, each layer serves a distinct role:
Together, these components create a scalable GPU-to-storage data path for accessing a much larger data pool beyond local HBM.
As GNN and graph-based vector-search datasets continue to scale, the architectural challenge is to preserve HBM for high-speed local GPU access while providing a practical path to data that resides beyond its capacity. NVIDIA SCADA addresses the storage-access layer of this problem by enabling GPU-initiated access to NVMe within the supported SCADA I/O path, while Falcon 6048 brings together the GPU, high-density NVMe storage, and PCIe Gen6 connectivity required to implement that data path.
The memory wall emerges when large GNN and graph-based vector-search datasets exceed practical local GPU HBM capacity while irregular or non-sequential access patterns make the required data harder to serve efficiently from outside local memory. [S3][S4]
GNNs can exhibit irregular memory access because message passing requires nodes to gather feature data from their graph neighbors. Since graph connectivity is irregular, these accesses do not necessarily follow contiguous or sequential memory locations. [S3]
Graph-based vector search traverses an ANN graph index to identify candidate vectors, which can involve non-sequential jumps among vectors. These accesses can result in random memory-access patterns, cache misses, and additional memory-access overhead. [S4]
NVIDIA® SCADA™ provides GPU-initiated, high-IOPS storage access for small-block workloads, allowing GPU workloads to access NVMe-resident data beyond local HBM capacity. This enables HBM to serve performance-sensitive working data while NVMe provides access to a much larger data pool. [S2]
Falcon 6048 provides the physical infrastructure for building the GPU-to-storage data path used with NVIDIA® SCADA™. It integrates GPU resources, high-density NVMe storage, and PCIe Gen6 connectivity in a converged system architecture. [S2]
