NVIDIA GPUDirect Storage (GDS) and NVIDIA SCADA both provide direct data movement between local NVMe storage and GPU memory. Their key differences are as below.
| GPUDirect Storage (GDS) | NVIDIA SCADA | |
|---|---|---|
| I/O Initiator | Host CPU | Thousands of GPU threads |
| I/O Granularity | 4 KB | 512 B |
| Number of NVMe SSDs | Typically 1–2 | Typically 4–8 or more |
| IOPS per SSD | ~6M IOPS | ~10M–100M IOPS |
| Primary Use Case | GPU Storage Access | GPU Memory Extension (Second-Tier Memory) |
This architectural difference becomes particularly important in large-scale graph neural network (GNN) workloads. As feature datasets grow beyond local GPU HBM capacity, GNN feature gathering can generate large volumes of small (512 B), irregular storage requests. Efficiently serving these requests requires an I/O architecture capable of supporting high concurrency and substantial request throughput. [S5]
This article will demonstrate why SCADA is the stronger architectural fit for storage-bound GNN feature gathering. Its GPU-initiated I/O model supports small, highly parallel storage requests, with 230M aggregate IOPS at 512 B random reads in a SCADA-based system. Combined with NVMe SSDs as second-tier memory, this approach addresses the capacity and data-access demands of large-scale graph workloads.
NVIDIA GPUDirect Storage (GDS) enables direct memory access (DMA) between supported storage and GPU memory with a CPU-initiated pattern. In a supported local NVMe configuration, the SSD transfers data directly over PCIe into GPU memory at 4K data size. [S8]
GDS Payload Path: NVMe SSD → Direct DMA over PCIe → GPU Memory
GDS Command Path: CPU / Host → cuFile API → Host I/O Stack → NVMe SSD
The CPU initiates storage requests through the cuFile API, with host-side software handling I/O submission and completion. The command path remains host-issued. [S2]
Figure 1. GDS combines host-issued I/O with direct DMA between NVMe storage and GPU memory.
Challenge 1 — Small, Irregular I/O Demands
Large-scale GNN feature gathering requires small (512 B to 4 KB), irregular reads. However, GDS favors coarse-grained (>4KB) streaming-oriented transfers. Micron found that 99% of storage I/O consisted of small-block reads in its BaM/GIDS GNN training workload. [S1][S5][S10]
Challenge 2 — CPU-Side I/O Scalability
CPU-side data preparation and I/O processing can become bottlenecks as request rates increase. The GIDS study measured 4.1M feature requests/s for CPU-based preparation versus 29M requests/s consumed by GPU training kernels. Separately, the GDS configuration used in this comparison delivered below 50M IOPS. [S2][S5]
Challenge 3 — HBM Capacity and Cost
Memory capacity adds another challenge. The GIDS study reports large-scale GNN datasets of 1,084 GB (IGB-Full) and 2,773 GB (IGBH-Full), both exceeding the local HBM capacity of a single GPU. You can expand available HBM capacity by deploying additional GPUs, but this will increase costs exponentially. [S5]
Together, these challenges highlight the need for high-capacity, cost-efficient memory infrastructure and scalable small-read I/O. NVIDIA SCADA offers a potential solution by enabling GPU-initiated storage access to large datasets stored on NVMe SSDs as second-tier memory. The following section examines how SCADA addresses these requirements. [S3][S9]
NVIDIA SCADA (SCaled Accelerated Data Access) is a software framework that enables the GPU to initiate highly parallel, fine-grained (512 B) storage I/O requests directly. It allows thousands of GPU threads to issue small requests in parallel. In other words, the command initiator changes from the CPU to GPU. The data moves directly between NVMe SSDs and GPU memory through PCIe DMA. [S3][S9]
SCADA Payload Path: NVMe SSD → Direct DMA over PCIe → GPU Memory
SCADA Command Path: GPU Threads → SCADA GPU-Side I/O Interface → NVMe SSD
Figure 2. Conceptual NVIDIA SCADA architecture showing GPU-initiated storage I/O through the SCADA path to NVMe.
For large-scale GNN feature gathering, SCADA supports fine-grained access to node features distributed across NVMe storage. For example, a 128-dimensional FP32 feature vector occupies just 512 B, illustrating the small access granularity involved. Local HBM retains performance-sensitive working data, while NVMe SSDs provide larger, cost-efficient second-tier memory for datasets beyond GPU memory capacity. [S7][S9]
Micron reported 230M aggregate 512 B random-read IOPS using the SCADA workload on an H3 Falcon 6048. The reported test used 44 Micron 9650 PCIe Gen6 E1.S 7.68 TB SSDs. [S6]
This result highlights SCADA's potential to support the highly parallel, fine-grained storage access demands of large-scale GNNs. By shifting I/O initiation from the CPU to GPU execution, SCADA offers a stronger architectural fit for storage-bound GNN feature gathering than GDS's host-issued approach. The following section examines these differences in detail. [S3][S5][S6]
The comparison below focuses on GNN feature gathering beyond local HBM, emphasizing request size, request-serving capability and the GPU-driven access model.
| Comparison | GPUDirect Storage | NVIDIA SCADA |
|---|---|---|
| I/O size | Coarse-grained, >4 KB in the scenario compared here. [S1] | Fine-grained, 512 B in the measured workload. [S6] |
| IOPS comparison | <50M IOPS, initiated by CPU | >200M IOPS, initiated by GPU parallel capability: 230M aggregate 512 B random-read IOPS in the reported SOL test. [S6] |
| I/O initiator / control path | CPU through cuFile APIs. [S2] | GPU threads through the SCADA I/O interface [S3] |
| I/O Parallelism | Host-issued requests with batching and asynchronous I/O support [S1][S2] | Thousands of GPU threads initiating fine-grained requests concurrently [S3] |
| Payload path | Direct PCIe DMA between local NVMe and GPU memory. [S8] | Direct PCIe DMA between local NVMe and GPU memory. [S4] |
| NVMe Access Model | Host-managed access to NVMe data | GPU-initiated access to NVMe-backed second-tier memory [S3][S9] |
For large-scale, storage-bound GNNs, NVIDIA SCADA offers a stronger architectural fit through GPU-initiated, fine-grained storage access and highly parallel request handling. These advantages become particularly important as feature datasets exceed local HBM capacity, making I/O granularity, storage performance, and second-tier memory capacity critical system design considerations. [S3][S5][S9]
Nowadays, large GNN feature datasets can grow to tens of terabytes. For example, a graph with 5 billion nodes and 1,024 FP32 features per node requires approximately 20 TB of node feature data alone. Efficiently accessing scattered features at this scale requires both high-capacity storage and highly parallel, fine-grained I/O.
As feature datasets exceed local GPU HBM capacity, NVMe SSDs provide a larger, cost-efficient second-tier memory while HBM retains performance-sensitive working data. NVIDIA SCADA enables GPU-initiated access to the required features stored on NVMe, supporting feature gathering across datasets that extend beyond local GPU memory. [S3][S9]
Large-scale GNN feature gathering can generate millions of small, irregular data requests. For a workload accessing 512 B feature rows, delivering data at high throughput requires substantial storage IOPS.
Consider an illustrative target of 100 GB/s of useful feature data delivered from SSDs:
Required IOPS = 100 GB/s ÷ 512 B ≈ 195.3M IOPS
This calculation assumes one 512 B storage read per required feature, without cache hits or request merging. Actual demand depends on the GNN workload and its memory access patterns.
Micron's reported 230M aggregate IOPS at 512 B random reads using H3’s Falcon 6048 demonstrates that SCADA-based infrastructure can achieve this level of fine-grained storage request throughput under the published test configuration. [S6]
By combining GPU-initiated I/O with high-IOPS NVMe SSDs, SCADA offers a scalable architectural approach to supplying the large volumes of small, scattered feature data required by storage-bound GNN workloads.
For large-scale, storage-bound GNNs, NVIDIA SCADA's GPU-initiated, fine-grained I/O model is a game-changer in storage architecture. By moving storage request initiation into GPU execution, SCADA enables highly parallel access to small, scattered features across datasets extending beyond local HBM.
Realizing SCADA's full potential requires careful system-level design. Architects must consider HBM working-set capacity, NVMe-backed second-tier memory, I/O granularity, request concurrency, and high-IOPS storage performance together. A well-balanced infrastructure is essential to sustaining efficient feature delivery as GNN datasets continue to grow in size and complexity.
H3 Platform's Falcon 6048 provides an ideal platform for putting this architecture into practice. As a converged GPU-accelerated memory/storage server featuring PCIe Gen6 connectivity and high-density NVMe storage, Falcon 6048 works with NVIDIA® SCADA™ to support GPU-initiated, fine-grained storage access. Its role in Micron's reported 230M IOPS demonstration further highlights its high-IOPS potential, making Falcon 6048 a compelling infrastructure choice for next-generation, data-intensive GNN workloads.
NVIDIA SCADA enables GPU threads to initiate fine-grained storage I/O requests in parallel, eliminating reliance on CPU-side request submission. This GPU-initiated model is particularly well suited to the small (512 B), irregular feature reads common in large-scale GNN workloads.
A node with 128 FP32 features requires 512 B of feature data. In an isolated-read scenario, retrieving 512 B instead of 4 KiB reduces transferred data by 87.5%, assuming no reuse of neighboring data. Fine-grained access can therefore improve data-transfer efficiency for scattered feature reads.
It demonstrates aggregate 512 B random-read capability in the published SOL/SCADA configuration using H3 Platform’s Falcon 6048. It shows the high-IOPS capability of a GPU-driven storage system to support large-scale GNN workloads.
When GNN feature datasets exceed local GPU HBM capacity, NVMe SSDs provide a cost-efficient, high-capacity storage tier for the additional data. With NVIDIA SCADA, GPU threads can initiate highly parallel, fine-grained I/O requests to access the required features from NVMe SSDs, allowing them to serve as storage-backed second-tier memory for supported GPU applications. NVMe SSDs retain different latency, bandwidth, and access characteristics from GPU-local HBM.
