PCIe Gen 5 NVMe MR-IOV Solution

NVMe MR-IOV allows multiple root ports to share an SR-IOV NVMe device. It significantly improves the utilization and efficiency of SSD with multi-root capability while securing the SSD I/O performance in a virtual environment like SR-IOV. Multiple hosts can simultaneously use the Gen 5 NVMe namespaces of the NVMe SSDs in the NVMe chassis via virtual functions with PCIe fabric. This Gen 5 NVMe MR-IOV system comprises 24 E3.S NVMe SSDs and provides up to 12 host connections via external PCIe cables. The SSDs are disaggregated from the host machines and installed in the chassis; therefore, the physical functions of these SSDs reside in the chassis rather than joining with the hosts.

In the same way, the SR-IOV functionality gets enabled within the system. Each SSD derives 32 virtual functions that are pre-assigned to the connected hosts. Users can later allocate NVM namespaces to any connected host machine by creating required namespaces and attaching them to the virtual functions assigned to the host.

1. Gen 5 NVMe chassis withholds all NVMe SSD resources and enables NVMe SR-IOV.

The Gen 5 NVMe chassis involves 24 E3.S NVMe SSDs. The physical function of these SSDs belongs to the NVMe MR-IOV solution system, and the system then creates 32 virtual functions under each SSD. These virtual functions can combine with namespaces created from the actual NVM resource of the SSDs.

2. NVMe virtual functions are pre-assigned to the host machines.

The virtual functions are pre-assigned to the host machines. These virtual functions would reside on the root complexes of the host machines and create IO paths for the hosts to access the NVM namespaces of the SSDs when the namespaces are attached.

Virtual functions derived from different SSDs are assigned to the host so that the host can later receive NVM resources in the form of namespaces from any of these SSDs.

3. Dynamically allocate NVM resources by attaching/detaching namespaces to the virtual functions.

Users can select any SSD to create a namespace(s). The property of the namespace is user-defined, including its NVM capacity and sharing capability (NMIC), etc., and which specific virtual function(s) to attach the namespace. Once the NVMe multi-path I/O and sharing capability (NMIC) are enabled, the namespace could be attached to multiple VFs. It means that different host machines can access the namespace at the same time.

In the above image, the virtual functions one and two derived from SSD one are pre-assigned to the two host machines and namespaces one and two are attached to virtual functions one and two, respectively. As a result, namespace one got mapped to the host with VF one and namespace two to the host with VF two.

4. Managing NVMe namespaces directly from the Gen 5 NVMe chassis.

The management port of the Gen 5 NVMe chassis is standard ethernet, so users can connect it directly to a management device (e.g., a PC) or a network router/switch for remote access. The graphical user interface of the Gen 5 NVMe chassis is web-based and can operate with any standard web browser. (Google Chrome, Firefox, Edge, etc.)

As the Gen 5 NVMe MR-IOV system takes charge of physical functions, users can configure the SSDs in the Gen 5 NVMe chassis through the independent management device. Users can allocate NVM resources to host machines by creating and attaching namespaces using the graphical user interface. And since all SSD configurations undergo within the Gen 5 NVMe chassis, it would not affect any ongoing tasks on the host machines.

SR-IOV and MR-IOV

SR-IOV is a specification that allows one PCIe device to appear to be multiple separated PCIe devices. It permits multi-tenancy (usually virtual guests) of an SSD to significantly improve storage utilization. SR-IOV consists of three key concepts: physical function, virtual function, and the namespace.

Multi-root share of SR-IOV (MR-IOV) leverages some sort of switching device, allowing an SR-IOV device to be shared to multiple root ports. The root ports may be associated with the same or different root complexes, meaning that devices could be shared among various physical host machines.

Physical Function, Virtual Function, and Namespace

Physical Functions (PFs)

Physical function refers to a whole PCIe device with SR-IOV capabilities. PFs are discovered, managed, and configured as normal PCIe devices. Physical functions configure and manage the SR-IOV functionalities by assigning virtual functions and namespaces.

Virtual Functions (VFs)

Virtual functions are lightweight PCIe functions that only process I/O. Derived from physical functions, the number of virtual functions of a device is defined by the device's hardware. A single SSD physical function may map to many virtual functions that can be shared with virtual guests. Each virtual function appears as a controller in the subsystem.

Namespace (NS)

Namespaces are isolation of non-volatile memory (NVM) logical blocks addressable by the host software. Namespaces are attached to virtual functions (a controller) to become accessible by the host. Each namespace is seen as a separate target device from the host. Multi-path I/O and sharing capability (NMIC) of a namespace define whether a namespace is private or shared. A private namespace can be attached to only one virtual function at a time, whereas a shared namespace can be attached to multiple virtual functions simultaneously.