Following the release of PCIe Gen 4 standard in 2018 by PCI-SIG, AMD announced its first PCIe Gen 4 GPU in 2019, and nVidia revealed its A100 GPU this year. Clearly, the industry standard is moving toward PCIe Gen 4 for better system performance and power consumptions. An expansion chassis is one of the easiest ways to incorporate these new and powerful GPUs to your existing system. Despite PCIe Gen 3 or Gen 4, you might have seen many PCIe switch-based expansion chassis on the market, and you might have noticed that some of these chassis are “fan-out” chassis while the others are “fabric” chassis. Both options allow you to add more GPUs to your system, but what exactly is the difference?
To explain it, let us look at the topology of fan-out switch and fabric switch
a. Fan-out topology
The fan-out topology is quite straightforward. The switch divides the PCIe lane into more lanes, thus it can bridge more devices to the CPU. Pros of fan-out switch include better signal integrity and easy to configure. However, the biggest downside of fan-out is that there is only one upstream port, meaning only one host can be connected.
b. Fabric topology
The fabric topology is more complex. The switch here actually plays a mapping role, linking more than one host to your devices. Yes, there are more things to do to configure a system with PCIe fabric switch, but the fabric switch enables powerful application that the fan-out switch would not do.
The capability of PCIe switch chip set, really, is depending on its SoC. The SoC in a fan-out switch chip set is designed to handle simple tasks, acting like a gateway between PCIe devices and the host. The fan-out switch does not do anything special to the data but passing it to the end devices. On the other hand, the SoC in a Fabric switch chip set is designed to handle more complicated tasks.
Single-host vs Multi-host
For single host usage, a fan-out PCIe switch can handle the expansion task more efficiently. Data will pass through less nodes until it reaches its destination, therefore the level of data integrity would not be much affected. In addition, fan-out switch does not require much settings, making the deployment of your expansion system easier.
PCIe fabric switch stands out when you need fan-out for more than one host. They way fabric switch enables multi-host usage is that the SoC emulates an empty PCIe device, and requests for bus number and memory space from the host BIOS or OS. Then, the requested memory space will be partitioned for its downstream PCIe devices. The switch not only receives data from hosts and send it to the relative endpoint/device, but it can also achieve dynamical device provision to hosts or dump them back to the resource pool by altering the memory partitions.
Port bifurcation-Utilizing your bandwidth and resource pool
You will find x16 PCIe downstream port in most servers, and some of them have BIOS that supports the bifurcation of the port, dividing a x16 port into two x8 or four x4 ports, and this allows better utilization of the bandwidth. Afterall, even for high end x16 GPUs, the performance would not be affected much when the bandwidth is reduced to x8 in most cases, it would be inefficient to use up the entire x16 bandwidth.
Recall that there is only one upstream form a fan-out switch, it will take all x16 bandwidth when connected to the host even though it may only need 4 or 8 lanes. The fabric switch though, can bifurcate its x16 upstream port to up to four x4 ports. This allows us to interconnect more systems together without having them affecting each other. Without taking all 16 PCIe lanes of the hosts, the rest of them can be used for other peripherals, leading to better bandwidth utilization.
Complex deployment-keep track of your devices
A fan-out switch is only bridging the devices to the host, you will most likely have to embed extra program to monitor your devices because the switch itself will not read the configurations. On the other hand, the SoC of fabric switch makes it acting as the root complex that can read the configuration space of your PCIe devices, therefore it gets all the basic information and recognize what each PCIe device does.
In H3 Platfrom expansion chassis, the turned-off slots get 3.3V power by default. This allows the BMC to recognize devices and get the information for your need, even when the slots/devices are at rest.
Device error occurrence-Dealing with reset processes
When errors occurred to one of your PCIe devices, a fan-out switch would not allow you to reset the device alone, you would reset all devices in the entire fan-out system. But a PCIe fabric switch will allow you to reset only the device that needs it. As we have discussed, a fabric switch enables memory space partitioning. And because every downstream is assigned to different partition, fabric switch allows you to reset the device with error without affecting other devices by recognizing which partition it is in.
We have gone through the major differences between PCIe fan-out switch and the PCIe fabric switch. Clearly, a fabric switch is capable of more complex things than a fan-out switch could, but the applications of fabric switches are not commonly seen so far. In high-performance computing industry, IT infrastructures are expanding rapidly to cope with the increasing IT needs. Servers are piled up in data warehouses and devices are added with fan-out modules, which really create a lot of wastes and inefficient usage of resources. With PCIe4 introduces to the market, IT manufacturers will be able to fit more computing power onto a smaller device, so we can expect certain drop in the amount of wastes created. In addition to that, PCIe fabric switch technology will allow the IT leaders to utilize their devices better. In stead of piling up servers and storage, it is more important to make every piece of them efficient.