Was watching the Swap GPUs at the Press of a Button: Liqid Fabric PCIe Magic Trick! (youtube.com) video, and was curious what the driver situation was like, but it looks like you install the driver like normal (see the doc below on page 12 for more details).
I found this white paper in liquid’s resource library that seems to explain a lot of the functionality that Wendell is talking about.
Here are some interesting things I found in the doc:
- The interconnect supports ethernet and infiniband, but is by default a fast interconnect, (Reminds me of oculink, but with a link that supports longer runs).
- “SR-IOV or Single Root IO Virtualization is a native feature of the PCIe specification” to support sharing between multiple servers
- They support up to 255 devices, which is the PCIe device limit and they segment devices into pods beyond that.
- P2P supports direct communication between PCIe devices if your program supports it
On a separate note, liquid’s documentation requires you to have an account to even look at which is kind of weird and makes it hard to review technical details, as most of their website is describes their product at a high level.