i am planning on a new Machine having the following features:
5950X or similar
64 GB RAM 3600 or faster
2 x 1TB Seagate FireCuda 520 NVME SSD (encrypted ZFS - RAID 0 for max Performance and Security)
QEMU ( i also want to run Games on a Windows 10 VM with a passthrou RTX2080 TI)
10GBit/s Mellanox fibreoptical (gonna be a bridge so the VM’s are connecting to that bridge )
i heard a lot about the ASRock 570 Taichi and am wondering, if this is gonna work, especially the RAID 0 and the 10GBit/s NIC, are there enough PCIe - Lanes or is there a bottleneck already?
Should i rather consider a Threadripper and a TRX40 Chipset Board ?
Any suggestions ?
tl;dr: your cpu/chipset cannot handle so many cards, switch to threadripper.
Hi,
got myself a threadripper/trx40 and have pretty much the same “layout”, but man… building a server with kvm and passthrough is not for beginners.
Lesson learned:
You need a dedicated host gpu - source: trust me dude
You need a proper usb card - source: trust me dude
Your nvme cards should be connected to the cpu and not the chipset: thermal issues ahoi
10gbe is cool, but most likely not worth the hassle
Look at this “simple” setup:
PCIe4x Host GPU (some nvidia gt710 is okay to post/terminal)
PCIex8/16 Guest GPU
PCIex8 USB 3.0/3.2 2x2 card
PCIex8/16 nvme expander (2 or 4 cards)
PCIex4 10gbe
You have 5 cards consuming 32 lanes, even if you swap the nvme cards to the chipset you are running out of lanes on the 5950x (20 lanes).
Btw: dump that zfs and switch to ext4 on lvm and passthrough each guest a volume.
Con: no “snapshot while running” / workaround via lvm snapshots
Pro: 99% Performance
PS: technically it is possible to connect >5 cards to the 5950x via PCIe Bifurcation/PLX riser, but do you wanna go that route?
Depends on what you are doing, and how you have your VM setup. For some things, yes it is pretty much required, others, it’s just nice.
Enh, it is really nice if you have a NAS that can also handle 10gbps.
Otherwise, yep, not really worth it. 2.5gbps is much easier to fit in somewhere.
So x570 has 24 lanes. 20 go directly to devices, and 4 go to the chipset.
Generally, you then get 16 of those lanes to the first two physical 16x slots. If you only have a card in the first 16 slot, then it gets all 16 lanes, if you have a card in both the first and second 16 slot then they get 8x each.
Then the remaining 4 direct lanes go directly to a M.2 slot.
The 4 lanes going to chipset then get split for devices like the onboard sound, networking, some of the USB ports (some of them come directly from the CPU), any second/third M.2 ports, any sata ports, any PCIe 1x or 4x slots, and any other devices.
PCIe 4.0 at 8x lanes is fine for any current GPUs, so you are ok with two GPUs. You also get 4x lanes direct for one of the m.2 drives, which is fine.
The problem comes in with the second m.2 drive and the 10gbps card. That is where a bottle neck is possible. AFAIK, the chipset is smart enough to dynamically allocate bandwidth pretty well, so if you are only hitting the drive, or are only hitting the network card, or only hitting the USB ports in terms of bandwidth you are going to be fine. The issue is that you could be bottlenecked if you are using more high bandwidth devices from the chipset at once.
Good IOMMU groups. Basically means that each PCIe slot, and anything else you might want to pass through to a VM (USB controllers, M.2 slots, etc), are all in their own IOMMU group. This is something that first depends on what chipset/platform the motherboard is, next on the motherboard model, and sometimes on the UEFI version. This is something to research per motherboard model, just search the model # + IOMMU groups. HEDT platforms tend to have better groups than desktop platforms.
You might want a motherboard that can select the boot GPU. It is just less hassle to pass through a non-boot GPU. This is because the UEFI can initialize stuff and then the VFIO driver does not get a clean GPU to work with. By default, motherboards will generally try to boot video from the first PCIe slot first, so if you have the GPU you are passing through in a lower slot then it is fine, but you would want it selectable if you have your passthrough GPU in the top 16x slot.
It is nice, but not always required for some use cases to passthrough an entire USB controller. If you pass through a controller, that means that hotplug works on the ports connected to the controller, and it is really easy to use USB devices in the VM. The problem is that lots of USB controllers have reset issues, so they do not work properly when passed through. If your motherboard has really good groups, you might be able to pass through an onboard USB controller, although it does have to support resetting properly. Otherwise, that is where getting a PCIe USB add in card comes in. Make sure you research the chipset on the card to make sure it supports resetting properly.
Using a disk image has significant overhead, so passing a ssd to the VM is much faster disk wise.
Nvidia GPUs have code 43, but that can be worked around no problem with current drivers. Many AMD GPUs have reset issues. Fortunately, there have been some breakthroughs made recently on workarounds, so this is liable to change. So Nvidia is probably the better choice currently for passing through to the VM. AMD and Nvidia both work fine for the host.
It is easier if you have different models of GPU for the guest and the host. Different models as in they have different PCI IDs, because the method most people use to select the VFIO driver for the card to be passed through uses the PCI ID, so if your host and passthrough cards have the same PCI ID you have to use separate binding method.