VFIO Build Advice - MoBo Recommendation?

Hardware Hub Build a PC
, ,
1

I heard Wendel’s call to action on VFIO and I immediately jumped into an idea for a new build /s

But (as you can see from my profile), my current build is… a little long in the tooth. Probably time for an upgrade, and to see if one of those SSD things is really as good as people say.

My thought on this build is that it’s going to be “everyday virtualization” - the host OS is going to be the KDE Plasma Desktop version of NixOS (I’ve heard good things, time to try this out), and then boot Windows or Fedora in a guest OS in KVM for applications like Firefox, VS Code, and gaming. Do I want to pass those applications back through to the host OS via some kind of X forwarding? Maybe, but we can cross that bridge when we get VFIO working.

My current build idea is here: Part List - AMD Ryzen 7 7800X3D, Radeon RX 7800 XT - PCPartPicker - no motherboard currently in it. Since the build is… not currently aggressively budget constrained, I’ll say try to keep it below $200US, but the biggest thing is API support. I would prefer to have as turnkey a system as possible, but I understand that there are no guarantees in VFIO. I’d prefer to use a B650 or B650E chipset, but I’m willing to step up to the 670 or down to the 620 chipset if necessary.

(Build TL;DR: 7800X3D, Sapphire 7800 XT, 96GB DDR5-6400, 2TB Crucial P3 Plus SSD, Probably one of several spinning rust hard drives as a backup drive)

If it matters, I’m few feet from my wifi router and my current build uses an ethernet cable, so on-board wifi is not critical here.

IMO, the entire build is fluid at this point. No money has been paid, no parts bought, and my current build is still… limping. But this seems like some cool tech, and I want to try it out.

I’ve been out of the PC building game for a few years, so I’m just kinda throwing this together. If you see something that I’m doing that’s dumb, PLEASE tell me.

Let me know what you think of the build and feel free to advise in any way possible. I want to do this right, whatever that looks like. I appreciate the help in advance!

EDIT: I have an older GPU in my current build that might be able to be repurposed for the new build. Not REQUIRED, but optional.

2 Likes
2

I started a similar thread, but very few answers in months. I use vfio on Intel + RX6000 + nVidia with no problems, but I want to upgrade to AM5 and cannot find positive feedbacks for motherboard choice.
As I know (for now) it’s better to avoid B650 and RX7000 cards, x670 should be better for iommu groups (in particular for dual GPU passthrough), and many rx7000 cards have reset bug, so unfortunately it’s better to go on a 6800xt/6900xt or any nVidia.
7800X3D is fine if you are aware that you can’t pass all cores/threads to the vm, so it will be a 6c/12t or 8c/8t…

3 Likes
3

There have been a lot of discussions on the forum already:

I have not seen any evidence of B650(E) being bad for vfio however (even the opposite). If you are only looking to pass a single gpu, all B650/x670 boards seem ok. If you want to pass usb controllers, nics, etc. it becomes more complicated.

I do agree that the Radeon 7000 cards seem to be a mixed bag, so Nvidia may be a safer choice (my 3060 has been good), though the drivers in Linux can be a pin in the ass (there’s still a lot of quirks on wayland)

4 Likes
4

I think B650 is ok for a single GPU passthrough, but for dual GPU? At least should be a B650 with more than one slot connected to CPU lanes…

1 Like
5

My current setup is remarkably similar to what you want. I have a 7950X on an X670E ProArt, with a 6900XT in the first PCIe slot and two M.2 SSDs in the second, plus another four M.2 SSDs in the slots on the motherboard itself. My monitor is plugged into both the iGPU and the dGPU, but I normally set it to display the iGPU. My desktop is KDE on NixOS. I’ve created a .desktop file that starts a Windows VM with my dGPU passed through, and then launches looking-glass as a borderless maximized application. I use the Windows VM for some work software and games that aren’t compatible with Windows, and thanks to looking-glass I can seamlessly switch between Linux and Windows. For example right now I’m typing into a browser window on the Linux host, while the client is displaying Starfield in the background. For FPS like Starfield I’ve got a USB bluetooth dongle passed through, and the second input on my mouse paired to that, so I can let it “capture” the cursor, but strategy games like Stellaris or productivity software like Excel I simply use with the mouse in Linux through looking-glass.

The ProArt is an ideal motherboard for VFIO and light workstation, if you’re willing to spend a little extra. It readily boots with four DIMMs installed (it’ll even overclock them a little, but the newer BIOSes have made this much trickier), it supports ECC memory, and in my experience each PCIe device is assigned its own IOMMU group. This makes it easy to do things like pass through an SSD, a graphics card, and a USB card, which is basically the perfect setup for a guest, giving you quick storage, GPU acceleration (ideally with looking-glass so you won’t have to switch inputs on your monitor all the time), and lets you plug USB peripherals into the guest without needing to open virt-manager to explicitly hand each one over. It’s a bit pricey for a motherboard, but you also get 10GbE and a lot of USB ports, the thing would be ideal for something like a music studio. I definitely think it’s worth it.

2 Likes
6

There’s only one B650 board with dual 8x pcie 4.0 slots, and that’s the proart B650. It has great IOMMU groups (I think everything isolated – better than my x670 proart) and seems like a good board. There are some users here, if you search the forum you’ll find their experiences.

2 Likes
7

Hey, I wanted to thank everyone in this thread. Y’all have been great, and a really welcoming community. When I get this thing ordered and built, I’ll come back and let you know how it’s going.

Thanks!

3 Likes
8

The latest bios seems to be broken a ton of things though (1710, Combo AM5 PI 1.0.8.0) :-/

1 Like
9

What’s broken for you? I’m having to retune my overclocks, as expected from a new AMD firmware, but so far it seems the difference is just a slight improvement in voltages.

1 Like
10

Mainly resource allocation broken of builtin devices, Linux even struggled to boot while FreeBSD did with some devices non working like USB and SATA.

1 Like
11

Oh, that’s unfortunate. I haven’t had that issue, everything USB seems to be working for me and right now I don’t have any SATA devices available to plug in and check that, my HDD set are in my NAS and my SSD set are in my second home. I did notice that it fails to detect SSDs on a bifurcation card in slot 2 if I unplug the GPU from slot 1 (which I do whenever I’m testing overclocks), but resetting by switching the PSU on and off solves that. That’s been a recurring issue though, I’ve seen it with previous BIOSes and on different zen4 motherboards. It’s possible you may be having this issue? Try powering off with the PSU switch between reboots and see if things start working again. It’s an annoyance, but if it works at least there’s a fix.

2 Likes
12

I have the X670 proart on 1710 bios and no issues, same as Susanna. Which board do you have @diizzy ?

I also had weird bluetooth problems at some point (some months and BIOSes ago), which were solved by switching off the PSU for a few seconds. AMD/Asus can be a bit flakey on this point (USB too…)

1 Like
13

ProArt X670E-CREATOR WIFI, this resource issue is also seen on at least some ASRock boards to some degree.

1 Like
14

This is very similar to the setup I have. NixOS for the hypervisor and AMD CPU with a GPU passed through for gaming. Only 2 differences. One is I use a nvidia for the passthrough. And 2 is I use 2 GPUs because I have a build with a ton of PCI lanes. If you are running B650 that is simply not doable. So good choice on the iGPU that should work well. AMD with iGPU is perfect because Intel CPUs don’t virtualize properly and you might end up gaming on a bunch of efficiency cores. I daily drive a Threadripper Pro but I also have a Ryzen 7 7700X build running Proxmox and for what it’s worth the pass-through has been a nightmare. I have had boards in the past that let me pass through all my ONBOARD controllers. I could send the onboard Wifi controller alone to a KVM and run an access point with hostapd on Linux. Doesn’t work on my Ryzen chip. On other boards I could pass through the SATA controllers andto give KVMs raw ZFS drives. And this new one is a high-end X670E chipset and none of that works. Not even raw devices. Not even GPU in its own group with ACS and all the usual workarounds. Instant crash every time and I have done this dozens of times at this point. So you may have better luck it really is a dice roll but in my experience the more you spend the better your luck and B650 is probably going to be finicky at best. I have also read about issues with AMD GPUs in passthrough as well. I use one for the NixOS hypervisor instead and it runs fairly well. But even after patching the reset bug there are many other potential issues when using them in KVMs. Nvidia simply does it better. I have a 3080 Ti passed through that has been seamless. Nvidia stopped blocking KVM usage with the code 43 error and now you don’t have to do any VBIOS flashing or anything.

ALSO: Looking Glass support is supposedly particularly bad for AMD GPUs. If you plan on using that I would go Nvidia.

2 Likes
15

Looking to get the ProArt X670E for this exact purpose. Did BIOS updates resolve any of the problems reported?

16

I can’t get 4 DIMMs over 5200MT/s with the latest BIOS (1904), but despite the lower frequency performance in benchmarks is actually better than it used to be.
Which problems are you referring to? I didn’t report any in that post and overall I’ve been very happy with the motherboard.
It’s currently in my server, though, with my workstation using a B650I Aorus Ultra.

17

People just complaining about “things being broken”. Great to hear you’ve had success!
I’m looking at 2x this kit:
www dot crucial dot com/memory/ddr5/cp2k48g56c46u5
Not on the QVL but since it’s EXPO it should be fine?

18

Should be fine. It’s Micron dies, so probably not as overclockable as my Hynixes, which eagerly do 6400 (rated for 4800) with just two installed, but since you only have two running them at the rated 5600 shouldn’t be a problem.
Remember to turn on ECC in the BIOS. It’s off by default and silently resets to off if memory training fails.

1 Like
19

Two kits of 2x48GB! I aim for 192GB total. If I have to slow i down then so be it.

20

In that case, you almost certainly will need to. The ProArt will default to running them at 3600. 4800 should be relatively easy to get stable. Beyond that is a gamble and a lot of tedious work.
Mine are 4x32GB bought individually, and 32GB is less stable than 48GB in general, so you may have a much easier time of it than I did.

1 Like