I’ve been pulled into the VFIO scene and am intrigued of the idea.
Right now my only concern is to not waste money due to mistakes of hardware, I’ve researched as much as I can and most of the sources are outdated such as IOMMU group issues and such.
So I’m coming out here to you vets to see if I’m in the right direction, the hardware I’m going to purchase is:
AMD Ryzen 9 3900X (used to be: AMD Ryzen 7 3700X)
ASUS ROG Strix X570-F
G.Skill Trident Z Neo 32GB (2x16GB) 3200MHz CL16 DDR4
ASUS ROG Strix Radeon RX 5700 XT OC 8GB (used to be: XFX Radeon RX 5700XT THICC III Ultra 8GB (out of stock)
ADATA XPG SX8200 Pro M.2 NVMe SSD 1TB
Corsair HX100 Platinum 1000W PSU
I’m salvaging parts from my current setup which are:
XFX Radeon RX Vega 56 8GB
ADATA XPG SX8200 Pro M.2 NVMe SSD 1TB
Western Digital WD Black 1TB WD1003FZEX
Fractal Design Define R6
The idea is to put Pop_OS! on the new M.2 drive, then put windows VM on the older one, the HDD is on the fence for the moment, I want to run the 5700XT on host for Freesync 2 (if your curious on what monitor it’s this one: https://www.gigabyte.com/au/Monitor/AORUS-CV27Q) and the Vega 56 on the windows guest.
And the looking glass software page for setup, I’ll be using Synergy for simplicity between the host and the guest, and an audio setup from here: https://passthroughpo.st/audio-hardware-guide-for-vfio-setups/ since I use both a mic and a headset constantly and is needed in windows.
I’ve heard issues with freesync on linux but I recall that there’s a hacky way of “fixing” it. Not too sure.
Is this an ideal build? Am I in the right direction?
@Shukaze I have a suggestion, Have you consider switching to an Intel build instead of using AMD? All the research I have done setting up what you want is easier on an Intel platform. I am not saying it isn’t possible with an AND platform, setting up a system like you want isn’t as simple as a lot of people make it out to be. I am not trying to discourage you, but trying to save you a lot of headaches. What you may want to do instead is set up a dual boot system. A dual boot system is where you have a choice between two operating systems. Or maybe just have two different builds one using Linux as its operating system the other uses Windows.
The option I am using currently is I have Windows 10 on my Laptop, and I have Linux Mint on my desktop. Also If I need to run Windows 10 on my desktop I have an external USB 3.0 hard drive (with an exact copy of my laptop’s main drive)to boot into Windows 10 on my desktop.
I tried dual booting for a while, it worked ok, but I found I spent more time with Windows 10 than Linux, so when I built my new desktop I decided to only have Linux Mint as a permanent operating software. I plan to force myself to learn all about Linux and end my dependence on Windows.
I don’t know how much better the Ryzen 3000 series is on IOMMU groups but the biggest issue was USB separation. My friend had to get a different USB card so can have two VMs with their own USB devices.
@FurryJackman Yeah, I’ve seen that in 5.5 I think. I’ll still use the patch though just in case since it doesn’t touch the quirks.c
@Shadowbane I’ve come from a IT background so messing with things I shouldn’t be is not a problem. I’ve had windows on my desktop for as long as I can remember and linux on my laptops, I’ve tried dual booting in the past and it was just tedious, I just need a solution that is just a few clicks away from launching windows applications that may not work in lutris/wine.
In regards to AMD and Intel, it’s all about pricing and technologies, I’ve permanently switched to AMD for the first time since the 2000 series Ryzens and was amazed of what I got (Ryzen 2600). I used to have a Intel 3570 before than and it really didn’t sit well with me.
It would be a looooooooooooooooong time before I go back to Intel, lol. Anyway, it’s not about how hard this is, it’s about asking the question “Is it doable with what I’ve got?”
Thanks for the replies, wasn’t sure if anyone was interested.
I’ve done a pretty similar build about half a year back.
Ryzen 2700x
Taichi X470
Radeon 570 Pulse for the Manjaro Host
Vega 56 for guest systems
The IOMMU separation is very good, though I would recommend to buy a decent USB switch for keyboard, mouse and USB audio interface.
I am able to run Windows and MacOS as guest OSes without much hassle.
Just the reset bug bites me now and then. A clean shutdown of the VM lets me start another or the same VM with passthrough so it mostly works. But some random time after I first use the VEGA for a VM the GPU cooler fan starts to run on max. Even when I previously cleanly shutdown the VM.
This happens sometimes a few hours after the first VEGA usage, sometimes after a few days.
I then have to reboot my entire machine.
I have not patched the kernel (because I never found a good guide how to do it) and the tool provided by gnif doesn’t work for me either.
It works good enough for me right now so I don’t bother patching (would be intersted in a comprehensive guide though) and wait for the day the patch finally arrives in the kernel.
I’m immediately going to do this after following the guide I’ve posted above. I was thinking of the hardware route for peripherals but I rather just spend $35 for a clean and seamless solution (plus it’s really cheap compared to usb switches I’ve looked at).
My only concern was the grouping of IOMMU but now that I’ve got 99% of the whole build worked out I think it’s safe to say that when I get it arriving later next month it should be fine, I’ll post my progress and findings.
Hey Shukaze,
I don’t think you need to reconsider your hardware purchase at all. If you take a look at my tutorial you’ll see that I used a 3900X, X570 Mobo, nvidia (guest) + amd (host) gpu. Even better, I also use Pop!_OS on my host and Windows 10 on my guest. You’ll find it very tailored to your needs (except there’s no part on looking glass).
So after all the product shortages and huge wait times, I’ve finally ordered my last internal part for this build.
But as I was preparing for the new part to arrive, I’ve found a blunder on my part…
The freaking 5700XT’s cooler is HUGE and now I have major concerns for heating. So my question is this, can I just put the vega in the first PCIE slot as guest (no output plugged in) and use the 5700 XT in the second slot for host (output plugged in to monitor)?
WIll that work? I’m asking this unorthodox workaround due to no fractal design raiser cables being available at this time due to you know who.
It’s all good now, got the last fractal design raiser cable (new) in Australia on amazon AU. Measured the distance between the Vega on the vertical and the 5700XT in the 1st PCIE slot, seems safe, about 3-6mm apart from the backplate of the Vega and the plastic cooler cover of the 5700XT.
There’ll be plenty of cooling coming from the front with those two 140mm FD fans that came with the case, the Drive base are removed except for one. Will post a image when all put together!
Sorry for the late update, things happened, moving on…
The riser cable was essentially useless, and fitting at least one 3ish slot cooler GPU card into any (I’ve tried, you can correct me if I’m wrong) ATX motherboard and lower that can accommodate it is foolhardy.
So I’m left with one decision, but first two very important questions:
How is the 5700 XT supported in 20.04 in VFIO and is it ok? If not, how is the geforce 2060 and > supported in VFIO?
If I put in a geforce card for the sole purpose of VFIO, and since it’s 3.0 PCIE, will this change the host card (5700 XT) to 3.0? Or is this only during operation in VFIO? I’m not familiar on how this works when talking about VFIO in general.
Now, my decision to support these beefy cards, I’ve considered moving to E-ATX. And the only choice here in Australia is the MSI Godlike, this gives any card in a 8x/0x/8x/0x setup more than enough breathing room.
I’m not concerned of the cost but other suggestions are greatly appreciated!
