Right, this will be a lenghty one.
I started using VFIO in 2018 when i wanted to move to linux for privacy reasons but still be able to paly games on windows.
My first build was a Ryzen 1800x with a Radeon 560 for the host and Vega 56 for the guest, with all the woes the amdgpu would have had. I surely never put my pc to sleep so much as in these times before the vendor-reset project.
With the years i have added and remade my setup to maximise my vfio endeavours, with the following setup at the height:
- Asus Prime Pro 570x Motherboard
- Ryzen 5950x
- 64gb 3200MT/S ECC Ram
- 1x Intel Arc A750 for my daily-Linux-VM
- 1x XFX 6900xt Watercooled GPU for “my” windows 10 gaming VM
- 1x 6600xt for a macOS vm, dual bootable to a second windows 10 gaming VM for when i have friends over who want to game.
- 1x p2200 quadro passed through to the linux vm to do de-/en-/transcoding tasks.
- 1x apple wifi/bt module to get imessage working in the macvm (totally useless otherwise)
- 1x fresco logic usb pcie controller card because its the only usb controller macos wants to use without drops. has reset bug on windows 10.
- All Storage is handled by the host, i run a headless host and do everything in a vm. I set up LVM storages for every vm and use the designated LVs as Sata-HDDs with the virtio drivers to let the vms directly install into the LVs. (yes, even macos has virtio storage drivers. looking at you, windows.)
I have a pile of Cables and Switches and Chinesium USB KVM switches to manage all that cabling. here is how it looked at the height:
You cant even See the 6600xt (water cooled), imagine the cable chaos behind the pc.
The P2200, as you can see, is connected via riser cable to a 1x slot, the other usb controller and the mac wifi sits on the other 1x slots.
EVERY PCI-E slot was populated. EVERY m2 slot was populated. plus 4x2.5" Sata SSDs for the VMs and hosts.
The Motherboard was maxed out to no end. If i was only using my daily linux VM and my gaming VM to play games, it was all fine, as soon as a third person started gaming or working you saw massive stutters and lag and we had to drop graphics fidelity down in every game to get it to work.
Nevertheless, it worked. I am an avid Looking-Glass user and penetrant tester of said Software (sorry gnif 
) with audio and video coming through the LG client.
I also stream with that rig. I think Gnif posted the video already in the forums, but here is a video i want to show:
excuse the german language, normally i stream in english even tho my german accent is very bad. here you can also see the two vms feeding into obs on my linux vm via the LG-obs plugin. we had to scale the resolution down to 864p to get it to run smooth. Also, imagine the heat coming off that thing when two windows vms are going full tilt and computing is happening all around.
At the moment i only have the 6900xt and the a750 installed as this bottlenecking was unsustainable so im down to a linux-vm and a win10-gaming-vm.
what i like about it the most is when i stream to the 0 viewers i have on average, i do not have to worry about any leaking or doxxing or any type of that, as the only captured screen is what the looking-glass-obs-plugin gets send. I Installed Firefox on the windows vm with no history enabled, did some tweaking to not have weather stuff displayed and i just can use the vm as freely as i want.
All in All, VFIO is, when set up correctly, very nice to use and there are few reasons to switch back from a vm only setup. I could see myself erasing the windows vm and installing steam os when valve releases steamos to the public, the anti cheat stuff is sorted out and when LG’s linux host is working ;-). otherwise, if it runs…
I hope it wasnt as rambly as i suspect it and you had a good time reading what i cooked up here. If you have questions, just ask. i might have missed a few things.
Cheers,
Corrgan.
let me shamelessy plug my streaming channels: