Even a 4x GPU should be able to handle this without issue, you likely had a tuning issue elsewhere that needed correcting.
Feel free to open a thread here or jump into the LG discord if you would like to try again and solve the issue.
Running Single GPU Passthrough VFIO in Arch using qemu. Running an i9 9900kf with aorus 2080ti on custom loop with 32gb ddr4, Asus rog Maximus xi Hero. Passing gpu, and two physical nvme drives through to windows 10 vm and using a seperate 1tb nvme for arch install. Play most games in linux nowadays but need windows for the few stubborn games. Wendell, let me know if you want any other info or have any questions about the build. Not using looking glass as I am not using a second GPU at the current time but do have one and may test that out in the future.
Been using VFIO and looking-glass for a few years as a convenient and ājust worksā way to play windows only games and occasionally to use windows only software.
My current setup:
- I use NixOS btw
- Ryzen 5900X
- 4x16GB ram
- MSI x470 gaming plus max
- rtx 3080 ti (bottom slot, host)
- rtx 2070 (top slot, guest)
- 2 TB nvme ssd for host
- 1 TB nvme ssd for guest
- current xml
- nix config
This mobo despite only costing close to $100 has been almost perfect for VFIO. Both 16x get 8x lanes to the CPU, the host SSD, is connected to the CPU, and i still get to use the second m.2 slot. I have my host GPU in the bottom slot because of space constraints and the inability to select the boot GPU in the bios which isnāt ideal but works fine as long as i donāt have anything in the top GPU. Because I donāt use a dummy plug I instead use nvidiaās internal EDID tool that was stolen out of geforce now.
looking-glass has worked very well but I currently use a patch that fixes an annoying pointer grabbing bug.
Problems Iāve had in the past:
- Had to revert a vfio commit that caused the efifb (or whatever you call the thing that shows systemd boot messages) to freeze
- When switching to NixOS I had to increase the memlock limit (not sure if this is still a problem)
- Passing through a CD drive (for fun) caused a very consistent and annoying stutter that took me a bit to figure out
- I didnāt like how evdev passthrough behaved in qemu so I used a patch for a while
- Before looking-glass supported it audio was a headache
Overall I am satisfied with my setup and the lack of compromises but I hate libvirt and I want a simpler tool that just outputs a qemu command.
1 Like
I think a good idea might be to explain relevant bios settings and their naming for different vendors and where to find them we have enough people here to piece it all together i think and it would help a lot of new people since 99% of people have no clue what the settings actually do and how they are named including me.
Tbh, the last time I tried to run F360 on Linux was about 1,5 years ago, and since then I migrated to the setup mentioned in my post so I didnāt have the need to run it outside of Windows VM.
My pass through video card is in the x16 slot. Performance in Rocket League is acceptable at a pretty steady 60 to 75 fps on high settings. Remember this is a pretty old card. The resolution is 1600x900 windowed. I like having the vm windowed so it sit nicely on my main monitor. I did play at 1080 also and it seemed fine while testing. LG runs within a frame or two of the windows fps and sometimes even higher.
I disable spice while testing and used vnc. It was just easier to pull vnc out out of the equation and put it back in over spice. Once I got it working I never put it back in.
Question: Will I get a performance boost by using spice, or is it mainly quality of life enhancements.
Pipewire/Jack seems to work well and I havenāt had any sync issues on this system, but hey Iām always willing to tinker so I may take a run at reintegrating spice again tonight.
I notice most of the builds here are on AMD Zen platforms. Has anyone had success doing this on Intel 13th Gen? I recently upgraded to 13900K on a Proart Z790 board with Powercolor Red Devil 6900XT and a 5700XT. I dual boot but want to see if I can get vfio setup working. Would love to know if anyone is doing it on the newer Intel platform E/P cores, and if this poses any challenges.
So you asked us to post what we do with VFIO, Iām going to post what Iād like to do, but canāt because Microsoft.
I want to do VFIO with Windows as the host OS. My logic is lots of games come with anti cheat that doesnāt work if the OS is in a VM. So Windows as the base OS gives me an easy platform to run my games on. Then I want to do linux VMs with VFIO so I can have a desktop with some video acceleration. The trouble is that even with the most expensive version of Windows the Microsoft will sell you (10/11 Pro for Workstations), youāre just not allowed to do this. You can do it with a Windows Server OS, but not the desktop OS.
Iāve done it with a 12900K, and it works fine. You can give the P-cores to the guest and run the host on E cores, or vice versa, or mix them. Windows guests did seem to understand what the difference between a P and an E core was, the system ran no worse than Iād expect it to on bare metal, which admittedly wasnāt great at the time (I ended up returning the 12900K for this reason, scheduling was a menace of microstutter even doing basic stuff).
The only real argument against doing VFIO on Intel is the limited amount of PCIe lanes available. 16+4 off the CPU and up to twenty of the Z790 chipset (connected through 4 lanes to the CPU, so effectively 16+4+4). If your motherboard supports bifurcating the x16 from the CPU into x8/x8, and gives each x8 device its own IOMMU group, you can do two GPUs and two NVMEs without encountering any bottlenecks, but that could be pretty limiting if you wanted to do storage pools with multiple M.2 SSDs, or pass one through to the guest. AMD on the other hand offers 16+4+4 off the CPU and an additional 20 through the X670E chipset (which is connected through 4 lanes, so effectively 16+4+4+4). It gives a lot more wiggle room for high performance networking or storage pools. Definitely a more attractive low end workstation platform than the Intel, where even the ProArt workstation motherboard is basically only good for gaming. 10GbE NIC is nice, but you donāt even get thunderbolt built-in (AFAIK), which is weird since you do get that on the AMD version.
1 Like
If I understand you correctly and youāre using VNC for input, yes, very yes. VNC input is terrible compared to all other options. I am not sure why you would even have done this to start with as the default SPICE setup from libVirt just works.
Good luck, if you run into problems feel free to open a thread here or join us in discord (Looking Glass)
Perhaps look at VMWare Workstationā¦ sorry but when it comes to VFIO under windows there is not much anyone can do as itās closed proprietary software.
As for anti-cheat VM detections, speak with your wallet like most of us in the VFIO community is. Donāt play games that ban VMs, refund them on steam, complain in the support forums, etcā¦ Some titles are already on our side for this due to our community being outspoken about it.
For example: https://docs.vrchat.com/docs/using-vrchat-in-a-virtual-machine
No I donāt use any vnc unless I canāt get my vm running or I run into a glitch then I might put vnc in with Virt-manager. Iāve used spice before and know it is much better than vnc. VNC was just easier to add and subtract then Spice.
Ah so youāre using libVirtās VNC instead of SPICE to recover a broken VM. In which case what are you using for input when itās not broken?
In the case of libVirt, there is no reason to use VNC over SPICE except if you plan to use a third party VNC client instead of the built in spice viewer built into virt-manager.
As of LG B6 we support video over spice as a fallback option when the VM breaks, as such you donāt even need to mess around with libvirt anymore if this happens. Simply leave the default VGA SPICE device on the VM and in windows disable the output in the display properties. If the VFIO gpu fails to operate, windows will re-enable the only other working display and the LG client will use it instead (without acceleration obviously).
1 Like
When itās not broken I pass through a secondary logitech receiver as a dedicated mouse and keyboard. As I use my vm mainly for gaming I need a regular mouse. I use the Logitech ERGO M575 Wireless Trackball Mouse on Linux but it is absolutely useless as a gaming mouse, so i require a second mouse anyways. In practice I only have one extra keyboard on my desktop which would be there anyway since my daily is wired MS ergo and I like have the wireless to move around the room.
Hope that all makes some sort of sense.
yup, entirely and itās one of the few niche caches where it makes more sense to not use spice for input. I do however encourage you to investigate not just disabling spice entirely (-s) for your VM but leaving the rest of the functionality enabled (VGA, Clipboard Sync, Audio). LG has parameters to disable each component individually.
Oh hey, a place for me to brag about relevant things.
So my first VFIO machine was
-Threadripper 3975WX
-Asus Pro WS WRX80E-SAGE SE WIFI
-512 Gb RAM
-Several USB cards, one per VM.
Over a year it accumulated a couple of GPUs
-ASRock Radeon RX 6900 XT Phantom Gaming D 16G OC
-Gigabyte GeForce RTX 3090 TURBO 24G
Everything got watercooled eventually.
Donāt have any good finished product pictures, here are a couple of in-progress ones.
It also had a bunch of other stuff coming and going over time, like Xeon Phi card, a 1070 Ti GPU, the 4x nvme card.
Naturally, this wasnāt built all at once and was slowly put together over about a year.
The AMD GPU came first, and it took some shenanigans to get it to pass through and get it to work.
First time around i used it for a Windows gaming VM, but it didnāt last all that long in it and started freezing up and crashing a few months in. So it was relegated to a Linux VM, where it worked flawlessly ever since, and the gaming VM switched to a recently acquired 3090.
3090 worked with passthrough like a charm out of the box with no shenanigans required other than passing both subdevices of it in a correct topology, and figuring out that it wonāt work in the top slot.
Well, some shenanigans were required i guess.
It was mainly used for gaming VM and occasionally for ML stuff in a Linux VM.
Through the power of long wires and an HDBaseT USB/HDMI over ethernet cable extender, this machine got split into several stations and/or monitors around the house.
The main station (back to the wall, so hard to take a picture of).
The cozy station under the bed, with RGB and stuff.
All in all, this was my pandemic project, when i sat at home with much income and not a lot to use it on.
As pandemic was coming to a close (or rather a no-one-cares-about-it-any-more-se), this abruptly came to an end.
Shit happened, i moved to another country, and most of this stuff got left behind and eventually auctioned off or reused elsewhere.
Nowadays i got another setup:
-Ryzen 5950x
-ASUS ROG STRIX X570-I GAMING
-64 Gb of RAM
-Dan A4 case (yes, the CPU is TDP limited in the BIOS)
-Gigabyte 3070 eagle (upgraded from Palit GTX 1070 Ti Dual, which moved to an eGPU enclosure)
Donāt remember having any new issues setting it up, it all just worked.
Used in about the same way, since i continue to dislike the idea of letting windows bareback any real hardware and like the simplicity of having setups in VMs.
Practically the only change is that itās a single station, single primary VM at a time setup. I need to ārebootā to switch from work VM to gaming VM or back.
An interesting problem with thit setup was that itās in the same room i sleep in (the other one was in itās own ventilated and soundproofed closet), and i donāt like shutting down the work VM since that loses state. So i tried, and succeeded, in making a VM hibernate.
Which is rather convenient. I can either power down the host afterwards, or even boot another VM in itās place without any usual fears associated with hibernation and dual boot.
So yeah. I do look forward to this way of using computers becoming easier and more common. Splitting one GPU between several VMs would have been nice. Double nice if you can assign half of the cardās ports to one and the other half to the other.
Oh, and i use Arch on everything thatās not the win10 gaming VM, btw.
Ill add my shine metal prone to this endevor !
I love VFIO. The only two things that make it less than pleasant are battleye anticheat for rainbow six siege (I have to dual boot an OS just for that game) and evdev not being resettable until the VM is restarted.
For the anti-cheat, there might be little we can do - would these systems be okay if memory was encrypted on the guest?
For the evdev, itās annoying when a USB hub resets and Iām in the middle of a game. I have to reboot the whole system to get my input back. I like evdev because I can switch between the host and guest for my keyboard and mouse with a simple combo, and just switch display inputs and use scream for audio. I donāt use looking glass even though itās a great tool.
I just got bored about current and next GPU news and bought a 2nd hand 7900XTX Reference card for $800.
I may be wrong, but i may say i now have GPU for next 2 gens.
And so with no further do, I passedthrough it succesfully without doing anything special. Just got assigned the pci reference numbers to the vfio kernel args and make a little requirement to load the vfio-pci driver before the amdgpu driver as found on a promox forum post (because using a 4750G Renoir Desktop Ryzen 7 Pro iGPU for host on a X570 MSI PRO MB).
The only tricky thing i had to do is start the VM without HDMI cable plugged in. Same symptom as with the Intel ARCs on their starts.
No need to use ReBAR or anything else. Performance is pretty good all the way, and i only noticed some delays on rendering depending on the game and/or Chromium-based apps/GUIs. FH5 AAA game at 4K60 with Extreme settings is using about 50% of the GPU and about 12 GB of VRAM, although in some scenarios ups to 80% of GPU utilization. I know that if i make things right i could have an extra performance on it, but the thing is that with ReBAR enabled i loose the possibility of having a two-way GPU passthrough working together with another Intel A770 16GB LE that i use with a Win11 Insider Dev preview to test out WSL2 (<-- custom WSL2 distros, as endeavourOSWSL2, doesnāt get working GPU accel) + WSA with nested virt enabled (+ with now with stable diffusion: tomshardware/news/stable-diffusion-for-intel-optimizations).
Background is that the VM i assigned the 7900XTX was already setup with a Win10 /ameliorated.io 21H1 image, and assigned to it bf a 6750XT Red Devil, with drivers set at 23.4.2 Adrenalin but through SDIO (didnāt update the drivers yet, nor planning to, as perf is good, just maybe when SDIO releases another stable driver iāll maybe consider it). That maybe did the trick also.
Didnāt tested tho if thereās any reset bug yet with this setup. The Intel surely does have it, being more harmful rebooting the VM that shutting it down as it canāt start again, but shutting it down gives me the ability to recover control of my evdev USB input devices and get a freeze on Virt-manager software that only leads to force closing warning dialog from Cinnamon, and so i force it.
Then i open it again and while it stays 30 secs until it connects to the libvirtd service, at the end service get āautoforcefully restarted?ā and all memory is freed.
I then reboot entire host and only remains the qemu-system-x64 emulator process hanged that gets killed by the rebooting cycle after a while (30 secs max).
Then on the next boot the Intel card will not be available, moreover it wonāt even show up as PCI device at all to be usable, but rebooting again solves that issue.
With two rebootings seems like is capable to recover to a normal state itself for the next passthrough.
Both of them canāt display the OVMF UEFI screens (6750XT did), so for updates and other pre-boot screens i have assigned the SPICE graphics with QXL video, and disabled it on the first place in device manager once the system and drivers have been setup (talking about Intel ARC VM).
And so to update between Win11 Insider Dev builds i donāt assign the pcie card and work a bit with that QXL alone to see what the VM is doing. The thing is i now remain stall on build 23493.1000 bc newer build fails at initializing VirtIO drivers. It wasnāt a new release of them at fedora group since January so iām guessing it has to do with that.
Have two monitors, one 4K60 Benq for gaming, other 1080p Benq for work, pretty rock solid except for the reset bug of the Intel rn. 2 keybords/2 mouses/2 audio interfaces (HDMI/Displayport through monitor speakers).
Kernel on Host is Manjaro 6.3 Mainline (which is EOL now BTW, no VFIO variant, good MB IOMMU groupings). 6.4 fucked me some peripherals so iām sticking with 6.3 until it gets better.
Iāll wrote down what AGESA version is using this MB once i get the chance of rebooting to check it out, as i updated it back in january 2021 for changing from the 2700X to the 4750G and since everything was working ok never updated it anymore.
I also have an Ansible role written by me to automate setting up VFIO passthrough. Donāt have it uploaded anywhere tho. Code isnāt clean and i am somewhat lazy to clean it. Is simple, it works, but itās not perfect. I feel Iām not worthy to show it up in public, although if someone asks for it, iām willing to share it, np.
Written from my Cromite web browser through Tor (root iptables Invizible Pro app) under WSA, under W11, under Manjaro.
P.D.: BTW Wendell why do you have your web/forum cloudflareeeeeed? github/zidansec/CloudPeler /crimeflare.eu.org
P.D.2: Use this against DDoS and not Crimeflare: github/SpiderLabs/ModSecurity