I’m using Arch Linux on the host. So I’m using the pretty recent kernel 6.14.4. I’m loading vfio-pci in initramfs, so I don’t use the card on my host yet.
My VMs all use UEFI and Q35 chipset.
I have managed to put together a stable configuration, but it took a lot of fumbling in the dark to get there. I was inspired by other threads on this forum from folks struggling to get 9070XT GPU’s to work with VFIO, so I figured I would try something similar to what they were doing. Here’s what I currently have working:
- Version 570.144 of the nvidia driver installed in the host OS.
- Hardware resets disabled for the GPU by writing an empty string to /sys/bus/pci/devices/<pci_addr>/reset_method (I have a startup script that does this automatically)
- modeset disabled for the nvidia driver by adding “module_blacklist=nvidia_modeset” to the kernel command line in my grub config. This is necessary in order to be able to bind / unbind the nvidia driver from the GPU.
The nvidia module unbinds and the vfio-pci module binds to the GPU when VM’s start up. Conversely, the vfio-pci module unbinds and the nvidia driver binds to the GPU when VM’s stop. The driver seems to reinitialize the GPU on each bind so that the next VM startup succeeds.
I should mention that my host GPU is an AMD RX570 that doesn’t depend on the nvidia driver, so I can get away with disabling modeset in the nvidia driver.
There’s almost certainly a better way to do this, but it works so I’m leaving well enough alone. 
So apparently, NVIDIA seems to have issued a fix at least for the RTX 5060 Ti.
I experienced almost the same exact issues while having none with an RTX 4070 Super. All I had to do was boot into Windows bare-metal, download the NVIDIA_UEFI_Firmware_Updater_2.0-x64.exe and run it. It’s a tool that updates the GPU vBIOS from NVIDIA, and after that I was able to use my RTX 5060 Ti in my VMs as normal, with ROM BAR enabled and with the OVMF screens appearing just as normal, no more crashes or anything.
I didn’t test for long though, and I’m absolutely not sure it’s going to work for everyone and for all RTX 5000 GPUs.
Unfortunately I cannot post links, but if you search “NVIDIA_UEFI_Firmware_Updater_2.0-x64.exe” on Qwant, you just have to click on the very first result.
1 Like
I experienced frequent reset problems with an RTX 5060 Ti for a while. They have so far not happened after upgrading to kernel 6.15 and nvidia-open 570.153 inside the VM.
This works for me even without any nvidia drivers on the host.
Thank you so much !! Finally I have VM bios.
You made my day!
Have you enabled Resizable Bar?
I’ve had the 5090 in my system since Tuesday, and switching from the 7900XTX to the 5090 was no problem at all.
I only changed the PCIe IDs in GRUB and have blacklisted all Nvidia relevant drivers.
I’ve enabled Resizable BAR; I suspect LLM benefits from it, but I haven’t tested it.
Unfortunately, I’ve already had two bluescreens with the 5090, both with no load on the system after several hours of gameplay without any issues.
The Linux host system continues to function normally but the 5090 needs a host system reboot so that I can start the VM again.
Without a bluescreen, it’s no problem to start the VM multiple times a day or to switch between Linux and Windows VMs.
Switching with KVM is also no problem, I haven’t had a black screen yet.
I am sure that my system is stable, the last BIOS update was about two months ago and in all that time I had not a single problem with the system.
I first thought that my RAM was getting too hot due to the flow-through design of the 5090, but the bluescreens happen without any system load.
Hi, I came across this thread as I’ve just ordered a new (to me) Threadripper Pro 3945WX and Gigabyte MC62-G40 with 128GB RAM to swap out an older proxmox node.
Is your Linux host with the 5090 a VM or an LXC?
The question I have failed trying to answer for myself is related to my 5090 under proxmox. CurrentlyI have it in a dedicated windows 11 machine for PCVR Quest3 simracing use. since getting the 5090 Ive been playing wiht LLMs, im really intrigued by it and I see myself wanting to learn a bit more and maybe even train my own models. So I fugred I’d treat myself to a new AI optimised Proxmox box to replace the old HP Prodesk. (hecne the TR Pro 3000 for all teh PCIE 4.0 lanes for multiple GPU’s
I was thinking to sell my old 4090 to acquire dual 3090’s but it seems like wasted resources with the 5090 sitting idle in the windows 11 machine. So now Im starting to wonder at the feasability of ditching the windows machine altogether and droping the 5090 into the threadripper box. negating the need for 3090’s and if justified cudl prurchase another 5090 later on down the road.
Im not particularly knowedgeable/skilled with this kind of stuff and have been strugling to find clear conclusions. I somehow stumble my way into required solutions, lol.
Could I get some advice from you on whether it is actually feasible to run a linux(Fedora KDE Plasma) desktop in an LXC with 5090 pass-through (ideallly LXC and not VM as that way I could still use transcoding on the 5090 in some of my other LXC’s) to handle both Monado>WiVRn for Quest3 PCVR and Cuda LLM duties?
I use the 5090 for Windows and Linux VMs, hence no LXC.
But using only one 5090 for LLM and simracing at the same time is probably not the best experience anyway.
I use two 7900XTX with docker and Stable Diffusion and LLM at the same time is not optimal either.
I haven’t tried GPU passthrough with LXC yet, but I’ve had good experiences with a TV card, it worked better with LXC than with KVM.
I would recommend not buying any more GPUs for now until you know what quality level you’re aiming for.
The 48GB of VRAM in my AI system are already insufficient, and I’m considering buying a ZEN6 Epyc system next year and running inference on the CPU.
Thank you so much for taking the time to read and reply to my message, really appreciate it.
You are absolutely right, I would never run the simracing and LLMs together at the same time it would always exclusively be one or the other. The reason I was considering putting the 5090 and both PCVR and AI functions into one virutalised instance was to check viability of abandoning the bare metal windows machine which currently hosts the 5090. And moving all of my compute/AI/gaming/PCVR needs into the rack within a more powerful (for my standards, lol) hypervisor, giving me future option for dual/triple GPU. Thereby getting more use out of the 5090 doing other things with it rather than having it idle when not gaming and spending money on alternative LLM GPU’s.
My current windows box is an ITX SFF so thats already stuffed to the gills and no room for expansion.
So far on the 5090 32GB VRAM I have been runing Gemma3_27b and Qwen3_32b. And I know I would like to test bigger models. But Im clumsily trying to figure out whether 2x32GB (2x5090, one day!) is significantly more capable than 2x24GB (2x3090)
Why its not an obvious answer to me (even though bigger number is always more) is because I noticed that having 32GB 5090 now compared to 24GB 4090 before I can see Im not really able to run larger parameter smarter models than the 4090(or 3090) would have been able to run. I’m just using the same parameter models but mainly with a bit more context and maybe a bit less quantization.
So I wonder with the standard crop of available model parameters if its the same situation with 48GB vs 64GB and to properly get to bigger and smarter models than 48GB VRAM permits, I would actually need to go even further beyond 64GB VRAM. Making the idea of future dual 5090 a waste of resources over dual 3090
That being the case would I be better off settling for the lower expense of 2x3090 when I eventually sell the old 4090 and leaving 5090 in windows gaming rig? I hope that makes sense
Well, I think the quality level of LLMs is a vague and moving target.
Discussing general questions about politics and society with an LLM is interesting for reasons other than the usefulness of the answers, if you know what I mean.
The answers are usually not good or at least the arguments are not properly weighted, in the worst case wrong and if you argue interactively with them, they will eventually start to hallucinate.
You won’t see a better result with 2x32GB than with 2x24GB, yes the two 5090 are faster, but that has no practical advantage and the quality of the responses is the same.
It’s hard to say whether a 70GB or 400GB model will be sufficient for me in two years.
I’m not in a situation where knowing how to use LLMs correctly or their answers gives me any advantage.
Nobody expects me to perform more or better and except as a programming assistant, I currently consider LLMs useless.
To be honest, I think my main motivation is that it gives me a reason to continue to work with workstation/server hardware and the various virtualization techniques, rather than the LLMs actually being of any use to me.
I already had the Epyc ROM system and also a 7900XTX, the second one didn’t cost much.
The system is very usable for 70GB models, if the responses don’t need to be interactive and you have some time, models up to 140GB will also work.
I’ll just wait and see what a ZEN6 Epyc can do and then probably sell everything I currently have except the 5090 and buy a 64-core Epyc with 768GB of RAM, put the 5090 in and have a viable system for AI applications and gaming.
If you haven’t seen it yet, AMD has announced that Epyc based on ZEN6 has a per-socket memory bandwidth up to 1.6 TB/s, which is three times more than current systems.
Oh guys u am having the same issue and tried everything ti solve this in past month.
RTX PRO 6000 and RTX 5090 both are crashing the same way.
After first boot everything works and passthrough ti both windows and linux works fine.
Then at some point some guest VMs when shutting down, they hang in shutting down state and host starts throwing CPU soft lockup and gpu cannot be accessible.
Motherboard is asrock. Anyone have thise issues on other motherboards as well ?
I dont want to run nvidia drivers on proxmox, is there any way to fix it or binding and unbinding at every vm start and stop solved this issue ?
Is it maybe happening if gues is using dome older driver ?
Applying firmware upgrade of gpus did solved aome startup issues but did nit solved thise shutdowns that sofl lockup cpu on host.
what vbios you have on rtx pro 6000? 2.52 seems bugged for this use case
Passing through a 5090 here, and was experiencing similar reset-bug-type issues:
Host:
- Linux (Debian 12, XFCE)
Guests:
- Windows 11 VM: no issues at all
- Linux (Debian 12, XFCE) VM: shutting down the VM crashes the host
The ONLY thing that fixed it 100% on my system was to disable the nvidia-drm modeset option on the VM:
- Edit /etc/modprobe.d/nvidia-modeset.conf and set ‘options nvidia-drm modeset=0’ (or just add .BAK to the filename of the conf file itself).
- Make sure the nvidia-drm modeset option is not activated by any other conf files anywhere else (typically in ‘/etc/X11/xorg.conf.d’ or ‘/usr/share/X11/xorg.conf.d’).
- Create a new conf file in /usr/share/X11/xorg.conf.d to manually set the X device driver, which need only be as basic as:
Section “Device”
Identifier “MyDevice”
Driver “nvidia”
EndSection
Section “Screen”
Identifier “MyScreen”
Device “MyDevice”
EndSection
- Don’t forget to update-initramfs -u
This completely fixed things on my system: rock solid now and no more lockups. But I am running X11 on the VM, and I seem to remember that having the modesetting option enabled is a dependency for wayland, so not sure if this will work there. Note I’m on Debian 12, but I imagine the same fix should work on Proxmox.
I tried lots of other things first, NONE of which seem to make ANY difference on my system:
- Flashed the 5090 firmware with NVIDIA GPU UEFI Firmware Update Tool v2.0 (it’s not only for 5060, but really for any 5 series).
- Installed various versions of the NVidia linux driver on the host (turns out none are needed).
- Blacklisted (and otherwise messed around with) the NVidia Audio Device (think that’s a red herring).
- Tried various hyperv flags.
- Tried various kernels on the host.
- Tried various kernel flags (pci=realloc=off, etc).
- Hook scripts to attach/detach the GPU instead of vfio-pci or pci-stub.
- Messed around with various combinations in the host BIOS (CSM, ROM BAR, SRV-IO).
So I’ve ruled most of those things out.
3 Likes
I have no more problems with the 5090.
I use a dual boot with Kubuntu 25.04 and CachyOS, both host systems are configured identically regarding PCIe passthrough, except that “mkinitcpio.conf” must be edited in CachyOS.
Something strange was that the GPU did not require a VBios copy for the first few weeks, but 3-4 days ago, the 5090 stopped showing any display output, neither in Kubuntu nor in CachyOS as a VM host, but CUDA applications continued to work.
All VMs were affected, Linux and Windows with different driver versions.
The solution was to give the GPU a vBIOS copy via Libvirt, no more problems since then.
I got the BIOS from Techpowerup.com, it’s not the same card, but that doesn’t seem to matter.
I use ResizableBAR and SR-IOV, but that’s because of my network card, no idea if that affects the GPU, but I saw that AMD recommends it for ROCm.
cat /etc/modprobe.d/blacklist.conf
blacklist nvidia
blacklist nouveau
blacklist nvidiafb
blacklist nvidiafb_drm
blacklist nvidia_drm
blacklist nvidia_uvm
blacklist nvidia_modeset
root@kubu01:~# cat /etc/modprobe.d/vfio.conf
softdep pcieport pre: vfio vfio_pci vfio-pci
options snd_hda_intel enable_msi=1
options kvm_amd avic=1
#Grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash apparmor=0 iommu=pt mitigations=off default_hugepagesz=1G hugepagesz=1G hugepages=96 vfio-pci.ids=10de:2b85,10de:22e8"
#Libvirt
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<runtime state="on"/>
<synic state="on"/>
<stimer state="on">
<direct state="on"/>
</stimer>
<reset state="on"/>
<vendor_id state="on" value="AuthenticAMD"/>
<frequencies state="on"/>
<reenlightenment state="on"/>
<tlbflush state="on"/>
<ipi state="on"/>
<evmcs state="off"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
<smm state="on"/>
<ioapic driver="kvm"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="off">
<topology sockets="1" dies="2" clusters="1" cores="8" threads="1"/>
<feature policy="require" name="topoext"/>
<feature policy="require" name="invtsc"/>
<feature policy="disable" name="monitor"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="kvmclock" present="no"/>
<timer name="hypervclock" present="yes"/>
<timer name="tsc" present="yes" mode="native"/>
</clock>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</source>
<rom file="/var/lib/libvirt/bios/msi.rom"/>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0" multifunction="on"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x1"/>
</hostdev>
Another Debian 12 user here with KVM and PCI passthrough.
omg, thanks so much for this; I also went through your list of tries (and others), but I never thought the issue can be solved in the VM…
I have seen no downsides (if there’s any) with this config change yet, but probably it solves the problem for me as I only use the blackwell cards for ML/LLM stuff.
1 Like
One of my clients was able to reproduce crash on unsloth training and then he fixed that case by adding modset and xorg provided by you !!!
Nice. I am talking tk nvidia support and passed them that info to see if they could fix it in bios. Setting anything in my old vm in each client is impossible lol.
So they would better do some fix in bios.
Will let know if i will find anytging.
2 Likes
Got response from nvidia that they were able to reproduce this issue and they are thinking about fix.
Also i have installed apt install proxmox-kernel-6.14.8-2-bpo12-pve/stable and i see that RTX6000 boots super fast now vs very slow when i had older 6.8 and 6.11 kernels. In 6.14 they added some support for blackwell so worth to try it out.
Anyway the crash on shutdown is caused by either specific training itself or/and some module options for nvidia.
The training that caused issues afte applying options nvidia-drm modeset=0 and /etc/X11/xorg.conf.d now it does not crash gpu anymore.
But since client can do any stuff in VM, this is not good solution.
1 Like
Still testing, but it appears to be helpful! We have a bug bounty for this issue - feel free to claim it. You can reach out to us at [email protected] or join our Discord (links are not allowed; please search for ‘cloudrift ai’).
7GQcJ2TVM3BmsbNn