I’ve been running a vfio setup for several years now, and I’m looking for suggestions on how to make runtime bind/unbind a bit more pleasant to work with.
What I have:
Two Nvidia GPUs, one of them used only for the guest VM, and another one used by my Arch (btw) host. Swapping the host GPU for a non-nvidia one is not entirely out of the question, if required.
Keep my guest GPU in power saving state when the VM is off. This is important as otherwise it will suck 100W permanently, getting hot and reducing fan lifespan.
What I knew:
To achieve 1, I must not let VFIO grab the guest GPU early on boot (that is, let the nvidia module have it)
To achieve 2, the guest GPU has to be bound to the nvidia module, and nvidia-persistenced has to be running. It needs to be kept running, it does not work (to my knowledge) to start it and then stop it.
To be able to launch the VM, I need to unbind the guest GPU from the nvidia module. To do this, no process must be using that GPU
X.org will start using any GPU bound to the nvidia module, so the guest GPU must not be bound to the nvidia module when X.org starts
With this info, I made an sketchy setup that roughly consists in the following:
VFIO is not started at boot, so nvidia binds both GPU
A systemd unit is run Before=display-manager.service, which unbinds the guest GPU
Another systemd unit is run After=display-manager.service, which rebinds the guest GPU to nvidia
nvidia-persistenced starts after that, putting the GPU in low power mode
When the VM is started, libvirt hooks stop nvidia-persistenced, reset the gpu with nvidia-smi, unbind the GPU from nvidia, and bind it to vfio before starting the VM.
I thought this was going to work fine, but it does not. It turns out that some applications (I believe those that use OpenGL?) will attach themselves to the guest GPU even if X.org itself is not. Oddly enough, applications only start using the guest GPU if nvidia-persistenced is running (I have no idea why)
The fan is not turning and the heatsink feels pretty warm but not hot. I have no idea how much power it is actually pulling but it is for sure much less than 100W.
Perhaps you could check bios options for PCI power management?
I do sometimes have the GPU bound to the host, and I also have some apps that keep running on the GPU spontaneously. Usually Firefox or Chrome…
I have a Gigabyte Z390 M GAMING (Intel-based), and the GPU is a 3080 Founders. I’ll take a look at PCI power management but I have the suspicion the PCI bus power state may not be the same thing as the GPU power state.
I check the GPU power state with nvidia-smi, which, when bound to the nvidia module and with nvidia-persistenced running, says:
The P8 there is a very nice power saving level. Obviously nvidia-smi cannot check those values while the gpu is bound to vfio, but if I swap from vfio to nvidia and check right after that, it says P0 and ~100W. Most noticeably, the fan is spinning and the GPU is very hot to the touch. I could not keep my finger on the backplate for more than a few seconds.
My guess here is that the 3060 might not get as hot in P0 as the 3080 does.
Does the power_state show D3hot while it is bound to vfio? If not, getting D3hot with vfio might solve your issue? There are some bios options on my board related to that, if it is not in D3hot.
While bound to nvidia, I also get D0. Right after binding I see 47 W / 170 W used, after a bit it goes down to 18 W.
Sadly there is no way to monitor power without binding nvidia AFAIK
After some testing I find that my card stays cooler with your method too. With both methods the fan does not spin up, but with yours the heatsink is much cooler to touch.
With both methods the fan does not spin up, but with yours the heatsink is much cooler to touch.
Makes sense and aligns with the idea that GPU and PCI have different (complementary) power saving settings. Howerver using nvidia for power saving is a huge pain in the ass, if you can live with just the PCI power savings I’d stick with that.
And perhaps using the environment variables of the iGPU explicitly for those programs would work?
Gave this a quick try, but no deal. I remember reading some years ago that the nvidia drivers did work with PRIME as a secondary device, but not as a primary one. I’m not sure if that still applies, but I’d bet it does.
The weirdest thing is that applications do not attach themselves to both GPUs if nvidia-persistenced is not running. I don’t know if there’s anything that could be done around that.
I’ve come up with a really, really stupid workaround for this that somehow works.
I’ve created a dummy linux VM, to which I pass the guest GPU when the Windows guest is off. This dummy linux VM just has the nvidia module and nvidia-persistenced enabled. When this VM launches, it grabs the GPU and puts it in power saving mode.
In the host, it is still bound to the vfio module so no application does funny things with it.
I still can believe how stupid it sounds that I can save 60W of power (and fan lifespan) by starting a VM that does nothing, but there’s that.
