I have made some progress (maybe)?
I while searching for the message “refused to change power state from” I came across this post in proxmox forums which suggested enabling option disable_idle_d3 to avoid this, and so I’ve now added this:
root@pve:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:73df,1002:ab28,1002:67df,1002:aaf0 disable_idle_d3=1
This improved the situation in that the card no longer gets stuck in the power state, but still can only boot once. It’s just that all the messages regarding power-related failures are no longer showing up, so none of this anymore:
Feb 11 21:41:08 pve kernel: vfio-pci 0000:0b:00.0: refused to change power state from D0 to D3hot
Feb 11 21:44:38 pve QEMU[71457]: kvm: vfio: Unable to power on device, stuck in D3
Feb 11 21:47:28 pve kernel: vfio-pci 0000:0b:00.1: can't change power state from D0 to D3hot (config space inaccessible)
Now, I still see these the second time I try to start the VM:
Feb 12 15:22:04 pve kernel: vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Feb 12 15:22:04 pve kernel: vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Feb 12 15:22:04 pve kernel: vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
Feb 12 15:22:04 pve kernel: vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x27@0x440
Feb 12 15:22:05 pve kernel: vfio-pci 0000:0b:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 12 15:22:05 pve kernel: vfio-pci 0000:0b:00.0: vfio_bar_restore: reset recovery - restoring BARs
# ... restoring BARs repeats several more times
And I still see these the third time I try to start the VM:
Feb 12 15:24:03 pve kernel: vfio-pci 0000:0b:00.0: timed out waiting for pending transaction; performing function level reset anyway
Feb 12 15:24:05 pve kernel: vfio-pci 0000:0b:00.0: not ready 1023ms after FLR; waiting
...
I will try downgrading my motherboard BIOS as suggested in that thread to see if it works. I had version F1 and updated to F3 but there is F2 to try.