Urgent Help vfio-pci 0000:03:00.0: can't change power state on RTX 4060 on Second PCI Slot at X8 in ProArt X670E

Hi My Hardware

ASUS PROART X670EAMD RYZEN 7900X 12 CORE64GB DDR5 4800MHZ RAM

Bios - Latest October, 2023

RESIZABLE BAR - OFFPCI Version 4PCI LANE BANDWITH X8 X8 for PCI 1 & PCI 2

Proxmox

Proxmox 7.4.-17 ***(i tried 8.1 same issue and rolled back to 7.4)***pve-manager/7.4-17/513c62be (running kernel: 5.15.131-2-pve)

Basically i have 2 X 4060,

GPU 1 on PCI 1 works fine,

GPU 2 on PCI 2 is unaccessble if i do vfio.

The gpu is not accessible at all forget about passthrough.

I have setup everything like blacklist and all

Grub - GRUB_CMDLINE_LINUX_DEFAULT=“quiet amd_iommu=on iommu=pt textonly nomodeset video=efifb📷ff video=simplefb📷ff video=vesafb📷ff”

vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem📷wns=none[ 84.589746] pci 0000:01:00.0: Removing from iommu group 13[ 84.609813] pci 0000:01:00.1: Removing from iommu group 13[ 84.609910] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem📷wns=none[ 84.609913] vfio-pci 0000:03:00.0: can’t change power state from D3hot to D0 (config space inaccessible)[ 84.609973] pci 0000:03:00.0: Removing from iommu group 15[ 84.610007] vfio-pci 0000:03:00.1: can’t change power state from D3hot to D0 (config space inaccessible)[ 84.611607] pci 0000:03:00.1: Removing from iommu group 15

Please help.

I have a X570E ProArt which works fine.

Update: 1080 X 2 GPU (PCI 3) works fine strangely, one 4060 (first slot) and 1080 (second slot) also works fine

But as soon as the 4060 is placed in second slot, the above error comes again

What are your pcie settings? Is the first slot on auto or x8? Is the second slot on x8 or RAID?

Try setting the link speed manually.

Do both 4060s behave the same (if you swap them)?

Try switching off ‘allow d3’, it is somewhere deep in the settings but you can search for it in the bios (F9 I think)

HI,

All are X8 and GEN 4 manually selected in bios

I tried one setting and it works for now with is disable_d30 in vfio

Thanks