Intel B50 disappearing when trying out pcie passthrough

I’m trying to get pcie passthrough with Intel b50 on MPG X570 GAMING PLUS moba under Proxmox 9… and it’s super weird. Basically each time I do anything related to vfio modules the gpu disappears, no longer shows with lspci. To make it show again I have to physically power off the machine.

Any ideas what might be going on?

The steps I have taken to get pcie passthrough working:

  1. Made sure IOMMU works. It has turned out with 7.0.6-2-pve I didn’t have to add amd_iommu=on, as it was already working
  2. I’ve added conf entry for vfio, vfio_iommu_type1, vfio_pci in /etc/modules.d.
  3. Blacklisted xe by creating entry in /etc/modprobe.d/
  4. Added options vfio-pci ids=<IDs> disable_vga=1 in /etc/modprobe.d/vfio.conf - in my case the id was 8086:e212
  5. Did update-initramfs -u -k all and rebooted

… and basically that’s when “fun” started. The device no longer showed with lspci . It’s like anything that was on 12:00.0 was gone. Further reboots did not help. But complete shutdown (with power completely disconnected) did.

Sadly the B50 still has tendency to disappear - and any suggestions are greatly appreciated.

Some info on my system:

lspci | grep Intel

10:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01)
11:01.0 PCI bridge: Intel Corporation Device e2f0
11:02.0 PCI bridge: Intel Corporation Device e2f1
12:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
13:00.0 Audio device: Intel Corporation Device e2f7

lspci -n -s 12:00.0 -v


12:00.0 0300: 8086:e212 (prog-if 00 [VGA controller])
Subsystem: 8086:1114
Flags: bus master, fast devsel, latency 0, IOMMU group 12
Memory at fb000000 (64-bit, non-prefetchable) [size=16M]
Memory at 7800000000 (64-bit, prefetchable) [size=16G]
Expansion ROM at fc000000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express Endpoint, IntMsgNum 0
Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [d0] Power Management version 3
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [110] Null
Capabilities: [200] Address Translation Service (ATS)
Capabilities: [420] Physical Resizable BAR
Capabilities: [400] Latency Tolerance Reporting
Kernel driver in use: vfio-pci
Kernel modules: xe
```

Anything related to the B50 in dmesg and/or journalctl?

The B50 is very tolerant to binding/unbinding even with the xe driver loaded (at least on my system!); I haven’t needed to blacklist xe, bind to vfio on boot, etc. As long as no processes on the host are actively running on the B50 I can pass it through (full device of vf’s). So maybe try that (no blacklisting, vfio shenanigans)

dmesg | grep -i 'intel.*gpu\|xe'
[    0.000000] NX (Execute Disable) protection: active
[    0.000154] MTRR map: 6 entries (4 fixed + 2 variable; max 21), built from 9 variable MTRRs
[    0.000370] e820: update [mem 0xe0000000-0xffffffff] System RAM ==> device reserved
[    0.004569] [gap 0xe0000000-0xfe9fffff] available for PCI devices
[    0.346408] ... fixed-purpose counters:    0
[    0.346411] ... fixed-purpose bitmap:      0000000000000000
[    0.436380] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfcffffff window]
[    0.436991] pci 0000:00:01.2:   bridge window [io  0xd000-0xefff]
[    0.441244] pci 0000:20:00.0:   bridge window [io  0xd000-0xefff]
[    0.442078] pci 0000:21:04.0:   bridge window [io  0xe000-0xefff]
[    0.445919] pci 0000:26:00.0: BAR 0 [io  0xe050-0xe057]
[    0.445924] pci 0000:26:00.0: BAR 1 [io  0xe040-0xe043]
[    0.445930] pci 0000:26:00.0: BAR 2 [io  0xe030-0xe037]
[    0.445936] pci 0000:26:00.0: BAR 3 [io  0xe020-0xe023]
[    0.445942] pci 0000:26:00.0: BAR 4 [io  0xe000-0xe01f]
[    0.496181] pci 0000:21:04.0:   bridge window [io  0xe000-0xefff]
[    0.496302] pci 0000:20:00.0:   bridge window [io  0xd000-0xefff]
[    0.496329] pci 0000:00:01.2:   bridge window [io  0xd000-0xefff]
[    0.496405] pci_bus 0000:00: resource 9 [mem 0xe0000000-0xfcffffff window]
[    0.496447] pci_bus 0000:20: resource 0 [io  0xd000-0xefff]
[    0.496456] pci_bus 0000:21: resource 0 [io  0xd000-0xefff]
[    0.496464] pci_bus 0000:26: resource 0 [io  0xe000-0xefff]
[    0.916184] audit: type=1807 audit(1780333896.770:2): action=measure func=KEXEC_KERNEL_CHECK res=1
[    0.917116] evm: security.SMACK64EXEC
[    3.811468] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    4.149505] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 163840 ms ovfl timer
[    6.262959] snd_hda_intel 0000:30:00.1: bound 0000:30:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
[ 2459.253193]  dsl_pool_sync+0xe2/0x510 [zfs]
[ 3196.537529]  ? zio_nowait+0xe4/0x1e0 [zfs]
[ 3196.552824]  dsl_pool_sync+0xe2/0x510 [zfs]
[ 3196.560805]  ? zio_nowait+0xe4/0x1e0 [zfs]
[ 3196.570681]  ? zio_nowait+0xe4/0x1e0 [zfs]
[ 3196.594088]  ? zio_nowait+0xe4/0x1e0 [zfs]
[ 3196.604667]  ? zio_nowait+0xe4/0x1e0 [zfs]
[ 3196.616426]  ? zio_nowait+0xe4/0x1e0 [zfs]

and journalctl:

journalctl -k -S "today" | grep -iE 'xe|intel'
Jun 01 19:11:39 virt-domek kernel:   Intel GenuineIntel
Jun 01 19:11:39 virt-domek kernel: NX (Execute Disable) protection: active
Jun 01 19:11:39 virt-domek kernel: MTRR map: 6 entries (4 fixed + 2 variable; max 21), built from 9 variable MTRRs
Jun 01 19:11:39 virt-domek kernel: e820: update [mem 0xe0000000-0xffffffff] System RAM ==> device reserved
Jun 01 19:11:39 virt-domek kernel: [gap 0xe0000000-0xfe9fffff] available for PCI devices
Jun 01 19:11:39 virt-domek kernel: ... fixed-purpose counters:    0
Jun 01 19:11:39 virt-domek kernel: ... fixed-purpose bitmap:      0000000000000000
Jun 01 19:11:39 virt-domek kernel: pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfcffffff window]
Jun 01 19:11:39 virt-domek kernel: pci 0000:00:01.2:   bridge window [io  0xd000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci 0000:20:00.0:   bridge window [io  0xd000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci 0000:21:04.0:   bridge window [io  0xe000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci 0000:26:00.0: BAR 0 [io  0xe050-0xe057]
Jun 01 19:11:39 virt-domek kernel: pci 0000:26:00.0: BAR 1 [io  0xe040-0xe043]
Jun 01 19:11:39 virt-domek kernel: pci 0000:26:00.0: BAR 2 [io  0xe030-0xe037]
Jun 01 19:11:39 virt-domek kernel: pci 0000:26:00.0: BAR 3 [io  0xe020-0xe023]
Jun 01 19:11:39 virt-domek kernel: pci 0000:26:00.0: BAR 4 [io  0xe000-0xe01f]
Jun 01 19:11:39 virt-domek kernel: pci 0000:21:04.0:   bridge window [io  0xe000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci 0000:20:00.0:   bridge window [io  0xd000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci 0000:00:01.2:   bridge window [io  0xd000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci_bus 0000:00: resource 9 [mem 0xe0000000-0xfcffffff window]
Jun 01 19:11:39 virt-domek kernel: pci_bus 0000:20: resource 0 [io  0xd000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci_bus 0000:21: resource 0 [io  0xd000-0xefff]
Jun 01 19:11:39 virt-domek kernel: pci_bus 0000:26: resource 0 [io  0xe000-0xefff]
Jun 01 19:11:39 virt-domek kernel: audit: type=1807 audit(1780333896.770:2): action=measure func=KEXEC_KERNEL_CHECK res=1
Jun 01 19:11:39 virt-domek kernel: evm: security.SMACK64EXEC
Jun 01 19:11:39 virt-domek systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
Jun 01 19:11:40 virt-domek kernel: RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 163840 ms ovfl timer
Jun 01 19:11:40 virt-domek kernel: snd_hda_intel 0000:13:00.0: enabling device (0000 -> 0002)
Jun 01 19:11:40 virt-domek kernel: snd_hda_intel 0000:30:00.1: enabling device (0000 -> 0002)
Jun 01 19:11:40 virt-domek kernel: snd_hda_intel 0000:30:00.1: Handle vga_switcheroo audio client
Jun 01 19:11:40 virt-domek kernel: snd_hda_intel 0000:30:00.6: enabling device (0000 -> 0002)
Jun 01 19:11:40 virt-domek kernel: intel_rapl_common: Found RAPL domain package
Jun 01 19:11:40 virt-domek kernel: intel_rapl_common: Found RAPL domain core
Jun 01 19:11:42 virt-domek kernel: snd_hda_intel 0000:30:00.1: bound 0000:30:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
Jun 01 19:11:52 virt-domek kernel: pci 0000:13:00.0: deferred probe pending: snd_hda_intel: couldn't bind with audio component
Jun 01 19:52:34 virt-domek kernel:  dsl_pool_sync+0xe2/0x510 [zfs]
Jun 01 20:04:51 virt-domek kernel:  ? zio_nowait+0xe4/0x1e0 [zfs]
Jun 01 20:04:51 virt-domek kernel:  dsl_pool_sync+0xe2/0x510 [zfs]
Jun 01 20:04:51 virt-domek kernel:  ? zio_nowait+0xe4/0x1e0 [zfs]
Jun 01 20:04:51 virt-domek kernel:  ? zio_nowait+0xe4/0x1e0 [zfs]
Jun 01 20:04:51 virt-domek kernel:  ? zio_nowait+0xe4/0x1e0 [zfs]
Jun 01 20:04:52 virt-domek kernel:  ? zio_nowait+0xe4/0x1e0 [zfs]
Jun 01 20:04:52 virt-domek kernel:  ? zio_nowait+0xe4/0x1e0 [zfs]

On my end if I had no blacklisting the xe driver intercepted the device. But sure, will double check that.

oh, damn…. just found this topic - it seems I’m a bit unlucky with this one:

https://community.intel.com/t5/Graphics/Intel-B50-UEFI-boot-issue-on-X470-system/m-p/1733587#M148483

It seems exactly what I’m experiencing - warm reboots and gpu is gone on 5700G. On top of that I’ve indeed noticed the gpu, even though it’s not doing anything is a bit hotter than it should…

So the answer is - you need Vermeer or later CPU for b50

arghhhhhh….