example?
I vaguely remember DMAR being an issue, and IIRC the solution was the kernel flag pcie_aspm=off
Right, so this is for libvirt. Which proxmox doesn’t use because “reasons”
So the important thing there is the qemu argument -set device.ua-sm2262.x-msix-relocation=bar2.
You can set command line args with theargs flag in the <vmnum>.conf file, in /etc/pve/nodes/<nodeid>/qemu-server/.
Either create or append to the args: flag in that conf file and you may have luck.
Just applied the ASPM flag, rebooting now. Assuming that works, will try the Qemu arg.
(BTW thanks so much! )
1 Like
Any time.
I’m not positive that the ASPM flag will fix the DMAR issues. It’s just what I experienced on my 1950x when the IOMMU was on.
1 Like
Am I right in thinking that I would also have to define the device in addition to passing the args to the Qemu server? (Also, ASPM flag worked! )
EDIT: yup, got this:
kvm: -set device.ua-sm2263.x-msix-relocation=bar2: there is no device "ua-sm2263" defined
TASK ERROR: start failed: QEMU exited with code 1
proxmox defines the device itself with hostpci${i}:
So, a GPU would look like this:
hostpci0: 00:05:00.0,pcie=1
hostpci1: 00:05:00.1,pcie=1
So I’m not sure what the alias (the ua-sm2263 segment of the argument) would be on proxmox, but if you notice in the XML, it’s setting an alias. I don’t think aliases are supported on proxmox.
So you might need to play around a bit with it. (or attach to the qemu console and pull some data from it, without the -set flag.
Gotcha!
Would I then switch out the args line to be
args: -set device.00:05.0.x-msix-relocation=bar2
Also, full .conf below
agent: 1
args: -set device.ua-sm2263.x-msix-relocation=bar2
balloon: 0
bootdisk: scsi0
cores: 4
hostpci0: 05:00.0
hostpci1: 08:00.0
ide2: local:iso/FreeNAS-11.3-U4.iso,media=cdrom
machine: q35
memory: 73728
name: FreeNAS
net0: virtio=DE:18:FD:90:5C:52,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=a3f26200-996d-48e4-a125-0abd750498e7
sockets: 1
startup: order=0,up=0
vmgenid: 1f348431-1ef7-4769-aa4c-2bbd20bf6761Possibly. I’m honestly not sure.
What comes to mind would be either the pci address or the flag in the conf, so hostpci0
But again, I can’t say for sure.
Oh, also, add the ,pcie=1 to your pci definitions, otherwise they’ll run as old, slow PCI rather than PCIe!
yuh, now this is surprising,
changing to hostpci seems to have allowed it to progress further! It’s booting up now! (thanks for the tip on pcie btw, didn’t know that was a thing!)
Alright, so just finished booting up, but now none of the device are being shown in freenas
Export if LSPCi from the freenas shell is:
Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.
root@freenas[~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
1 Like
They don’t show in the PCI device tree?
Can you share the current config again?
Let’s see the device/vendor codes.
1 Like
100.conf:
agent: 1
args: -set device.hostpci1.x-msix-relocation=bar2
balloon: 0
bootdisk: scsi0
cores: 4
hostpci0: 05:00.0,pcie=1
hostpci1: 08:00.0,pcie=1
ide2: local:iso/FreeNAS-11.3-U4.iso,media=cdrom
machine: q35
memory: 73728
name: FreeNAS
net0: virtio=DE:18:FD:90:5C:52,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=a3f26200-996d-48e4-a125-0abd750498e7
sockets: 1
startup: order=0,up=0
vmgenid: 1f348431-1ef7-4769-aa4c-2bbd20bf6761
and here’s lspci -v
root@freenas[~]# lspci -v
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: fast devsel, IRQ 255
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller])
Subsystem: Red Hat, Inc. Device 1100
Flags: fast devsel, IRQ 255
Memory at fb000000 (32-bit, prefetchable)
Memory at fea14000 (32-bit, non-prefetchable)
Expansion ROM at fea00000 [disabled]
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03) (prog-if 00 [UHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at 6040
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03) (prog-if 00 [UHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 17
I/O ports at 6060
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03) (prog-if 00 [UHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 18
I/O ports at 6080
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03) (prog-if 20 [EHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at fea15000 (32-bit, non-prefetchable)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: fast devsel, IRQ 16
Memory at fea10000 (32-bit, non-prefetchable)
Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fea16000 (32-bit, non-prefetchable)
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00005000-00005fff [size=4K]
Memory behind bridge: fe800000-fe9fffff [size=2M]
Prefetchable memory behind bridge: 00000000fce00000-00000000fcffffff [size=2M]
Capabilities: [54] Express Root Port (Slot+), MSI 00
Capabilities: [48] MSI-X: Enable- Count=1 Masked-
Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Access Control Services
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fea17000 (32-bit, non-prefetchable)
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: None
Memory behind bridge: fe600000-fe7fffff [size=2M]
Prefetchable memory behind bridge: 00000000fcc00000-00000000fcdfffff [size=2M]
Capabilities: [54] Express Root Port (Slot+), MSI 00
Capabilities: [48] MSI-X: Enable- Count=1 Masked-
Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Access Control Services
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fea18000 (32-bit, non-prefetchable)
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: None
Memory behind bridge: fe400000-fe5fffff [size=2M]
Prefetchable memory behind bridge: 00000000fca00000-00000000fcbfffff [size=2M]
Capabilities: [54] Express Root Port (Slot+), MSI 00
Capabilities: [48] MSI-X: Enable- Count=1 Masked-
Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Access Control Services
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fea19000 (32-bit, non-prefetchable)
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: None
Memory behind bridge: fe200000-fe3fffff [size=2M]
Prefetchable memory behind bridge: 00000000fc800000-00000000fc9fffff [size=2M]
Capabilities: [54] Express Root Port (Slot+), MSI 00
Capabilities: [48] MSI-X: Enable- Count=1 Masked-
Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Access Control Services
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) (prog-if 00 [UHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at 60a0
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) (prog-if 00 [UHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 17
I/O ports at 60c0
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) (prog-if 00 [UHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 18
I/O ports at 60e0
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) (prog-if 20 [EHCI])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at fea1a000 (32-bit, non-prefetchable)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 255
Bus: primary=00, secondary=05, subordinate=09, sec-latency=0
I/O behind bridge: 00001000-00004fff [size=16K]
Memory behind bridge: fd800000-fe1fffff [size=10M]
Prefetchable memory behind bridge: 00000000fc000000-00000000fc7fffff [size=8M]
Capabilities: [50] Subsystem: Device 0000:0000
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: fast devsel, IRQ 255
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) (prog-if 01 [AHCI 1.0])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at 6100
Memory at fea1b000 (32-bit, non-prefetchable)
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [a8] SATA HBA v1.0
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Flags: fast devsel, IRQ 16
I/O ports at 0700
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 21
Memory at fe000000 (64-bit, non-prefetchable)
Bus: primary=05, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: 00004000-00004fff [size=4K]
Memory behind bridge: fde00000-fdffffff [size=2M]
Prefetchable memory behind bridge: 00000000fc600000-00000000fc7fffff [size=2M]
Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [48] Slot ID: 0 slots, First+, chassis 01
Capabilities: [40] Hot-plug capable
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 22
Memory at fe001000 (64-bit, non-prefetchable)
Bus: primary=05, secondary=07, subordinate=07, sec-latency=0
I/O behind bridge: 00003000-00003fff [size=4K]
Memory behind bridge: fdc00000-fddfffff [size=2M]
Prefetchable memory behind bridge: 00000000fc400000-00000000fc5fffff [size=2M]
Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [48] Slot ID: 0 slots, First+, chassis 02
Capabilities: [40] Hot-plug capable
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 23
Memory at fe002000 (64-bit, non-prefetchable)
Bus: primary=05, secondary=08, subordinate=08, sec-latency=0
I/O behind bridge: 00002000-00002fff [size=4K]
Memory behind bridge: fda00000-fdbfffff [size=2M]
Prefetchable memory behind bridge: 00000000fc200000-00000000fc3fffff [size=2M]
Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [48] Slot ID: 0 slots, First+, chassis 03
Capabilities: [40] Hot-plug capable
05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 20
Memory at fe003000 (64-bit, non-prefetchable)
Bus: primary=05, secondary=09, subordinate=09, sec-latency=0
I/O behind bridge: 00001000-00001fff [size=4K]
Memory behind bridge: fd800000-fd9fffff [size=2M]
Prefetchable memory behind bridge: 00000000fc000000-00000000fc1fffff [size=2M]
Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [48] Slot ID: 0 slots, First+, chassis 04
Capabilities: [40] Hot-plug capable
06:05.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
Subsystem: Red Hat, Inc. Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 22
I/O ports at 4000
Memory at fde40000 (32-bit, non-prefetchable)
Memory at fc600000 (64-bit, prefetchable)
Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
Capabilities: [70] Vendor Specific Information: VirtIO: Notify
Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
Capabilities: [50] Vendor Specific Information: VirtIO: ISR
Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
06:08.0 Communication controller: Red Hat, Inc. Virtio console
Subsystem: Red Hat, Inc. Device 0003
Flags: bus master, fast devsel, latency 0, IRQ 21
I/O ports at 4040
Memory at fde41000 (32-bit, non-prefetchable)
Memory at fc604000 (64-bit, prefetchable)
Capabilities: [98] MSI-X: Enable- Count=2 Masked-
Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
Capabilities: [70] Vendor Specific Information: VirtIO: Notify
Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
Capabilities: [50] Vendor Specific Information: VirtIO: ISR
Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
Subsystem: Red Hat, Inc. Device 0001
Flags: bus master, fast devsel, latency 0, IRQ 23
I/O ports at 4080
Memory at fde42000 (32-bit, non-prefetchable)
Memory at fc608000 (64-bit, prefetchable)
Expansion ROM at fde00000 [disabled]
Capabilities: [98
1 Like
Maybe pcie=1 isn’t helping? 
Not sure what those QEMU PCI-PCI bridge devices are.
1 Like
Had a similar hunch! trying that now
In the meantime, I went to reconfigure my windows vm (12 cores, 72gb ram, with numa option selected) with a 5600xt connected. I havent been getting a display out of it yet. changed it to have pcie=1 in the .conf, and the entire host crashed, with the following being sent to my ssh shell:
kernel:[ 2377.418483] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kvm:7561]
Message from syslogd@pve at Jul 28 15:03:12 ...
kernel:[ 2381.462540] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [qm:7813]
Message from syslogd@pve at Jul 28 15:03:28 ...
kernel:[ 2397.462753] watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [pvesr:7831]
Message from syslogd@pve at Jul 28 15:03:36 ...
kernel:[ 2405.418852] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kvm:7561]
Message from syslogd@pve at Jul 28 15:03:40 ...
kernel:[ 2409.462903] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [qm:7813]
Message from syslogd@pve at Jul 28 15:03:41 ...
kernel:[ 2411.116063] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
Message from syslogd@pve at Jul 28 15:03:56 ...
kernel:[ 2425.463088] watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [pvesr:7831]
Message from syslogd@pve at Jul 28 15:04:04 ...
kernel:[ 2433.419173] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kvm:7561]
I’m restarting the system now, will keep the windows vm off in the meantime .
so maybe pcie=1 is definitely not something we want.
That’s directly contrary to the proxmox documentation.
Yeah! so removing pcie=1 did the trick!
All devices showing up properly in the freenas VM
1 Like
Nice! I’m still fairly confused about that one, but glad it’s working now.
1 Like
Yeah! now to figure out the windows vm xD
it’s the one that crashes the entire host when I try to start it with PCIe=1.
Other issue is that it won’t detect any displays attached to the gpu (5600xt, with the patch applied)
I’ve installed the and windows driver, and it does detect the card correctly. However it will not detect any attached displays. tried both HDMI and an active DP->DVI, with neither doing anything
GPU, cables and monitor are known good (where working in an ESXi windows vm a few days ago)
the .conf for the vm is:
bootdisk: ide0
cores: 12
hostpci0: 0d:00
ide0: local-lvm:vm-101-disk-0,size=48G
ide2: local:iso/Windows-10-0.iso,media=cdrom
machine: q35
memory: 65536
name: windows
net0: e1000=D6:98:2F:41:01:B3,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=ee5160e6-e116-42ae-9575-a9e25648ad91
sockets: 1
vmgenid: a46c4555-550e-422f-a831-1af24d1a93bcDoes that GPU have a separate HDMI audio device?
all GPUs I’ve ever seen have had a 00.0 for the GPU, and a 00.1 for the audio.
You need to pass both together or it won’t work properly.
