Linux (debian-ProxMox) Recompile needing over 60Gb (and counting) to compile

example?

I vaguely remember DMAR being an issue, and IIRC the solution was the kernel flag pcie_aspm=off

Right, so this is for libvirt. Which proxmox doesn’t use because “reasons”

So the important thing there is the qemu argument -set device.ua-sm2262.x-msix-relocation=bar2.

You can set command line args with theargs flag in the <vmnum>.conf file, in /etc/pve/nodes/<nodeid>/qemu-server/.

Either create or append to the args: flag in that conf file and you may have luck.

Just applied the ASPM flag, rebooting now. Assuming that works, will try the Qemu arg.

(BTW thanks so much! )

1 Like

Any time.

I’m not positive that the ASPM flag will fix the DMAR issues. It’s just what I experienced on my 1950x when the IOMMU was on.

1 Like

Am I right in thinking that I would also have to define the device in addition to passing the args to the Qemu server? (Also, ASPM flag worked! )

EDIT: yup, got this:

kvm: -set device.ua-sm2263.x-msix-relocation=bar2: there is no device "ua-sm2263" defined
TASK ERROR: start failed: QEMU exited with code 1

proxmox defines the device itself with hostpci${i}:

So, a GPU would look like this:

hostpci0: 00:05:00.0,pcie=1
hostpci1: 00:05:00.1,pcie=1

So I’m not sure what the alias (the ua-sm2263 segment of the argument) would be on proxmox, but if you notice in the XML, it’s setting an alias. I don’t think aliases are supported on proxmox.

So you might need to play around a bit with it. (or attach to the qemu console and pull some data from it, without the -set flag.

Gotcha!

Would I then switch out the args line to be

args: -set device.00:05.0.x-msix-relocation=bar2

Also, full .conf below

agent: 1

args: -set device.ua-sm2263.x-msix-relocation=bar2

balloon: 0

bootdisk: scsi0

cores: 4

hostpci0: 05:00.0

hostpci1: 08:00.0

ide2: local:iso/FreeNAS-11.3-U4.iso,media=cdrom

machine: q35

memory: 73728

name: FreeNAS

net0: virtio=DE:18:FD:90:5C:52,bridge=vmbr0,firewall=1

numa: 1

ostype: l26

scsi0: local-lvm:vm-100-disk-0,size=16G

scsihw: virtio-scsi-pci

smbios1: uuid=a3f26200-996d-48e4-a125-0abd750498e7

sockets: 1

startup: order=0,up=0

vmgenid: 1f348431-1ef7-4769-aa4c-2bbd20bf6761

Possibly. I’m honestly not sure.

What comes to mind would be either the pci address or the flag in the conf, so hostpci0

But again, I can’t say for sure.

Oh, also, add the ,pcie=1 to your pci definitions, otherwise they’ll run as old, slow PCI rather than PCIe!

yuh, now this is surprising,

changing to hostpci seems to have allowed it to progress further! It’s booting up now! (thanks for the tip on pcie btw, didn’t know that was a thing!)

Alright, so just finished booting up, but now none of the device are being shown in freenas

Export if LSPCi from the freenas shell is:


Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

[email protected][~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
1 Like

They don’t show in the PCI device tree?

Can you share the current config again?

Let’s see the device/vendor codes.

1 Like

100.conf:

agent: 1
args: -set device.hostpci1.x-msix-relocation=bar2
balloon: 0
bootdisk: scsi0
cores: 4
hostpci0: 05:00.0,pcie=1
hostpci1: 08:00.0,pcie=1
ide2: local:iso/FreeNAS-11.3-U4.iso,media=cdrom
machine: q35
memory: 73728
name: FreeNAS
net0: virtio=DE:18:FD:90:5C:52,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=a3f26200-996d-48e4-a125-0abd750498e7
sockets: 1
startup: order=0,up=0
vmgenid: 1f348431-1ef7-4769-aa4c-2bbd20bf6761

and here’s lspci -v

[email protected][~]# lspci -v
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: fast devsel, IRQ 255

00:01.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc. Device 1100
        Flags: fast devsel, IRQ 255
        Memory at fb000000 (32-bit, prefetchable)
        Memory at fea14000 (32-bit, non-prefetchable)
        Expansion ROM at fea00000 [disabled]

00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 16
        I/O ports at 6040

00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 17
        I/O ports at 6060

00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 18
        I/O ports at 6080

00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03) (prog-if 20 [EHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 19
        Memory at fea15000 (32-bit, non-prefetchable)

00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: fast devsel, IRQ 16
        Memory at fea10000 (32-bit, non-prefetchable)
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+

00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at fea16000 (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00005000-00005fff [size=4K]
        Memory behind bridge: fe800000-fe9fffff [size=2M]
        Prefetchable memory behind bridge: 00000000fce00000-00000000fcffffff [size=2M]
        Capabilities: [54] Express Root Port (Slot+), MSI 00
        Capabilities: [48] MSI-X: Enable- Count=1 Masked-
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Access Control Services

00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at fea17000 (32-bit, non-prefetchable)
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: fe600000-fe7fffff [size=2M]
        Prefetchable memory behind bridge: 00000000fcc00000-00000000fcdfffff [size=2M]
        Capabilities: [54] Express Root Port (Slot+), MSI 00
        Capabilities: [48] MSI-X: Enable- Count=1 Masked-
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Access Control Services

00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at fea18000 (32-bit, non-prefetchable)
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: fe400000-fe5fffff [size=2M]
        Prefetchable memory behind bridge: 00000000fca00000-00000000fcbfffff [size=2M]
        Capabilities: [54] Express Root Port (Slot+), MSI 00
        Capabilities: [48] MSI-X: Enable- Count=1 Masked-
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Access Control Services

00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at fea19000 (32-bit, non-prefetchable)
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: fe200000-fe3fffff [size=2M]
        Prefetchable memory behind bridge: 00000000fc800000-00000000fc9fffff [size=2M]
        Capabilities: [54] Express Root Port (Slot+), MSI 00
        Capabilities: [48] MSI-X: Enable- Count=1 Masked-
        Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Access Control Services

00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 16
        I/O ports at 60a0

00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 17
        I/O ports at 60c0

00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 18
        I/O ports at 60e0

00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) (prog-if 20 [EHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 19
        Memory at fea1a000 (32-bit, non-prefetchable)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 255
        Bus: primary=00, secondary=05, subordinate=09, sec-latency=0
        I/O behind bridge: 00001000-00004fff [size=16K]
        Memory behind bridge: fd800000-fe1fffff [size=10M]
        Prefetchable memory behind bridge: 00000000fc000000-00000000fc7fffff [size=8M]
        Capabilities: [50] Subsystem: Device 0000:0000

00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: fast devsel, IRQ 255

00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) (prog-if 01 [AHCI 1.0])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: bus master, fast devsel, latency 0, IRQ 16
        I/O ports at 6100
        Memory at fea1b000 (32-bit, non-prefetchable)
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [a8] SATA HBA v1.0

00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Flags: fast devsel, IRQ 16
        I/O ports at 0700

05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 21
        Memory at fe000000 (64-bit, non-prefetchable)
        Bus: primary=05, secondary=06, subordinate=06, sec-latency=0
        I/O behind bridge: 00004000-00004fff [size=4K]
        Memory behind bridge: fde00000-fdffffff [size=2M]
        Prefetchable memory behind bridge: 00000000fc600000-00000000fc7fffff [size=2M]
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 01
        Capabilities: [40] Hot-plug capable

05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 22
        Memory at fe001000 (64-bit, non-prefetchable)
        Bus: primary=05, secondary=07, subordinate=07, sec-latency=0
        I/O behind bridge: 00003000-00003fff [size=4K]
        Memory behind bridge: fdc00000-fddfffff [size=2M]
        Prefetchable memory behind bridge: 00000000fc400000-00000000fc5fffff [size=2M]
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 02
        Capabilities: [40] Hot-plug capable

05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 23
        Memory at fe002000 (64-bit, non-prefetchable)
        Bus: primary=05, secondary=08, subordinate=08, sec-latency=0
        I/O behind bridge: 00002000-00002fff [size=4K]
        Memory behind bridge: fda00000-fdbfffff [size=2M]
        Prefetchable memory behind bridge: 00000000fc200000-00000000fc3fffff [size=2M]
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 03
        Capabilities: [40] Hot-plug capable

05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 20
        Memory at fe003000 (64-bit, non-prefetchable)
        Bus: primary=05, secondary=09, subordinate=09, sec-latency=0
        I/O behind bridge: 00001000-00001fff [size=4K]
        Memory behind bridge: fd800000-fd9fffff [size=2M]
        Prefetchable memory behind bridge: 00000000fc000000-00000000fc1fffff [size=2M]
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [48] Slot ID: 0 slots, First+, chassis 04
        Capabilities: [40] Hot-plug capable

06:05.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
        Subsystem: Red Hat, Inc. Device 0008
        Flags: bus master, fast devsel, latency 0, IRQ 22
        I/O ports at 4000
        Memory at fde40000 (32-bit, non-prefetchable)
        Memory at fc600000 (64-bit, prefetchable)
        Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg

06:08.0 Communication controller: Red Hat, Inc. Virtio console
        Subsystem: Red Hat, Inc. Device 0003
        Flags: bus master, fast devsel, latency 0, IRQ 21
        I/O ports at 4040
        Memory at fde41000 (32-bit, non-prefetchable)
        Memory at fc604000 (64-bit, prefetchable)
        Capabilities: [98] MSI-X: Enable- Count=2 Masked-
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg

06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
        Subsystem: Red Hat, Inc. Device 0001
        Flags: bus master, fast devsel, latency 0, IRQ 23
        I/O ports at 4080
        Memory at fde42000 (32-bit, non-prefetchable)
        Memory at fc608000 (64-bit, prefetchable)
        Expansion ROM at fde00000 [disabled]
        Capabilities: [98
1 Like

Maybe pcie=1 isn’t helping? :thinking:

Not sure what those QEMU PCI-PCI bridge devices are.

1 Like

Had a similar hunch! trying that now

In the meantime, I went to reconfigure my windows vm (12 cores, 72gb ram, with numa option selected) with a 5600xt connected. I havent been getting a display out of it yet. changed it to have pcie=1 in the .conf, and the entire host crashed, with the following being sent to my ssh shell:

 kernel:[ 2377.418483] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kvm:7561]

Message from [email protected] at Jul 28 15:03:12 ...
 kernel:[ 2381.462540] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [qm:7813]

Message from [email protected] at Jul 28 15:03:28 ...
 kernel:[ 2397.462753] watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [pvesr:7831]

Message from [email protected] at Jul 28 15:03:36 ...
 kernel:[ 2405.418852] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kvm:7561]

Message from [email protected] at Jul 28 15:03:40 ...
 kernel:[ 2409.462903] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [qm:7813]

Message from [email protected] at Jul 28 15:03:41 ...
 kernel:[ 2411.116063] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0

Message from [email protected] at Jul 28 15:03:56 ...
 kernel:[ 2425.463088] watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [pvesr:7831]

Message from [email protected] at Jul 28 15:04:04 ...
 kernel:[ 2433.419173] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kvm:7561]

I’m restarting the system now, will keep the windows vm off in the meantime .

so maybe pcie=1 is definitely not something we want.

That’s directly contrary to the proxmox documentation.

Yeah! so removing pcie=1 did the trick!

All devices showing up properly in the freenas VM

1 Like

Nice! I’m still fairly confused about that one, but glad it’s working now.

1 Like

Yeah! now to figure out the windows vm xD

it’s the one that crashes the entire host when I try to start it with PCIe=1.

Other issue is that it won’t detect any displays attached to the gpu (5600xt, with the patch applied)

I’ve installed the and windows driver, and it does detect the card correctly. However it will not detect any attached displays. tried both HDMI and an active DP->DVI, with neither doing anything

GPU, cables and monitor are known good (where working in an ESXi windows vm a few days ago)

the .conf for the vm is:

bootdisk: ide0
cores: 12
hostpci0: 0d:00
ide0: local-lvm:vm-101-disk-0,size=48G
ide2: local:iso/Windows-10-0.iso,media=cdrom
machine: q35
memory: 65536
name: windows
net0: e1000=D6:98:2F:41:01:B3,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=ee5160e6-e116-42ae-9575-a9e25648ad91
sockets: 1
vmgenid: a46c4555-550e-422f-a831-1af24d1a93bc

Does that GPU have a separate HDMI audio device?

all GPUs I’ve ever seen have had a 00.0 for the GPU, and a 00.1 for the audio.

You need to pass both together or it won’t work properly.

I passed the “All functions” flag to it via the gui, will reconfigure as a separate device since thats the recommendation I often see