Oh yeah the reason why I did VirtIO SCSI Single is because when you have SSD’s with the I/O thread and discard options you get better non-blocking performance because each connection to the drive is it’s own thread instead of a shared bus (or something like that).
I have my vm’s as ZVOLS.
2 Likes
Thanks for that!
Installing windows RN,
in the meantime here’s my current .conf pre GPU! Anything look off before I try to passthrough the card?
bios: ovmf
boot: dcn
bootdisk: sata0
cores: 12
efidisk0: local-lvm:vm-101-disk-1,size=4M
ide2: local:iso/Windows-10-0.iso,media=cdrom
machine: q35
memory: 65536
name: Windows
net0: virtio=DA:DC:B1:8E:70:09,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
sata0: local-lvm:vm-101-disk-0,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=7bb1afe7-f4e1-481f-93e9-7d2f6521d712
sockets: 1
vmgenid: 61ba34fc-7b54-4647-91ac-37a16d8d6691
It all looks good to me.
I highly recommend the virtio scsi disk adapter though, it’s significantly faster and lower latency, especially on older hardware. That’s just a performance tweak though.
(Removed previous post due to not having hit the reply button)
Is it possible to change it after having setup the machine? or is that a full fresh install?
Currently applying updates btw!
So you need to install the virtio scsi driver into windows. There’s also a way to tell windows to make sure the driver is loaded in the initial boot process, so it can access the disk properly.
I don’t know the exact process for that, since I usually just click “load additional drivers” in the installer and it handles all that automagically.
Someone with more Windows experience would have to help you there.
1 Like
So I managed to do this, but it was a year or two ago, so I might be shaky on the details.
First, create a new disk image, and give it to the VM using the virtio scsi bus.
Second, boot the VM, and load the drivers to enable the new disk, using device manager, same as any other windows driver.
Third, shut down the VM.
Fourth, change the main boot disk of the VM over to virtio
Fifth, boot the VM, this is the fail/pass point.
Sixth, shutdown the VM, remove and delete the temporary disk image.
2 Likes
Sounds good! will follow @TheCakeIsNaOH for how to switch that over!
I’ve now installed all applicable updates and just finished installing drivers for the gpu. So far I’m seeing the same behaviour as prior.
A display adapter is detected in device manager, and the driver correctly installed and detected the 5600xt. I still however get no display out.
Here’s the current .conf:
bios: ovmf
boot: cdn
bootdisk: sata0
cores: 12
efidisk0: local-lvm:vm-101-disk-1,size=4M
hostpci0: 0d:00
ide2: local:iso/Windows-10-0.iso,media=cdrom
machine: q35
memory: 65536
name: Windows
net0: virtio=DA:DC:B1:8E:70:09,bridge=vmbr0,firewall=1
net1: e1000=12:87:92:B3:98:F3,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
sata0: local-lvm:vm-101-disk-0,size=48G
scsihw: virtio-scsi-single
smbios1: uuid=7bb1afe7-f4e1-481f-93e9-7d2f6521d712
sockets: 1
vmgenid: 61ba34fc-7b54-4647-91ac-37a16d8d6691
after restarting the VM with the driver installed, System has soft CPU lock up
Hmmm,
Can you share the full host system specs? I’d like to get a better idea of the hardware we’re working with.
And, I’m assuming you did all the proper vfio-ifying of the GPU, right? (assigning vfio-pci driver to the gpu, and whatnot, so it’s not initialized) (lspci -knn will be helpful in determining this here)
1 Like
For Sure!
System is a Dual socket HPe ML350 G6 server
CPU: 2* Xeon 5670’s ([email protected])
Ram: 144GB of HPe Buffered ECC
Built in dual gigabit Lan + HP IPMI interface (ilo-2)
2*750W PSU’s
LSI PCIe Sas card connected to 6 HDD’s (33tb WD Red’s, 310TB seagates)
M.2 PCIe Card with 2 slots: one slot is a M.2sata to Sata adapter (connected to the native sata controller) and a 240GB NVMe from Adata using a silicon motion 2263 controller.
Rx5600xt gpu to be passed to windows VM with VFIO settings applied as per (https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/)
A few USB keyboard and mice to eventually be used for host and for the Windows VM
Due to PC RMRR implementation on the HP MB, a patch disabling the feature had to be compiled into the linux kernel. This was in addition to the Navi patch.
VT-d and VT-x are supported and working. All components are known good as of Saturday when I was running Esxi 6.7 as my hypervisor.
IOMMU groups are very good, without having to resort to any weird trickery.
Output of LSPCI -knn is:
00:00.0 Host bridge [0600]: Intel Corporation 5520 I/O Hub to ESI Port [8086:3406] (rev 13)
Subsystem: Hewlett-Packard Company ProLiant G6 series [103c:330b]
00:01.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 [8086:3408] (rev 13)
Kernel driver in use: pcieport
00:02.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 [8086:3409] (rev 13)
Kernel driver in use: pcieport
00:03.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 [8086:340a] (rev 13)
Kernel driver in use: pcieport
00:04.0 PCI bridge [0604]: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 [8086:340b] (rev 13)
Kernel driver in use: pcieport
00:05.0 PCI bridge [0604]: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 [8086:340c] (rev 13)
Kernel driver in use: pcieport
00:06.0 PCI bridge [0604]: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 [8086:340d] (rev 13)
Kernel driver in use: pcieport
00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 13)
Kernel driver in use: pcieport
00:08.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 [8086:340f] (rev 13)
Kernel driver in use: pcieport
00:09.0 PCI bridge [0604]: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 [8086:3410] (rev 13)
Kernel driver in use: pcieport
00:0a.0 PCI bridge [0604]: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 [8086:3411] (rev 13)
Kernel driver in use: pcieport
00:0d.0 Host bridge [0600]: Intel Corporation Device [8086:343a] (rev 13)
00:0d.1 Host bridge [0600]: Intel Corporation Device [8086:343b] (rev 13)
00:0d.2 Host bridge [0600]: Intel Corporation Device [8086:343c] (rev 13)
00:0d.3 Host bridge [0600]: Intel Corporation Device [8086:343d] (rev 13)
00:0d.4 Host bridge [0600]: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 [8086:3418] (rev 13)
00:0d.5 Host bridge [0600]: Intel Corporation 7500/5520/5500 Physical Layer Port 1 [8086:3419] (rev 13)
00:0d.6 Host bridge [0600]: Intel Corporation Device [8086:341a] (rev 13)
00:0e.0 Host bridge [0600]: Intel Corporation Device [8086:341c] (rev 13)
00:0e.1 Host bridge [0600]: Intel Corporation Device [8086:341d] (rev 13)
00:0e.2 Host bridge [0600]: Intel Corporation Device [8086:341e] (rev 13)
00:0e.3 Host bridge [0600]: Intel Corporation Device [8086:341f] (rev 13)
00:0e.4 Host bridge [0600]: Intel Corporation Device [8086:3439] (rev 13)
00:14.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] (rev 13)
Subsystem: Device [003c:000b]
Kernel driver in use: i7core_edac
Kernel modules: i7core_edac
00:14.1 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] (rev 13)
Subsystem: Device [003c:000b]
00:14.2 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423] (rev 13)
Subsystem: Device [003c:000b]
00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40]
Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 [8086:3a34]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) USB UHCI Controller [103c:330d]
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 [8086:3a35]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) USB UHCI Controller [103c:330d]
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 [8086:3a36]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) USB UHCI Controller [103c:330d]
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.3 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 [8086:3a39]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) USB UHCI Controller [103c:330d]
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 [8086:3a3a]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) USB2 EHCI Controller [103c:330d]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 90)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801JIB (ICH10) LPC Interface Controller [8086:3a18]
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 IDE interface [0101]: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1 [8086:3a20]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) 4 port SATA IDE Controller [103c:330d]
Kernel driver in use: ata_piix
Kernel modules: pata_acpi
00:1f.5 IDE interface [0101]: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2 [8086:3a26]
Subsystem: Hewlett-Packard Company 82801JI (ICH10 Family) 2 port SATA IDE Controller [103c:330d]
Kernel driver in use: ata_piix
Kernel modules: pata_acpi
01:03.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 [1002:515e] (rev 02)
Subsystem: Hewlett-Packard Company ES1000 [103c:31fb]
Kernel modules: radeonfb, radeon
01:04.0 System peripheral [0880]: Compaq Computer Corporation Integrated Lights Out Controller [0e11:b203] (rev 03)
Subsystem: Hewlett-Packard Company Integrated Lights Out Controller [103c:3305]
Kernel modules: hpwdt
01:04.2 System peripheral [0880]: Compaq Computer Corporation Integrated Lights Out Processor [0e11:b204] (rev 03)
Subsystem: Hewlett-Packard Company Integrated Lights Out Processor [103c:3305]
Kernel driver in use: hpilo
Kernel modules: hpilo
01:04.4 USB controller [0c03]: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller [103c:3300]
Subsystem: Hewlett-Packard Company iLO2 [103c:3305]
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
01:04.6 IPMI SMIC interface [0c07]: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface [103c:3302]
Subsystem: Hewlett-Packard Company iLO2 [103c:3305]
Kernel driver in use: ipmi_si
Kernel modules: ipmi_si
02:00.0 PCI bridge [0604]: Broadcom EPB PCI-Express to PCI-X Bridge [1166:0103] (rev b5)
03:04.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5715 Gigabit Ethernet [14e4:1678] (rev a3)
Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server Adapter [103c:703e]
Kernel driver in use: tg3
Kernel modules: tg3
03:04.1 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5715 Gigabit Ethernet [14e4:1678] (rev a3)
Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server Adapter [103c:703e]
Kernel driver in use: tg3
Kernel modules: tg3
05:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: IBM SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1014:03cb]
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas
08:00.0 Non-Volatile memory controller [0108]: Silicon Motion, Inc. Device [126f:2263] (rev 03)
Subsystem: Silicon Motion, Inc. Device [126f:2263]
Kernel driver in use: nvme
0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1478] (rev ca)
Kernel driver in use: pcieport
0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1479]
Kernel driver in use: pcieport
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [1002:731f] (rev ca)
Subsystem: Gigabyte Technology Co., Ltd Navi 10 [1458:231c]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
3e:00.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [103c:330c]
3e:00.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QuickPath Architecture System Address Decoder [103c:330c]
3e:02.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Link 0 [103c:330c]
3e:02.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Physical 0 [103c:330c]
3e:02.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Mirror Port Link 0 [103c:330c]
3e:02.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Mirror Port Link 1 [103c:330c]
3e:02.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Link 1 [103c:330c]
3e:02.5 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Physical 1 [103c:330c]
3e:03.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Registers [103c:330c]
3e:03.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Target Address Decoder [103c:330c]
3e:03.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller RAS Registers [103c:330c]
3e:03.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Test Registers [103c:330c]
3e:04.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Control [103c:330c]
3e:04.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Address [103c:330c]
3e:04.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [103c:330c]
3e:04.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [103c:330c]
3e:05.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Control [103c:330c]
3e:05.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Address [103c:330c]
3e:05.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [103c:330c]
3e:05.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [103c:330c]
3e:06.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Control [103c:330c]
3e:06.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Address [103c:330c]
3e:06.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [103c:330c]
3e:06.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [103c:330c]
3f:00.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [103c:330c]
3f:00.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QuickPath Architecture System Address Decoder [103c:330c]
3f:02.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Link 0 [103c:330c]
3f:02.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Physical 0 [103c:330c]
3f:02.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Mirror Port Link 0 [103c:330c]
3f:02.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Mirror Port Link 1 [103c:330c]
3f:02.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Link 1 [103c:330c]
3f:02.5 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series QPI Physical 1 [103c:330c]
3f:03.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Registers [103c:330c]
3f:03.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Target Address Decoder [103c:330c]
3f:03.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller RAS Registers [103c:330c]
3f:03.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Test Registers [103c:330c]
3f:04.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Control [103c:330c]
3f:04.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Address [103c:330c]
3f:04.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [103c:330c]
3f:04.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [103c:330c]
3f:05.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Control [103c:330c]
3f:05.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Address [103c:330c]
3f:05.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [103c:330c]
3f:05.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [103c:330c]
3f:06.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Control [103c:330c]
3f:06.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Address [103c:330c]
3f:06.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [103c:330c]
3f:06.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3] (rev 02)
Subsystem: Hewlett-Packard Company Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [103c:330c]I’m not familiar with this term. Is there a primer you can link me?
Okay, so we’re at the point where we’re going to need to try to dump the rom. Go ahead and reboot the host (we want the GPU fresh) and do the following from a root shell:
cd /sys/bus/pci/devices/0000:0d:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/5600xt-231c.bin
echo 0 > rom
ls -lh /usr/share/kvm/5600xt-231c.bin
Should be a few kb.
Any IO errors or anything of the likes, report back. Should be none.
lspci -knn looks good.
Here’s an RHL white paper on RMRR: https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf
it’s needed because of how IOMMU Group security was implemented on older hardware.
Here’s a tutorial for how to apply the kernel patch to get around it on older HW (mine included)
(https://forum.proxmox.com/threads/compile-proxmox-ve-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/)
the line:
cat rom > /usr/share/kvm/5600xt-231c.bin
Returns an input output error unfortunately
cat: rom: Input/output error
Dump is below, but file is empty
5600xt-231c.bin.zip (210 Bytes)
Restarting the host now
RMRR definitions associate a physical memory region with one or more
devices, so that when the IOMMU is initialized the device will continue to have access to the physical
memory at the original address.
sigh
grrrr… I’ve had issues with AMD cards doing this before. I was hoping this wouldn’t be the case here.
Might need to de-vfio it?
1 Like
Will try again after a fresh boot and see if that changes anything (but it shouldn’t right?)
After that will try the de-VFIO :’(
Just temporary to get the rom out
Just remove the device id definitions for the GPU. You don’t have to remove the vfio early load and whatnot.
after a fresh boot no effect unfortunately, still the same error :’(
For the device definition, do you mean in the vm config?
No the options vfio-pci.ids=Blabla
1 Like
Just did it and same error. will reset the system
It worked!!! 
1 Like
Nice. Now, if you include the rom file in your config and boot, it should work better.
1 Like