Return to Level1Techs.com

Windows 10VM , GPU Passthrough -> PCI Header Error Nr .127

Hello there,

I’m new to this Forum so please be gentle :smiley:
So what’s the Problem here? I followed the “Wendells” Guide VFIO in 2019 using PopOS 18.04 and everything seems to fit until I reach the Point of starting my VM after the GPU is attached. At first it starts with no Output on the second Monitor, after I stop the VM (forced shutdown due to no VGA Output) the Virtual Machine Manager throws in “Unknown PCI header type ‘127’”. This can only be solved by restarting the Host. But the Problem of no Output remains. The NVIDIA GPU (Gainward 960GTX Phantom 4GB) is claimed by “vfio-pci” and only the second GPU (a very old HD3450) is used by GNOME. The NVIDIA GPU and it’s Audio Device are in the only Devices that are in IOMMU Group 15, so that seems also to work.

Did anyone encounter this or has a solution?

I have been fighting this issue for about a week, you need to patch and recompile the kernel with the pci patch

https://clbin.com/VCiYJ

This patch might work for you, ive only tested this is with a AMD GPU

2 Likes

I think there is a better version of this patch here on the forum somewhere. It might be nice to know more about the host system?

Hello there,

the Host System consists of the following Components:

  • AMD Ryzen 5 2600
  • ASROCK B450 Pro 4 Motherboard
  • 16GB GSkill DDR4
  • 240 GB NVME SSD for PopOS (Slot: M2_2 on Mainboard, as in Slot M2_1 PCIE4 would be deactivated)
  • 240 GB SATA SSD for VM (not used at the Moment, the VM Image is on the NVMe atm)
  • Gainward GTX960 Phantom 4GB (for passthrough, claimed by vfio) - PCIE2 Slot
  • very old HD3450 Passive cooled GPU used by Host System - PCIE4 Slot

The Gainward GTX960 ist in IOMMU Group 15 with it’s Audio Device…

using PopOS! 18.04 …

You probably have the acs patch which can be sketchy. Do you know if the bios for that board lists combopi 1.0.0.3m

I will take a look when my Kids let me :wink:

taken a look, newest bios (already applied) lists the following “Update AMD AGESA Combo-AM4 1.0.0.1”. I guess that is what you are refering to…

Just wanted to express my gratitude Here. Thanks to the PCI patch and recompiling the kernel, the GPU passthrough works Luke charm. OBS running with NVEnc in VM. Now I need to passthrough the USB controller in order to get more performance with the avermedia lgp lite… Again thank you both for helping me out…

1 Like

Has this or any similar patch been submitted to the kernel as far as you know?

1 Like

Yeah I’m also facing the same problem with:

OS: Fedora 30 / 5.1.18-300.fc30.x86_64
CPU: AMD RYZEN R7 1700
Motherboard: GA-AX370-Gaming-K7
GPU 01: (Host): NVIDIA GTX 1080
GPU 02: AMD R9 285/300

@wendell is this a patch we can push upstream?

ASRock has recently released a new BIOS v3.40 vs. the BIOS you mention that is v3.31, it purportedly only improves Matisse support, but may be worth a shot.

Your motherboard also has some AGESA updates that may help if you haven’t already tried updating your BIOS? Also be careful with kernel 5.1.1x as there was some regression regarding VFIO if memory serves me correctly. I would also recommend 5.2 in the interim.

I believe there are some new merges for the 5.3 kernel specifically relating to VirtIO-PMEM and VirtIO-IOMMU drivers.

Me2

OS: Fedora 30 / 5.1.17-300.fc30.x86_64
CPU: Ryzen R5 1600
Motherboard: GA-AB350N-GAMING WIFI with latest (available) Agesa (1.0.0.3 ab)
GPU: AMD Radeon HD 7870 GHZ

For all of u running Fedora, I made a Dockerfile to automatically compile the Fedora-Kernel with the patch enabled!

FROM fedora:30

ENV KVER kernel-5.1.17-300.fc30

RUN sudo dnf -y install fedpkg fedora-packager rpmdevtools ncurses-devel pesign
RUN rpmdev-setuptree && cd ~ && koji download-build --arch=src $KVER.src.rpm
RUN useradd -s /sbin/nologin mockbuild && rpm -Uvh ~/$KVER.src.rpm && cd ~/rpmbuild/SOURCES && curl -o agesa.patch.org https://clbin.com/VCiYJ
RUN cd ~/rpmbuild/SOURCES && echo 'From: Aiber Who <[email protected]>' > agesa.patch && echo 'Subject: [PATCH] agesa patch' >> agesa.patch && echo '' >> agesa.patch &&\ 
	cat agesa.patch.org >> agesa.patch && rm agesa.patch.org && cat agesa.patch
RUN cd ~/rpmbuild/SPECS/ && sudo dnf -y builddep kernel.spec
RUN cd ~/rpmbuild/SPECS/ && sed -i -e 's/# define buildid .local/%define buildid .agesa/g' kernel.spec && sed -i "s/# END OF PATCH DEFINITIONS/# END OF PATCH DEFINITIONS\nPatch9001: agesa.patch/" kernel.spec
RUN cd ~/rpmbuild/SPECS/ && rpmbuild -bb --without debug --target=x86_64 kernel.spec

ENTRYPOINT cp ~/rpmbuild/RPMS/x86_64/kernel* /output

To build the kernel just run sth like:

mkdir ~/custom-kernel && cd ~/custom-kernel
curl -L -o Dockerfile https://pastebin.com/raw/fPWv6NVe
docker buid -t custom/fedora-kernel . && mkdir out
docker --rm  -v ~/custom-kernel/out:/output custom/fedora-kernel
2 Likes

Hi,

I tried applying this patch when compiling a centos 7 kernel but i get errors in connection with virt.patch; the patch that I am attempting to apply (“1 out of 4 hunks FAILED - saving rejects to file drivers/pci/pci.c.rej”). This is my first kernel compilation attempt and I am way out of my depth. I followed the guide at https://wiki.centos.org/HowTos/Custom_Kernel#head-72952b3465e2fc1d84e08512448aa60c7a2e6ec2 and https://wiki.centos.org/HowTos/I_need_the_Kernel_Source. Does this have something to do with the fact that this patch is meant for Fedora kernel? Any suggestions?

[[email protected] pci]$ cat pci.c.rej

“— drivers/pci/pci.c
+++ drivers/pci/pci.c
@@ -1382,15 +1385,7 @@
pci_restore_config_space_range(pdev, 4, 9, 10, false);
pci_restore_config_space_range(pdev, 0, 3, 0, false);
} else if (pdev->hdr_type == PCI_HEADER_TYPE_BRIDGE) {
- pci_restore_config_space_range(pdev, 12, 15, 0, false);
-
- /*
- * Force rewriting of prefetch registers to avoid S3 resume
- * issues on Intel PCI bridges that occur when these
- * registers are not explicitly written.
- */
- pci_restore_config_space_range(pdev, 9, 11, 0, true);
- pci_restore_config_space_range(pdev, 0, 8, 0, false);
+ pci_restore_config_space_range(pdev, 0, 15, 0, true);
} else {
pci_restore_config_space_range(pdev, 0, 15, 0, false);
}”

[[email protected] SPECS]$ cat build-out.log

Building target platforms: x86_64
Building for target x86_64
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.hnniRh
Patch40000: virt.patch
1 out of 4 hunks FAILED – saving rejects to file drivers/pci/pci.c.rej

[[email protected] SPECS]$ cat build-err.log

"
+ umask 022
+ cd /home/user0/rpmbuild/BUILD
+ patch_command=‘patch -p1 -F1 -s’
+ cd /home/user0/rpmbuild/BUILD
+ rm -rf kernel-3.10.0-1062.1.1.el7
+ /usr/bin/mkdir -p kernel-3.10.0-1062.1.1.el7
+ cd kernel-3.10.0-1062.1.1.el7
+ /usr/bin/xz -dc /home/user0/rpmbuild/SOURCES/linux-3.10.0-1062.1.1.el7.tar.xz
+ /usr/bin/tar -xf -
+ STATUS=0
+ ‘[’ 0 -ne 0 ‘]’
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ mv linux-3.10.0-1062.1.1.el7 linux-3.10.0-1062.1.1.el7.centos7virtpatch09282019.x86_64
+ cd linux-3.10.0-1062.1.1.el7.centos7virtpatch09282019.x86_64
+ cp /home/user0/rpmbuild/SOURCES/kernel-3.10.0-ppc64-debug.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-ppc64.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-ppc64le-debug.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-ppc64le.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-s390x-debug.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-s390x-kdump.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-s390x.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-x86_64-debug.config /home/user0/rpmbuild/SOURCES/kernel-3.10.0-x86_64.config .
+ ApplyOptionalPatch virt.patch
+ local patch=virt.patch
+ shift
+ ‘[’ ‘!’ -f /home/user0/rpmbuild/SOURCES/virt.patch ‘]’
++ wc -l /home/user0/rpmbuild/SOURCES/virt.patch
++ awk ‘{print $1}’
+ local C=49
+ ‘[’ 49 -gt 9 ‘]’
+ ApplyPatch virt.patch
+ local patch=virt.patch
+ shift
+ ‘[’ ‘!’ -f /home/user0/rpmbuild/SOURCES/virt.patch ‘]’
+ case “$patch” in
+ patch -p1 -F1 -s
error: Bad exit status from /var/tmp/rpm-tmp.hnniRh (%prep)
Bad exit status from /var/tmp/rpm-tmp.hnniRh (%prep)"

Not too sure on compiling the kernel on CentOS, but i have compiled this patch with Arch, Ubuntu and Debian and had no issues. I will make a VM and see if i can compile this and ill let you know the result

Many thanks! Tried Redhat 8 and can’t get that to work either for vfio drivers do not load :frowning:

Ok, was able to patch and compile the kernel, I will see if i can create a script to do it and ill post it here when it works

Thanks a million. If there is anything that I can do, I’m glad to pitch in. It be a great learning experience for me so let me know if you want me to try something.

Ok so it downloads and patches the kernel but you need to run two commands and it should compile for you https://github.com/AcidzDevelopment/CentOS-header127/tree/master

make localconfig
make rpm-pkg

1 Like

Let me know if you have any issues with it

Great, looks like it complies successfully. So I take it that it has to be the 5.3.1 kernel.

Now that I have it though, can you please tell me where would i run these commands and get it so I can boot from it?

patching file drivers/pci/pci.c
Hunk #2 succeeded at 1412 (offset 30 lines).
Hunk #3 succeeded at 4657 (offset 26 lines).
Hunk #4 succeeded at 4672 (offset 26 lines).
[[email protected] CentOS-header127-master]$