[SOLVED] Cannot resolve Nvidia Code:43 error in VFIO

Changed platform recently, Intel > AMD, and had a fresh install of Manjaro:

Manjaro Lycia 20 64bit
KDE Plasma 5.19.3
Qt: 5.15.0
Kernel: 5.7.9-1-MANJARO
qemu: 5.0.0-7
libvirt: 6.4.0-1
virt-manager: 2.2.1-2

Then I started setting up my Win10 VM with passtrhough. I had some issues with the AMD and had to make some changes, but I have it working. What it doesn’t work, although I have already done the necessary changes to hide the KVM status, is my 1080ti. Whatever I try, I am getting the “Code: 43” error. Only when I first installed Windows, with the default driver, it looked it was working, but after the first reboot, it failed.

Also, I am using 2 VGAs, not a IGP like with Intel, and Host VGA is at PCIe2, while 1080ti is at PCIe1. That cause some issues at start, described here, but they seem to be over.

Below are my settings. I also passing through my 2nd LAN adapter (the 3rd ID).

EVGA GTX 1080ti (id: 10de:1b06,10de:10ef)

GRUB:

set gfxpayload=auto
insmod gzio
insmod part_gpt
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root  2c8e84da-b359-4d2a-8eb3-3c53ee14eca3
else
  search --no-floppy --fs-uuid --set=root 2c8e84da-b359-4d2a-8eb3-3c53ee14eca3
fi
linux	/boot/vmlinuz-5.7-x86_64 root=UUID=2c8e84da-b359-4d2a-8eb3-3c53ee14eca3 rw  quiet apparmor=1 security=apparmor resume=UUID=855cc007-54fd-4ea0-8dcd-8db736df2099 udev.log_priority=3 amd_iommu=on iommu=pt hugepages=16384 vfio-pci.ids=10de:1b06,10de:10ef,8086:1539 systemd.unified_cgroup_hierarchy=1
initrd	/boot/amd-ucode.img /boot/initramfs-5.7-x86_64.img

mkinitcpio.conf

MODULES="crc32c vfio_pci vfio vfio_iommu_type1 vfio_virqfd"
HOOKS="base udev autodetect modconf block keyboard keymap resume filesystems"

KVM XML:

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>win10-games</name>
  <uuid>b4cb6295-504f-4d61-829a-7663793e4db9</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">33554432</memory>
  <currentMemory unit="KiB">33554432</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static">18</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-games_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="off"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <vendor_id state="on" value="0123456789ab"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" dies="1" cores="9" threads="2"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/wizard/Data/Linux/Manjaro/virtio-win-0.1.160.iso"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/sdc"/>
      <target dev="sdc" bus="sata"/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/sdd"/>
      <target dev="sdd" bus="sata"/>
      <address type="drive" controller="1" bus="0" target="0" unit="1"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="sata" index="1">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x01" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </controller>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="2"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on" file="/home/wizard/Downloads/Linux/EVGA.GTX1080Ti.11264.170418.rom"/>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0b" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1b1c"/>
        <product id="0x1b81"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="4"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="host,topoext=on,invtsc=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-frequencies,host-cache-info=on,l3-cache=off,-amd-stibp"/>
  </qemu:commandline>
</domain>

Any help would be appreciated… :slight_smile:

I believe it might be because you didn’t specify the driver for video cards audio passthrough. I would update it to:

<hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0b" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>

The only other things i noticed different from my config is that you’re passing in a rom for the video card and i’m not and my hyperv block is a bit simpler

<hyperv>
  <relaxed state="on"/>
  <vapic state="on"/>
  <spinlocks state="on" retries="8191"/>
  <vendor_id state="on" value="whatever"/>
</hyperv>

Thank you for taking time to answer.
The ROM is a result of (literally) 100nds of tests in order to make it work. I tried with no ROM, official ROM, dumped ROM, no-headers ROM, and counting…
Today I got an update to qemu 5.0.0-8 and I am going to try this too, along with your suggestion for the HDMI sound.

Edit: I tried to add the vfio driver to audio too, with ROM, without ROM and with the simplified hyperv. All these changes, after i updated to qemu 5.0.0-8.
None worked…

Hey again!

I’m sorry to hear that the damned code 43 is popping up! :frowning_face:

I had the same problem - It cleared after adding

<vendor_id state=“on” value=“0123456789ab”/>

I know it must be frustrating to read archwiki time and time again but lets start from the VM installation:
Did you install win10 first without the gpu passed through as described in the " Setting up an OVMF-based guest VM" archwiki article?
After fresh win10 install pass the gpu into the VM and install drivers (I went the geforce experience route)

Good luck - I was able to get mine working following closely archwiki.

I have done so many tests and changes while trying to make it work, and I don’t remember.
Since my setup is not working anyway, I deleted the VM, and start all over. Followed the WiKi step by step and still have the same issue…
Please note here, that I had it working before without all that… Just the simple process of creating a passthrough VGA.

Damn…

Please note here, that I had it working before without all that… Just the simple process of creating a passthrough VGA.

What has changed? Can you recall the steps that you made when it worked?

I see you have spice/qxl video enabled inside your xml-file. Have you tried removing the spice/qxl video after you have passed the GPU? Iirc that can also cause problems.

Could you tell me what are these

<vapic state="on"/>
<vpindex state="on"/>
<synic state="on"/>
<stimer state="on"/>
<frequencies state="on"/>

in the xml-file inside “hyperv” tag? A quick googling on “vapic” lead me to a post from 2014 talking about qemu version 1.0… Are they necessary?
I remember adding only the

<vendor_id state="on" value="0123456789ab"/>

to my xml-file inside “hyperv” tag

Also could you tell me a bit more about this line inside the xml-file

<qemu:arg value="host,topoext=on,invtsc=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-frequencies,host-cache-info=on,l3-cache=off,-amd-stibp"/>

From my understanding you seem to be “calling” some things twice (Inside libvirt xml-file and qemu commandline)

Good luck! I really hope you get it working :slightly_smiling_face:

What changed is the platform, Intel > AMD. Intel had an IGP, but Ryzen 9 needs a 2nd PCIe card.
As for the QXL:

  • previously I had it working cause I need it
  • tried removing it, didn’t make any difference

Now for the qemu arguments, I am not sure how I ended up there, but I had to add them to make R9 work. I had BSoD before adding these lines.

I have been making changes to this file, even deleting it and edit from scratch, for over 10 days now. The only parts I have from my old working XML, are the two SSDs, cause I pass them through. Most changes in that file were recommended from someone here or Reddit, to solve a problem I was having, I am not aware for all of them what they do.

However, I removed vapic, vpindex, synic, stimer and freq for my last test after your suggestion. Still same issue.

However, I removed vapic, vpindex, synic, stimer and freq for my last test after your suggestion. Still same issue.

:frowning: I’m starting to get a little out of depth here - I’m just a wiki reading bozo :nerd_face:

From what I can see you’ve got all of the steps right but I/we just can’t seem to figure out what needs to be done for that final step…

We need to summon higher powers! :mage:

Who? I have this thread and on Reddit going around for weeks, and less than 5 people helped…

1 Like

Until higher powers help, maybe check a couple of basics?

The VM does output on the GPU’s monitor, even with code 43?
Then the card is being passed through, just need to make nvidia not detect the virtual environment.

The vendor is needs to be 12 charachters or less. It is 12, but maybe take a charachter or two off?

If there is output on the GPU’s monitor, then remove any other video or graphical option from the list in VMM.

Are there any errors on dmesg when The vm or run?

The rom should not be needed, and you did mention about it working before, but I would try again without.

Lastly, just to make sure, both the GPU and it’s audio is being isolated as per lspci -knn or -knv or whatever?

1 Like

Thank you for taking time to reply…

No, there is no output in the DP port where I have my monitor connected, goes to sleep with “No signal” message
I tried multiple vendors from “ab1234567890”, “1234567890”, “012345”, etc.
Although I need the QXL/VGA/Vfio cards, I tried removing them and ended in a screen showing the 4 UEFI bios lines.
The only error I was getting, while trying to pass the ROM file, was “error accessing…”. I was trying to change permissions, just to make sure that if it is not accessible, I will get an error, which I did
It was working before without the ROM. I tried with the ROM on this setup, while trying to make it work.
Yes, both devices are passed through, both IDs are on kernel parameters and “/etc/modprobe.d/vfio.conf” file

4 things are different from my previously working setup:

  1. Intel > AMD, because of no IGP, I had to install a 2nd VGA on PCIe2, with my Gaming/Guest VGA on PCIe1.
  2. Passthtough onboard LAN, instead of PCI. Separate IOMMU group, shouldn’t affect. Tried removing it, didn’t help
  3. Windows 2004. I also tried to setup 1903 and 1909. Same problem
  4. NVidia drivers: Tried with a year old version, 422.xx. Didn’t help.

Also, in order to make Ryzen9 work, I was getting “KERNEL CHECK” BSoD, I had to add this in my XML file:

  <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="host,topoext=on,invtsc=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-frequencies,host-cache-info=on,l3-cache=off,-amd-stibp"/>
  </qemu:commandline>

No, there is no output in the DP port where I have my monitor connected, goes to sleep with “No signal” message

Oh - You should see an output from the passed through GPU when you start the VM. Also the gpu must be connected to a monitor when booting the vm to initialize the gpu.

Although I need the QXL/VGA/Vfio cards, I tried removing them and ended in a screen showing the 4 UEFI bios lines.

Can you navigate the UEFI to select the installed win10. This situation is described in archwiki

it is possible you will be dropped into the UEFI menu instead of starting the installation upon powering your VM for the first time. Sometimes the correct ISO file was not automatically detected and you will need to manually specify the drive to boot. By typing exit and navigating to “boot manager” you will enter a menu that allows you to choose between devices.

Actually I do… After so many tests and tries, I don’t even remember what I already did…
So, when I boot, I see the 4 UEFI lines on both monitors (DP port & Virtual screen), but as soon as the virtual screen displays the Windows login screen, the DP, which is connected to 1080, remains displaying the 4 UEFI lines along with the “TianoCore” logo.

as soon as the virtual screen displays the Windows login screen, the DP, which is connected to 1080, remains displaying the 4 UEFI lines along with the “TianoCore” logo.

This seems to be the problem. Your gpu is not initializing/binding correctly - You need to be able to see the windows screen from the monitor that the 1080 is connected to.

If you remove the “Virtual screen” and try to boot the VM do you see the windows screen? or is it still stuck on the “tianocore” logo?

Stuck in the logo…
But, how is supposed to display Windows screen if the driver is not loaded?

If it cannot load the Nvidia driver, it reverts back to built in “basic driver” and still shows on nvidia card, but less performant than with nvidia drivers.

Then, when you click device manager, you see the error next to the video adapter, with error, exit code 43.

What does the guest windows device manager show for the graphics card?

Um, that was not supposed to be so sarcastic/patronising…

DevMgr shows “Nvidia GTX 1080ti” with the exclamation mark. When you go to properties, you can see the “Code:43” error.
When you setup W10 for the first time, and loads MS drivers, looks like it is ok. However, after the 1st reboot, you get the error. I am removing the drivers with DDU, prevent Windows for loading the one from Windows update and tried 10-12 different drivers from Nvidia and Guru3d sites.

1 Like

Could you do a fresh start - Install a fresh win10 (I’d go with the newest version 20.04?) on your ssd(s) without the passthrough only adding the necessary things to your xml-file such as:

<vendor_id state=“on” value=“0123456789ab”/>

and

 <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="host,topoext=on,invtsc=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-frequencies,host-cache-info=on,l3-cache=off,-amd-stibp"/>
  </qemu:commandline>

After a fresh install remove the virtual screen stuff (spice channel and virtual display, the QXL video adapter,) add the GPU and boot. You should be able to land on the windows desktop with the monitor connected to your GPU showing it. If it works (there should be no code 43) - try to install the nvidia drivers with “nvidia geforce experience”

If that fails - try it again with the same steps but w/o adding the

<vendor_id state=“on” value=“0123456789ab”/>

to your xml-file. Only add it after booting into win10 with the GPU passed through and shutting the VM down (You should see code 43 this way - but it should go away after adding the vendor_id state)

Okay, cool, looks like the device is being isolated from the host, and is being passed through fine. Like Pollo is saying, it’s more a case of Nvidia detecting the VM and not playing ball, but looks like you are doing pretty much the right things. :frowning:

The hidden state, the vendor id and stuff looks about right. Not sure what simple bit is giving the thing away, apart from the qvl/spice output Like Pollo is mentioning.

I understand this does not help, because you already played around with it.

A bit stumped at the mo, sorry.