Hmm, not sure how to make that happen.
Correct.
The log shows what the command is that is used to generate the VM.
I will test these things at work tomorrow.
Honestly the only thing I’ve ever had to do in proxmox was add the pci ID to the config file it tells you to edit and the rest worked perfectly.
1 Like
Even with Nvidia cards and drivers?
Here is the command used to start the VM:
command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=f675c872-c390-4668-9c48-423f5b4ff239' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,id=drive-efidisk0,file=/dev/pve/vm-100-disk-2' -name Test-Windows -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,kvm=off' -m 7168 -k en-us 'hv_vendor_id=null' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=02:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'usb-host,hostbus=6,hostport=1.2,id=usb0' -device 'usb-host,hostbus=2,hostport=4,id=usb1' -device 'usb-host,hostbus=3,hostport=1.2.3.4,id=usb2' -device 'usb-host,hostbus=1,hostport=1.2.3.4,id=usb3' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:6a10a080c99' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:24:CF:B4:AA:A2,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'accel=tcg,type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
This section is when I use args: hv_vendor_id=null:
... -k en-us 'hv_vendor_id=null' -readconfig ... and that’s causing it to fail.
Here is the actual command it is using when it successfully boots the machine, using x-vga, which results in no “Microsoft Basic Display Adapter”, but Error 43 on the Nvidia GT 710.
/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=f675c872-c390-4668-9c48-423f5b4ff239' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,id=drive-efidisk0,file=/dev/pve/vm-100-disk-2' -name Test-Windows -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,kvm=off' -m 7168 -k en-us 'hv_vendor_id=null' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=02:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'usb-host,hostbus=6,hostport=1.2,id=usb0' -device 'usb-host,hostbus=2,hostport=4,id=usb1' -device 'usb-host,hostbus=3,hostport=1.2.3.4,id=usb2' -device 'usb-host,hostbus=1,hostport=1.2.3.4,id=usb3' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:6a10a080c99' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:24:CF:B4:AA:A2,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'accel=tcg,type=q35' -global 'kvm-pit.lost_tick_policy=discard'
This remined me on one specific line on the Arch wiki:
It may help for the ID to be set to a 12-character alphanumeric (e.g. ‘123456789ab’) as opposed to longer or shorter strings.
While I am not sure if this will solve your problem it’s definately worth trying. Something like args: hv_vendor_id=1234567legit (not sure if this needs quotation marks).
I attempted it with null using the terminal instead of the GUI, which did boot the system, but did not fix the Code 43 Error.
I did the same with 1234567legit and it is still Code 43 Error.
To clarify @oegat, I believe this is an issue with QEMU/KVM passing through whatever is needed to trick Nvidia drivers into thinking it’s not in a VM.
Edit:
I tried using hidden=1. I get this:
kvm: can't apply global host-x86_64-cpu.hidden=1: Property '.hidden' not found
I believe I’ve tried every option and combination I can think of so far aside from the following options:
- ACS patch
- AMD GPUs
Which just seems there is something else wrong since other people get this to work using these types of cards, unless having the PCI-e slot bundled with the GPU is wrong.
The important part of the command I’m using is this:
-vga none -nographic -no-hpet -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=1234567legit,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off
Honestly, it’s using /usr/bin/kvm, so at this point, I’m trouble shooting QEMU/KVM instead of Proxmox. 
In the above the hv_vendor_id=null item is not part of the -cpu argument, I don’t recognise this usage. But:
This on the other hand looks reasonable. My corresponding -cpu line is
-cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=nvidia_sucks,kvm=off hv_vendor_id=null works for me as well. So your line looks sufficiently similar.
I see that you are emulating the q35 chipset, have you tried using pc-i440fx-2.10 (or similar) instead? Beware that changing chipsets may likely trigger re-activation of Windows, but it should be OK when you switch back again, provided you do not activate (IME).
Yes, this was an error in the config I was making.
The wiki didn’t mention it, and the GUI doesn’t list it as an option?
Checking the Wiki here, https://pve.proxmox.com/wiki/Manual:_qm.conf#_options
cputype=<486 | Broadwell | Broadwell-noTSX | Conroe | Haswell | Haswell-noTSX | IvyBridge | Nehalem | Opteron_G1 | Opteron_G2 | Opteron_G3 | Opteron_G4 | Opteron_G5 | Penryn | SandyBridge | Skylake-Client | Westmere | athlon | core2duo | coreduo | host | kvm32 | kvm64 | pentium | pentium2 | pentium3 | phenom | qemu32 | qemu64> (default = kvm64)
It doesn’t seem to list that as an option?
I made an attempt by strictly following the Wiki, but I still get the Code 43 Error. This means I just added options to vfio.conf, and used hostpci0: 02:00,x-vga=on in the configuration file for the VM without changing the CPU type, kvm on or off, hidden state, hv_vendor_id, or any other options aside from what the wiki specified.
I checked my GPU’s VBIOS using the rom-parser in the wiki, this was my result:
root@pve-001:~/rom-parser# ./rom-parser /tmp/image.rom
Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 128b, class: 030000
PCIR: revision 0, vendor revision: 1
Error, ran off the end
It says I need a Type 3 setting for it to work, but I think I have to test this with the normal drivers working? It doesn’t mention that, but I assume that’s true. This means I have to undo the vfio-stub setup, then check the rom.
On to that I guess.
q35 vs pc-i440fx are options for the “-machine” keyword, not “-cpu”.
This part of your command line:
-machine 'accel=tcg,type=q35'is in my version:
-machine pc-i440fx-artful,accel=kvm,usb=off,vmport=off,dump-guest-core=offI don’t know what accel=tcg means, and I don’t see it discussed in the manual that you linked. I suggest that you test first to only replace “tcg” with “kvm”, as this is a small change (edit: this of course implies that you change the command line directly, just for testing). Note: afaik “hidden state = on” is just libvirt’s way of defining “kvm=off”, so I don’t think these are different options. accel=kvm is not an opposite of kvm=off.
When searching I don’t find good reason to believe that q35 is the culprit, as I find several examples of people using it with geforce passed through. But if you want to try, then you can replace “q35” with “pc”. This will render an activated windows install un-activated, but it should be in an activated state again when switching back. “pc” is a shorthand for the highest version i440fx emulation that the host supports. Don’t try to copy my exact machine type, “pc-i440fx-artful” is an ubuntu-specific alias.
I know very little about how GPU rom matters, as I have not so far been forced to look into that. So I can’t give advice there.
1 Like
So I still get the same thing when loading the Nouveau drivers.
root@pve-001:~/rom-parser# ./rom-parser /tmp/image.rom
Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 128b, class: 030000
PCIR: revision 0, vendor revision: 1
Error, ran off the end
So I’m thinking that’s part of my issue.
https://pve.proxmox.com/wiki/Pci_passthrough#romfile
then you can pass the vbios file (must be located in /usr/share/kvm/) with
hostpci0: 01:00,x-vga=on,romfile=vbios.bin
So I’m going to have to do that somehow. May need to grab the PCI-e 1x to 16x riser to test it out.
This is relatively fine. It’s a one and done thing. We’ve been buying GT 710’s for our machines @SgtAwesomesauce since we didn’t realize it would affect our ability to do this (and that we didn’t even know we’d want to do this until the last few days).
Swapping GPUs in all our workstations would be costly, so we would rather not. I’ll probably just manually set up each one, and write out the instructions for how I did it.
1 Like
How old are they? If they’re relatively new, you could try to return them.
@Vitalius Did you install Proxmox on LVM or ZFS? That decision prior to installation also determines if you install in in UEFI (LVM) or BIOS (ZFS) mode.
In that case you may need to check instructions for PCI-Passthrough in BIOS mode or provide your own VBIOS-ROM as I mentioned earlier since there may not be an EFI-partition to get it from.
Last straw I can think of (when selling the GPUs is no option) is switching to Fedora Server and administrate it remotely via oVirt or Kimchi. There is quite a lot of frontends which use libvirt underneath, making the passthrough a cake run from my perspective since all you need to do is following the Arch Wiki instructions and adjust the script to your needs.
They’re in user machines, most are GT 710’s, and they’re in use at the moment.
We need to do maintenance on the machines anyway to fix a blunder of ours in installation but I don’t think buying new GPUs (even AMD 410 or 510s if they exist) would be something my supervisor would be cool with.
LVM
Hmm, then we would have less management tools since Fedora Server doesn’t have what Proxmox has built-in (which is one major reason we were interested in it).
If all else fails, I’ll try that and see what my co-workers think, but I have a few more things to try on Proxmox before I give up fully on it.
Yeah. I keep forgetting that a lot of companies don’t really operate the way mine does. I more or less have free reign on my area as long as nothing breaks and the bill doesn’t get too high.
1 Like
That must be nice.
Right now, the machines are working and administer-able. We’d just have a much easier time handling down machines with Linux as the host and Windows as a guest for obvious reasons (Windows 10 update snapshots among other things).
It’s “worth touching every workstation to put this on it” as long as “it’s free aside from labor” and “it’s relatively easy copy/paste GUI-able work”.
1 Like
So an update (finally).
I remembered to bring my PCI-e x16 to x1 riser from home to work.
Putting the GPU in the 2nd PCI-e slot (1x) gives me the same ROM error when I try to parse it (as listed in the Arch wiki).
root@pve-001:~/rom-parser# ./rom-parser /tmp/image.rom
Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 128b, class: 030000
PCIR: revision 0, vendor revision: 1
Error, ran off the end
There’s only 2 PCI-e slots on this board, so that’s my only other option.
The IOMMU grouping for the 2nd slot includes the GPU with a USB controller and the slot itself.
I’m also getting this error, but I’m not sure if this is just something caused by using a riser in general:
[ 3888.457702] nouveau 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
Though that error sounds like it’s related since I am checking the ROM.
Unless you’re just wanting to go linux for this, microsoft has a free solution to this called Microsoft Hyper-V server. It’s completely headless and will just run as a hyper-v server.
Hyper-V is pretty great but if you’re wanting to run virtualization in linux I can get that.
1 Like
The goal with this setup is to be less dependent on Microsoft. Thanks though. I didn’t know they had a headless server. That’s very interesting.
1 Like
Server Core is headless, it’s actually best practices now to run everything on server core and have one management server that you can run all the GUI off of, or use powershell for management. It’s good stuff.
1 Like
So, all the research I’ve done tells me that the GT 710 comes with a UEFI vBIOS. In fact, some guy had an issue with his motherboard not supporting UEFI and that meant he couldn’t use the GT 710 without a legacy vBIOS.
This tells me that the issue isn’t the ROM lacking that support.
I had thought that was the issue when I found this:
Which is the issue I’ve experienced, and the maintainer of the rom-parser mentions that the BIOS looks to lack UEFI support due to its size.
Given that switching PCI-e slots hasn’t made it easier to get the ROM for my GPU, since a different issue is that the ROM is shadowed (whatever that means):
BIOS assigns the VGA display to a specific memory address so the OS can find it. It also places that “shadow” copy of the ROM at a known address for the OS (for some legacy reasons i’m not quite sure about). Some more info at section 3.2.0.3
sigh why does it have to be so hard.
I guess I’ll try getting a clean vBIOS using GPU-Z.
Edit:
Oh, I found this: https://www.reddit.com/r/VFIO/comments/5sh41p/any_other_reasons_for_nvidia_driver_code_43/ddfhq30/
Video BIOS dump:
You need a secondary GPU that you can use as the primary for this process. You cannot dump a clean copy of the BIOS without having the passthrough GPU as a secondary card
Ah, this makes sense. I’ll need another GPU then.