No display with virt-manager, QEMU works

anon27075190 · February 13, 2021, 10:26pm

Have you installed and setup ovmf, for your VM to support booting from UEFI? Your current XML is definitely missing the instructions for ovmf to boot the VM. Here is an example how it could look like:

  <os>
    <type arch="x86_64" machine="pc-q35-5.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/vmname_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>

This is how the options in virt-manager look for me after installing ovmf. Pay attention to the firmware drop-down menu:

examen1996 · February 14, 2021, 9:30am

First off, make sure you are using a graphics card that supports uefi, gpu-z could tell you that.
Regarding virt-manager, if you see the bios screen, that means you havent removed the virtual gpu(whichever one that is).

totalgaara · February 14, 2021, 9:33am

Hi, thanks for your reply. I already tried to boot with UEFI but since it’s a legacy gpu, i’m not sure if i would get trouble, so i stick with Q35 to avoid installing windows again (i think i installed windows 15 time this week…)

The good news is : It work now !

Here is how i fixed it , i spend some time this night looking at my QEMU script and the “QEMU” logs running with Virt-Manager, and everything seems to look fine except one line.

on my script i was using :
-device vfio-pci,host=05:00.0,x-vga=on \

and found “x-vga=on” (i took the base script from Gentoo Wiki ) witch wasn’t there on the virt-manager machine, so i edited my xml configuration to put :

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">

/*   AT THE VERY BOTTOM after </device> */

  <qemu:commandline>
    <qemu:arg value="-set"/>
    <qemu:arg value="device.hostdev0.x-vga=on"/>
  </qemu:commandline>

AND that did the tricks ! it works just fine now and even Nvidia driver installed without issues.

NOTE :
I maybe have a “small” question, that isn’t really the thread issue, but since i installed this gpu, my host feels “slow”, is it possible that two gpu slow down my computer ? my config is :

MB : Gigabyte "GA-970A-DS3P FX " with the latest high end chipset AMD 990FX
CPU : AMD FX 8320
RAM : 16 GO DDR3
GPU 1 (host) : AMD R9 290
GPU 2 (VM) : Nvidia GT 320

I hope it’s not that … otherwise i can live with that, just hope that doesn’t impact games

examen1996 · February 14, 2021, 10:06am

Ooo, fellow fx user, had a sabertooth 990fx with a FX 8120.
Ok, your iasue is that this old gpu need a vbios when being passed through, also you must remove the header if you take it from techpowerup.
All in all you have some fun ahead of you

examen1996 · February 14, 2021, 10:08am

Going back tonyour offtopic question, check to see at what pciexpress speed you are running your primary gpu at, you might have a case of pciexpress 2.0 at 8x , and that you preety much explain the slowness.

totalgaara · February 14, 2021, 10:20am

Yep, i extracted the rom with Windows, gpu-z, it’s a old gpu so don’t have the special headers, if i run a hex editor it start with “IBM VGA …”

following up I’ve check on Windows (still have my dualboot for games that doesn’t run well), GPU-Z show me first
“PCI Express 16X 3.0 running at 1.1”, but i GPU-Z show a “?” asking to run a video test, because “dynamic gpu ajustement speed” and sure enough, if i play the gpu-z video, it’s change to : PCI Express 16X 2.0 running at 2.0" (will never got 3.0, motherboard don’t support that).

And for the GT 320 it’s shows “PCI Express 4X 2.0 running at 2.0”.

Witch is true, my R9 is my main GPU connected on the first pci express port, the GT 320 is on the second port and it’s 4x only.

I also found that on linux by typing : “radeontop”
both memory clock and shadow clock sometime maxing out , is it a struggling framebus ?

examen1996 · February 14, 2021, 10:30am

As you said, pcie 3.0 will never bee correct as the chipset simply doesn’t support it.
In the past I used lspci -vv to see the speed on linux.
As for your vm, if you can use the vm when running with qemu using a monitor connected to the passed through gpu, then it should work with virt-manager as well.
I would say recheck and make sure you passed throug the entire gpu group in the .xml, make sure that there is no gpu and no diaplay attached(if you have a display it will automatically assign a vga something something ) and try again

totalgaara · February 14, 2021, 10:38am

It’s working now i just add a “display” with spice and a “video adapter : none” to try “looking-glass”.

I don’t know if i should make another thread for my “other issue” because it’s not more with the “main” issue.

Anyway, here is lspci -vv output (i just took the gpus):

1:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290/390] (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited / Sapphire Technology Device e289
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 33
	NUMA node: 0
	IOMMU group: 14
	Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at d0000000 (64-bit, prefetchable) [size=8M]
	Region 4: I/O ports at e000 [size=256]
	Region 5: Memory at fe300000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: amdgpu
	Kernel modules: radeon, amdgpu

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X]
	Subsystem: PC Partner Limited / Sapphire Technology Device aac8
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 39
	NUMA node: 0
	IOMMU group: 14
	Region 0: Memory at fe360000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

05:00.0 VGA compatible controller: NVIDIA Corporation GT215 [GeForce GT 320] (rev a2) (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited / Sapphire Technology Device 2150
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	IOMMU group: 12
	Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at a0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at b0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at c000 [size=128]
	Expansion ROM at fe000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

05:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1)
	Subsystem: PC Partner Limited / Sapphire Technology Device 2150
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 17
	NUMA node: 0
	IOMMU group: 12
	Region 0: Memory at fe080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

could it be because i’m using the “amdgpu” drivers ? this graphics card is not really supported, it works, but it’s in a “beta” stage still. (and i think it will not leave this beta stage because… start to age too)

anon27075190 · February 14, 2021, 10:44am

Glad you made progress. This might be unrelated but my host slows down in anything rendering related when I have a second graphics card available to the host. I always need to make sure to use the vfio-pci.ids=10de:13c2,10de:0fbb kernel parameter, so that the graphics card I want to pass though is not available to the host.

totalgaara · February 14, 2021, 10:57am

I already put that in /etc/modprobe.d/vfio.conf :
options vfio-pci ids=10de:0ca2,10de:0be4

i don’t know if i’ve to put that on grub too?

here is lspci -nnk result :

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT215 [GeForce GT 320] [10de:0ca2] (rev a2)
	Subsystem: PC Partner Limited / Sapphire Technology Device [174b:2150]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau
05:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be4] (rev a1)
	Subsystem: PC Partner Limited / Sapphire Technology Device [174b:2150]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

examen1996 · February 14, 2021, 10:59am

Great, I was sure that the display was the issue .
In regards to your question, it might be that the second slot goes through the chipset and not directly to the cpu, but dont take my word for it as my Fx knowledge is starting to go away (lstopo would tell you ).

From your output I did not see the speed, maybe use sudo, or try another command.

All in all, I always ask , why not single gpu passthrough ? Do you need 2 , does your first one do anything while you use your vm ?

Thermally and power-wise, you are far better off using a single gpu.

examen1996 · February 14, 2021, 11:01am

Graphics card and any other devices are not being used if no module is loaded for it.
Even that command is doing exactly that, telling the kernel to use vfio, this however can be done on the fly if needed, and it shouldnt influence the behaviour.
I would rather advise on blacklisting noveau .

totalgaara · February 14, 2021, 11:36am

I already tried single GPU passtrough, but it wasn’t good. And i can understand that on a Ryzen it may be useful, but on a FX, we got 2 core on 4 packages, it’s more like 4 core hyperthreaded and you have to give to the host at least 2 core, so already a performance drop.

Other issue was that my R9 was running at full speed all the time, can’t even watch a video, sometime when was shutting down the vm, i’had kernel panic.
I decided that was enough, it’s too many issue just for that, both of my OS are still on SSD, it’s quicker to just reboot to Windows.

Now you may ask, so why don’t you run your apps on your dualboot windows ?
All my personal file and etc are on linux, when i’m dualbooting i feel lke it’s not my computer but for gaming when proton / wine doesn’t work, it’s good. But it’s a pain to reboot just to use one app like Fusion 360…

So i had this idea of VM with GPU passtrough, i’ve some other GPU, a HD5450 (17W) / A GT 610 (27W) the GT320 is the less power efficient CG but also the fastest considering that i’ve only a PCI 4x, if i use the GT 610 or HD5450 witch are bad to start, it will be worse

But your’re right, if i check my iommu groups, i’m using the motherboard chipset on the GT 320.

IOMMU Group 12:
	00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]
	05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT215 [GeForce GT 320] [10de:0ca2] (rev a2)
	05:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be4] (rev a1)

Isn’t this supposed to have consequences only in the VM?

totalgaara · February 14, 2021, 7:23pm

Unfortunatly i’ve to stop here. I still have a question, does CSM could be the cause of all those lags ? I removed the GT 320, disabled iommu, then i rebooted and it was already working better.

Then i rebooted again, and disabled CSM and … it’s feel faster ! But i don’t know if it’s really that, i should have done this step by step…

You may ask, why did you give up? Well turns out that all of my GPU : GT320 / GT610 are too old to support “NVENC”, with that feature missing, i cannot use a software like “Parsec” for fast “remote desktop” and probably the reason why looking glass was running so badly and make a high CPU usage on the VM even when just moving the mouse.

I may purchase another GPU, for now i’m not really sure because i still have to find what was causing the slow down, CSM , the fact that the second gpu is using the southbridge and not the CPU?

Have you got an idea ?

examen1996 · February 14, 2021, 10:27pm

My past PC had the following config:
AMD FX8120(stable oc 4.2 ), sabertooth 990fx(the chipset being the important part), 16gb ram, Nvidia 750ti(old bessy had nvenc) and a hd 5450(didn’t know about single gpu passthrough back then).

I did not have any slowdowns, also the performance difference when running something like a cinebench r20 was about 1%, gaming was better virtualized then bare-metal.

My 750ti was in the second x16 slot (my motherboard had 2 pciexpress 2.0 x16 slots).
What I’m trying to say is that, the cpu or the gpus(being multiple that is) should not be the problem.
What could be the problem is a underpowered psu, or a misspalced pcie card(usually your motherboard manual should let you know whch slots to use in crossfire/sli scenarious, which is lames terms for fastest pcie slots).
Took a peek at you mobo manual, you second full pcie slot is hardware limited to 4x, so this is preety useless.

I would simply stick to one gpu since that would allow you to use the full speed.
Also, do a bios reset and start from scratch with you bios.
Csm will not boost, nor speed your pc, only the boot process.

All in all, keep in mind that you are using an old platform with a cpu that was always far from best(even thou I still miss mine) .

Good luck

totalgaara · February 15, 2021, 9:19am

Crazy that we share similar things haha, both have tried FX and both of us have a HD 5450 and yesterday after giving up i honnestly tried it and weird thing : It works O_o i don’t feel my computer going slow, i did some tests, the 3 gpu with the same videos, always using 1440p60 fps.

R9 + GT320 : Very bad, almost 300 frames lost in 15 sec
R9 + GT610 : Same experience
R9 + HD5450 : A lot better, like if i had only one GPU O_o is it the “crossfire thing”?

But yeah, i follow what you say, i give up with that motherboard, another friend also have a Ryzen processor with a 2nd pci “4x” and he don’t have this issues, don’t think it’s really related, yeah you lost performance, but not “crazy”, have to keep in mind that it’s not for gaming and he do the same.

But mine mothercboard, for some reason, maybe be because it’s the last gigabyte motherboard made for this CPU and they didn’t really care about features (for example, i don’t have any bios update) , before i had the “Gigabyte GA-970A DS3P” “non FX model”, i had a PSU issue, a long story, but i tough it was a motherboard fault, i sent it to RMA and they just refund me. When i had to choose another motherboard, all was left is this one and some other that didn’t have any VRM radiator and… i would never go with that, specially with FX.

I’m one of the “Bad Lucky Coolermaster”; since i had this computer i had to change it 3 time.
Coolermaster G750M (750W) → Died, PC keep crashing my brother had the same PSU and the same issue.

Coolarmaster V750M GOLD → Died on arrival, they sent me another and it work, but honnestly i don’t trust that brand anymore and it’s true that yesterday it was feeling a bit warmer but can’t tell if due to the other GPU or just computer usage time.

Now i’m back to one GPU setup, could stay with the HD5450 but it’s a so weak GPU, even a integrated intel hd graphics would be powerfull, it’s worst on a 4X x) and it’s reducing the cooling efficiency of my R9 by limitating the airflow of, 2 fans.

Maybe i could retry single gpu passtrough but … i can’t find the goal of doing that, isn’t it just faster to reboot ?.

Personnally even if i had two 16X slot i couldn’t use all of the other GPU because i’ve an ultrawide monitor and none of them except the R9 290 support resolutions about 3340x1440"

PS : i’ve seen you on a random reddit thread on VFIO and you are telling to somewone

I would just do x forwarding over ssh one one of you VMs, as for the vendor-reset patch, dude , that is the answer, it work outstandingly good.

what is that ?

anon27075190 · February 15, 2021, 10:57pm

Some GPUs have a reset bug. Meaning you can turn on the VM with a certain GPU passed through once and when you turn it off the GPU remains in an bad state. If you try to turn the VM on again the GPU won’t be able to start. The vendor-reset resolves this issue for certain GPUs. Link: AMD Polaris, Vega & Navi Reset Project - vendor-reset.

x-forwarding over ssh however is a technology that enables you to open a window of a application on your local computer while the application itself runs on a remote computer. For example I could open and use Firefox. Firefox will run on a remote computer and the information for drawing the window you would normally see is transmitted to your local computer via ssh. SSH itself is a technology to connect from your terminal to a remote computer and run commands on it. Link: SSH (Secure Shell) - Wikipedia

totalgaara · February 16, 2021, 11:21pm

Hi, about my slow issue, i think i just found the culprit, i don’t know why, but it’s Manjaro.

I’ve did some more testing because i’m “like that”, i never want to give up, that can be good and not good but in this case, it’s good.

On manjaro i got severe drop using for example Chromium, then i tried on POP OS and … it’s fine…

Now i don’t really know why Manjaro have this issue, i tried to switch back to LTS Kernel and this had no effect at all. I’m not really sur what causing it, i don’t see too much on my journal CTL.

And for Windows you may ask ? Since i removed my GPU i keept switching back to windows / linux (that’s a pain) and i realized that i had some little drops than i wasn’t aware about, mostly i came from a 1080p 23" monitor to 34" “3340x1440” ultrawide and since i had it (2 month) , i spend 98% of my time on linux so my “memory” was wrong. There is some little drop and adding the GT 320 is not causing more drop.

Now i’m looking forward for the issue on Manjaro, if i can’t find it i would like to stay on arch to get the latests feature but maybe POP_OS could be an option too

anon27075190 · February 16, 2021, 11:36pm

Manjaro offers different kernel version, the newest being 5.10 while POP OS as a distribution based on Ubuntu should be at 5.8 the latest. You might check out if it had something to do with the kernel version. Other software packaged could be the culprit as well. It would not be that great if you run into the same kind of problems after the next upgrade.

totalgaara · February 21, 2021, 11:28pm

Hi, just a quick update to say that the issue is now fixed !

I had sent a ticket to GIgabyte e-support to ask them if it was normal and they quickly sent me a bios to flash, that wasn’t on their website for some weird reason.

At the begining i was very skeptical, i didn’t want to flash a bios that “only me” have, but then i secured myself with the dual bios + a bios backup. So i flashed it and :

It’s day ‘n’ night ! No more issue, no more slow feeling sensation, no more framedrop, i’m so happy with that Also i’had a “bonus fix” that i never tough it was related to a bios issue, 75 Hz on linux, whatever distro was terrible, i was getting huge screen tearing, blackscreen every 10sec etc. Now : it’s fixed too

Thanks again for your reply, now my virtual machine is working perfectly fine (i just think that i will quickly change to a UEFI gpu because of “no legacy reset”)