Adding a PCI Express Root bus device to a VM [ANSWERED]

Software & Operating Systems Linux
1

I am trying to add a PCIe root bus device to a Windows 10 VM. I first learned of this from gnif’s post on October 2018, post is here.

Please do not go there and start asking for support.

Wendell in this post provided some changes for the xml file, post here.

I’ve been trying unsuccessfully to make changes to a working xml file. Here are the related sections of my working VM file.

PCI controller section:

    <controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='1' port='0x10'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-to-pci-bridge'>
  <model name='pcie-pci-bridge'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='3' port='0x11'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='4' port='0x12'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='5' port='0x13'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='6' port='0x14'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='7' port='0x15'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='8' port='0x8'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='9' port='0x9'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='10' port='0xa'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='virtio-serial' index='0'>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>

GPU pass-through section:

    <hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x2f' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x2f' slot='0x00' function='0x1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</hostdev>

Please share if you know what needs to be changed to add a PCIe root device to this VM.

QEMU version 3.1

2

But you already have pcie-root-port . Gnifs post is about setting proper PCIe lanes ans speed as the driver reacts differently if left to default v1 x1. And for that you would need patched qemu.

Wendells post is about passing the GPU and its soundcard as on hw. Your GPU section would look like this for it:

    <hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x2f' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x2f' slot='0x00' function='0x1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
</hostdev>
3

Thank you for the reply!

I think I misworded the problem, I quote, Gnif, “this means the device would be physicially integrated into the PCIe controller, not on a PCIe bus.” I think this is my problem the GPU is attached to the PCIe Controller and not the PCIe bus. My frame rate performance appears to be about 25% lower than bare metal and when I shutdown the VM I have to reboot the server to get the AMD card to reset.

Also Virsh complains that I have duplicate devices on the same bus when I set them both for bus=8 so I haven’t been able to make that work either.

4

Here is the exact error message when I change the bus from 9 to 8:

/etc/libvirt/qemu$ sudo virsh define Win10.xml

error: Failed to define domain from Win10.xml
error: XML error: Attempted double use of PCI Address 0000:08:00.0

5

@Pixo After further research I see what you are saying that it is already working.

I think part of my confusion is why does it fail when I assigned the sound (function 1) to bus 8?

6

And here is the answer to separate ports for the video and audio.

“If using virt-manager, it appears to automatically create the correct PCIe heirarchy, though, on my system, it created a separate root port for the GPU and its HDMI audio device. Others’ testing has shown this to not matter much, or at all.”

There are also many other good tips over at that link.

7

Dont know if you have fixed it but the important part was that you use the same bus but the function for sound needs to be 1. The error provided shows that the function is 0 for sound.

8

@Pixo Once QEMU version 4 is released I’ll have the patch and should be good.

I think the XML shows the sound is function 1.

    <controller type='pci' index='8' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='8' port='0x8'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
  <model name='pcie-root-port'/>
  <target chassis='9' port='0x9'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>

…

    <hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x2f' slot='0x00' function='0x0'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x2f' slot='0x00' function='0x1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</hostdev>
9

That shows that the address for guest is not function 1 as function='0x0' is 0.
So of course you cant have it on the same virtual address as GPU.
Also the GPU would need the multifunction='on' specified in its virtual address for this to work.

1 Like
10

@Pixo I see now, I was looking at this line,

<address domain='0x0000' bus='0x2f' slot='0x00' function='0x1'/>

The above line references the sound and the line you posted reference which ‘pcie-root-port’ and function to use.

Thank you for your helping me to understand the xml file construct! It is very clear now, not sure why I couldn’t spot that before.

1 Like