Pci-stub not claming secondary graphics card

Software & Operating Systems Linux
,
1

im trying to get the gpu pass through setup for qeum to have a dedicated graphics card but after setting up pci-stub it will only grab the sound part of the card and not the card itself

dmesg | grep pci-stub
[ 1.962446] pci-stub: add 10DE:13C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 1.962449] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 1.962453] pci-stub 0000:02:00.1: claimed by stub

am i missing something obvious, im an beginner/intermediate at ubuntu so im not sure what info you guys need

ubuntu 15.10
lspci -nn | grep NVIDIA
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 670] [10de:1189] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

only line in initramfs-tools not commented out
pci_stub ids=10de:13c0,10de:0fbb

thanks!

2

im having the same issue. any luck with a fix?

3

When I run the command on my system this is the output I get.

[blanger@IYF-HOST ~]$ dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.0.6-300.fc22.x86_64+debug root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet amd_iommu=on pci-stub.ids=1002:6810,1002:aab0,8086:10d3 LANG=en_US.UTF-8
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.0.6-300.fc22.x86_64+debug root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet amd_iommu=on pci-stub.ids=1002:6810,1002:aab0,8086:10d3 LANG=en_US.UTF-8
[    1.408461] pci-stub: add 1002:6810 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.408495] pci-stub 0000:05:00.0: claimed by stub
[    1.408500] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.408532] pci-stub 0000:01:00.1: claimed by stub
[    1.408552] pci-stub 0000:04:00.1: claimed by stub
[    1.408571] pci-stub 0000:05:00.1: claimed by stub
[    1.408576] pci-stub: add 8086:10D3 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.408611] pci-stub 0000:07:00.0: claimed by stub

Of course I'm passing through more devices than just the GPU, and this is on Fedora, but I'd look at the syntax of the line in your grub configuration.

You can run 

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-4.0.6-300.fc22.x86_64+debug root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet amd_iommu=on pci-stub.ids=1002:6810,1002:aab0,8086:10d3 LANG=en_US.UTF-8

And see if your output looks similar, if it's grabbing the audio portion but not the GPU then I'd say there is a syntax error somewhere, but I'm just guessin'.

4

$ cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-4.2.0-12-generic.efi.signed root=UUID=0167d477-2928-48c8-902a-5188ee0d88f2 ro quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7

i was actually able to fix one part of the issue by giving my gpu that im running ubuntu on a different set of drivers i enabled the restricted drivers in order to use the native Steam games available on ubuntu so that ended up fixing the issue with stub grabbing the whole device now

dmesg | grep pci-stub
[ 2.258442] pci-stub: add 10DE:13C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.258472] pci-stub 0000:02:00.0: claimed by stub
[ 2.258491] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.258510] pci-stub 0000:02:00.1: claimed by stub
[ 2.687899] pci-stub 0000:02:00.0: enabling device (0000 -> 0003)

but now i get this issue when starting QEMU

sudo /usr/vm1
[sudo] password for spawn:
cat: /etc/vfio-pci.cfg: No such file or directory
WARNING: Image format was not specified for '/home/spawn/Downloads/Windows.iso' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

any help with that?

5

Not really since I didn't use vifo-pci but used only pci-stub in my configuration I too don't have directory, so I'm no help, so did you write a script? and is it calling the file from that location? (etc/vifo-pci/vfio-pci.cfg or etc/vfio.cfg)

If you haven't read this it might be of some help....

My guess is that the errors are being generated because your missing either the vfio-pci.cfg file or the path to where the file is located is wrong...but again I'm just guessin'.

6

If he is missing the config file. Then make one.

For example, in order to bind my GPU with vfio I used the following:

 echo "options vfio-pci ids=10de:1200,10de:0e0c" > /etc/modprobe.d/vifo.conf

This created the vfio.conf file in the /etc/modprobe.d/ directory containing the string "options vfio-pci ids=10de:1200,10de:0e0c". From there I regenerated my initramfs (with mkinitcpio -p linux) and I was good to go.

@TheSpawnMan, Have you tried binding the drive to vifo instead of pci-stub? I was able to get the binding down with just vfio modules, no pci-stub necessary.

1 Like
7

i created the file that was missing and made it match to how the guide said which was to add these lines
0000:02:00.0
0000:02:00.1

but im still getting this error

sudo /usr/vm1
[sudo] password for spawn:
WARNING: Image format was not specified for '/home/spawn/Downloads/Windows.iso' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

so i took your suggestion... but you didnt provide specific commands so i did it the way i know how so ill walk you though what i did

first i added a config file to /etc/modprobe.d/ call vifo.conf using

sudo gedit /etc/modprobe.d/vifo.conf

and added the line

options vfio-pci ids=10DE:13C0,10DE:0FBB

then i remade initramfs by using the code

update-initramfs -u

and rebooted and got the same error

sudo /usr/vm1
WARNING: Image format was not specified for '/home/spawn/Downloads/Windows.iso' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

hopefully i did that all correctly let me know.

8

Are you sure you have your vfio-pci module enabled? This would be in your initramfs config file

Arch example:

[root@mups mups]# cat /etc/mkinitcpio.conf 
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES="piix ide_disk reiserfs"
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

Also, did you switch your GPU over to vfio drivers? What does your lspci -nnv say for your "Kernel driver in use"?

1 Like
9

it shows that pci-stub is being used... is that not what i want?

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device [3842:2983]
Flags: fast devsel, IRQ 17
Memory at dc000000 (32-bit, non-prefetchable) [size=16M]
Memory at b0000000 (64-bit, prefetchable) [size=256M]
Memory at c0000000 (64-bit, prefetchable) [size=32M]
I/O ports at d000 [size=128]
Expansion ROM at dd000000 [disabled] [size=512K]
Capabilities:
Kernel driver in use: pci-stub

02:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:2983]
Flags: fast devsel, IRQ 15
Memory at dd080000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities:
Kernel driver in use: pci-stub

10

so it turns out i never actually saved the bios changes when i enabled vt-d
so now that its enabled i get a new error

sudo /usr/vm1
[sudo] password for spawn:
WARNING: Image format was not specified for '/home/spawn/Downloads/Windows.iso' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

i dont know what to do with this new error though.... i was googleing around and found a kerrnel patch but that doesn't seem right. if someone would advice my next step id be very appreciative

11

Bumping for help...

12

hi

im working on the same problem at the moment.
If i read correctly then you're running mulitple PCIe- GPus and want to passthrough just one?
In that case run this:
find /sys/kernel/iommu_groups/ -type l
(as seen here: http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html)
The output shows your iommu groups.

My understanding is that all your stuff is grouped in iommu groups. To pass one thing of a group to a guest, all the items in the group must be isolated form the host. Even if you don't want to pass all of them through.

Thus if your GPUs both are in the same group, and you want to pass only one of them, and the other is the host GPU, then you have a problem.
Are your GPUs in the same group? (See output from above command)
If so there is not much to do.
(Detailed explanation and solutions in the link from above)
The summary, as i understand it, try putting your GPUs in another PCIe-slot and see if they come out in different groups.
If not, then you can try something with ACS override, but i refer to the link for that, as i don't understand it that well.
If both don't work then i think you're out of luck :/
(As am i at the moment btw :/ )

PS: Since your post seems a bit older. Have you solved the issue?
If so, please tell me how! :)

1 Like
13

As @Brutkruemel replied, unless the two cards are in separate IOMMU groups (not likely) it's not going to be possible to separate two cards with the same identity, I have this issue using two 270's that are a matched set from the same manufacturer, I didn't beat my head against the wall for very long before it was just easier, quicker, and probably more stable in the long run to just add another GPU from a different manufacturer which will have a different ID.

In my case with 3 GPUs I'm going to take the matched set and pass them through to the KVM in crossfire in hopes that I'll see a video performance improvement in games like The Witcher 2 that I'm just getting started playing, but it'll be a few more months before I get back to having the time to build a new KVM with the two cards and test it out.

I know that doesn't help you.....but it's all I've got.

14

i was unable to get the help needed and the further i looked into the issue the more i found that it just wasnt going to work how i wanted it to so i installed windows again.

im sorry i failed you my linux brethren.

1 Like
15

No need to be sorry, you tried, and you learned a lot, I'm sure down the road you'll try again and have a better understanding of what you will want to do and what it will take, no shame in that at all.

16

i know the feeling @TheSpawnMan
i installed windows again because i can't seem to seperate my two gpus too.
Despite them being different cards, and thus having different pci-ids, i only have 2 pcie-slots and they end up in the same iommu group :/

i'm giving the other thing i mentioned a try (acs override) but i think i'm doing it wrong.
@blanger did you encounter your cards being in the same iommu-group?
Did you have to use this 'ace override= downstream'-thing (sorry can't remember exact name and string)?
If so could you give any advice? :)

@TheSpawnMan No shame, Blangers said it right!
@blanger thanks for all your help again!

17

Yeah, the two cards with the same IDs are in the same IOMMU group, the third card by a different manufacturer (although still another 270) was/is in a separate group, I though at one time that moving the cards around and putting them in different PCI slots might change the groups but on my system it had no effect. I got lucky that I had a friend with a spare card that he had upgraded out of and loaned it to me to test with, I wound up just buying it from him.

I'm not sure what separates the cards into separate groups other than the ID, I would think it would or could be divided by slot or PCI lane used but in my case that wasn't the case as far as I could tell, I guess on my MB all the 8x or 16x slots might be using the same PCI lanes or controller or whatever, I'm just not that sure how all of that works to be honest....lol

18

I'm also not sure how that works :D
So i made a little experiment yesterday to see what determines the iommu group, and of course to get the two gpus into different ones so i can finalize my setup.

I put the 2 GPUs of mine, which have different IDs in several combinations into the 3 PCIe slots i have.
Turns out the top 2 pcie slots always end up in the same iommu group. and the bottom one ends up in a different one as the other two. So i don't think it is IDs that determines that. But as well might be a coinsidence with my hardware :D

The result unfortunately is that i need to use the bottom pcie slot and one of the two top ones to end up in different groups.
The bottom slot is right above my PSU. So i need a GPU that supports 3 displays (i.e. has at least one displayport output, and preferably be amd because i dont want to mix drivers) AND is a one slot card :(

Fortunately every HD (5/6)xxx card in theory supports 3 displays. Unfortunately no vendor puts a displayport output on their version. they all go wiht hdmi. so they abandon 3 display support :/

Anybody know a card like that? (1 slot, at least 1 displayport)

19

You can buy a HDMI to DP adapter like this....

They are pricey, and you have to watch for if your need active or passive type, but I have used them in the past and they do work for cards that don't have a DP port.

Thanks for doing the test, it confirms a theory I had that it is broken down by slot on some MBs, and I'd guess that changes by the mother board, PCI controller, number of lanes, and phase of the moon...lol I have the same issue needing one card to drive three monitors, but I'm going to opt to change two of my smaller older monitors (23") to newer 27" that have more connectivity options and get more screen size at the same time, the two I'm currently using are VGA only which limits my choices.

20

wow. i could buy a r9 270x for that money! Or a new mainboard ^^

A DP to HDMI active converter probably wont work right?
(Because they are around 20€)

Yes probably there are different ways of grouping on different mainboards.
Mine is a GA-z77x-D3H in case that interests.

The thing is i probably would buy another mainboard if i could be sure that it would work then.
But this seems more like gambling to me.
Well i guess i have to stay on windows till i upgrade :(