XCP-NG GPU Passthrough project

Hello Everyone,

I’m new to using XCP-NG and I’m wondering if anyone has any tips on GPU pass through.

The best video I’ve found so far on YouTube on GPU pass through on XCP-NG, unfortunately is in German. If you want to look it up, its title is:

XCP-ng - AMD Radeon GPU Passthrough

I have an old HP Prolaint G6 ML350 running 2 Xeon x5670 and 144gb of ram. I have added a AMD Sapphire Radeon RX 550 card and I am attempting GPU pass through as in the video link above however I get an error message.

Internal error: xenopsd internal error: Cannot_add(0000:0b:00.0, Xenctrlext.Unix_error(30, “1:Operation not permitted”))

I noticed most my cards are on the same IRQ 11, is this an issue?

VT-d is enabled and USB pass through works, and I have an HDMI bank/dongle connected to the output display of the sapphire card.

Also the Embedded mother board video card ( ATI ES1000 ) works to pass through also on IRQ 11.

I’ve tried disabling the on board card, running both cards and swapping IRQ.

All attempts have the same error message.

Internal error: xenopsd internal error: Cannot_add(0000:0b:00.0, Xenctrlext.Unix_error(30, “1:Operation not permitted”))

Any Ideas?

So apparently I was able to change the error message after some tinkering to:

Internal error: xenopsd internal error: Cannot_add(0000:0b:00.1, Xenctrlext.Unix_error(30, “1:Operation not permitted”))

I think XCP-NG sees this Radeon card as to different cards.

A video card and a sound card.

When i run lspci from the console line I can see 2 listings:

0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev ff)

0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]

When I run a xl pci-assignable-list I get

0000:0b:00.0
0000:0b:00.1

as assignable. When I try to assign them they assign ok with no errors.

xe vm-param-set other-config:pci=0/0000:0b:00.0,0/0000:0b:00.1 uuid=a0552ff6-87f1-2044-ba59-ed920db6377e

Its only after I try boot up the VM I get the new error message:

Internal error: xenopsd internal error: Cannot_add(0000:0b:00.1, Xenctrlext.Unix_error(30, “1:Operation not permitted”))

I still think this is a issue that’s related to the BIOS , like an IRQ issue.

Any Ideas?

Yes, that’s because that’s exactly what it is.

You need to pass both.

Check your IOMMU groups.

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done

Make sure your GPU is in a group by itself.

I ran the command and got this:

0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] [1002:67ff] (rev ff)

0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]

Which is the IOMMU Group?

The [0300] ?

Apologies for the RTFM (but, GPU Passthrough is esoteric and looking for information through Google is the best way to find out information).

I would suggest reading (or glancing at):
(In increasing level of complexity)

https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#:~:text=An%20IOMMU%20group%20is%20the,can%20only%20be%20passed%20together