Interesting. Yes, I bought a used 560Ti for host graphics. Maybe its specific to my board/card combo then. Thanks anyway!
Hello everybody,
I'm pretty new in Linux and i got a bad english so sorry in advance.
Since a long time i have wait Ryzen to finnaly switch to Linux with a nice VGA passthrough. Time have come recently to throw away my old FX4100. I have seen on many forum that Ryzen got problem with IOMMU and i'm glad Level 1 Tech provide an good ACS patch.
But i still got a problem, i just can apply the patch to my favorite Debian :( .
As far i understand the patch cant found the file /drivers/pci/quirck.c (or something similar). For applying the patch i used "patch -p0 < a.patch" in the dir "/lib/modules/4.11.0-041100rc3-generic/build" Can somebody provide me some help ? (btw im on linux 4.11-RC3 and debian 8, IOMMU enabled, and the AMD-V thing too)
Many thanks in advance :D
I've managed to get pci-e passthough working on a gigabyte gaming 3 matx MB and Ryzen 1700 as well, but using only 1 GPU - AMD to r9 290. However I'm facing a problem with the whole KVM setup and not sure what it's related to. For the Windows10 guest with the GPU passed through it crashes (guest only, host is fine) whenever I try anything graphics intensive, for example running Witcher3. Normal desktop is fine.
Also my Linux guests are acting odd when I try to SSH to them, I notice that the SSH terminals just stop working randomly. And then there's the issue with very slow network throughout to both VMs. I have no idea what's going on. It used to work fine with my Intel setup. There's no logs in dmesg to show a problem either.
I'm going to try Seabios instead of OVMF to see if I can stop the crashing.
Any one having similar issues or anyone can advise?
IT WORKS!!
It took some doing, but my 1070 is running in passthrough to the Windows guest using the primary x16 slot. The last few sticking points:
You need to use a romfile rather than passing through the VGA rom from linux straight to the guest. Linux does things to the boot rom as it starts, so if you are booting from the card you intend to pass through you'll need to supply a clean rom dump. Not all ROM's work - I had issues with the rom I downloaded for my card. I wound up using a different rom file for another manufacturers card. This rom file works for me:
https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0Windows will complain if both a virtual and a real graphics adapter are on the same machine. Just watch and see if OVMF will start on your card. If it does, things are looking good.
NVIDIA seem to be actively trying to stop vfio passthrough on their cards. If you get 'error 43' and nothing else seems to be wrong, follow the advice on the Arch Wiki Passthrough guide and try to trick the Nvidia drivers into thinking they are running on a real machine.
I'll upload some benchmarks when I can be stuffed. Asus uploaded a new bios, so I'm gonna play with my memory and try to get it to go a bit faster first.
1 Like
Make sure it's stable too plz. Works but not stable is where I am unless I use nvidia
1 Like
Just build a Kernel from source, it is actually pretty straightforward. There is a nice tutorial on the Fedora Wiki, which should be valid for Debian too (Just make sure, you got the right build dependencies installed.) It includes the patching process (which is just this command but in the source folder.)
so using AMD passthrough cards is a no-go? Anyone else have data on this?
Maybe not nogo but the thing crashes as soon as I load a driver.
If I don't load any driver on the host the guest drivers work fine.
1 Like
Is this the case even with nvidia on the host and blacklisted AMD drivers? I need a card on the host system because I use it for GPGPU, that's my main motivation in wanting a passthrough VM. (doing heavy compute work on your display driving card tends to make UX pretty terrible and it lets me get more work done or play games while I render or train my NNs)
No, with the latest uefis, it works to mix nvidia + amd and seems reasonably stable.
Even though it is impossible to boot from a graphics card attached to the PCH, it is also possible to disable the primary graphics card at boot time and pass it through to a VM IFF the gpu supports uefi reset. And you may still have to pass through an external rom!
But it does work. If you don't mind fiddling with it, you should be safe in a mixed config. And it is also safe to use the gpu off of the PCH IFF your primary graphics card supports reinitialization.
1 Like
That's a load off. I don't mind tweaking at all, my current, z170 board's IOMMU stuff was totally busted out of the box, and I had to do some similar fiddling to get it up and running too.
I managed to resolve the issues I was having with the guest crashing. Turns out that the graphics card wasn't being cooled properly causing the guest to crash when it got too hot. So I've been successful in passing through my one and only primary amd GPU to a Win10 guest. Drivers installed ok and games seem to play fine.
In case it helps someone. I'm running Arch Linux, Kernel 4.10. I had to use OVMF, A dumped BIOS of my GPU as a rom file. You can use either i440fx or Q35 for the guest. No ACS patch was needed with my Gigabyte gaming 3 Ab350M motherboard.
Here is my kernel params:
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off isolcpus=0-7
Here's my libvirt XML file
Windows10
d45c3b5f-be8a-41e8-a22c-02e91c4c6245
8388608
8388608
4
hvm
/home/virtualguests/windows10/ovmf_code_x64.bin
/home/virtualguests/windows10/ovmf_vars_x64.bin
destroy
restart
restart
/usr/bin/qemu-system-x86_64
This discussion seems to focussed on a 2 GPU scenario, with the first GPU assigned to the host executive system, and the second GPU passed through to a virtual machine, using an acpi patch.
Are there any problems with a single GPU scenario, with A headless host executive system, and the first GPU passed through to a virtual machine? Does this scenario require the acpi patch? Which Ryzen chipsets would this headless scenario work with? How stable would this solution be? Will I have any difficulty passing through a USB chipset on the gigabyte gaming 5? Thanks
Do you think I could get PCIe passthrough working with two Nvidia cards on an ASRock Fatal1ty X370 Gaming K4? I wanted to get a Ryzen 5 CPU with that board, get a reasonably cheap card like a 1050 for Linux and use my 980 Ti for a Windows VM but I'm not sure if I should wait until I have some kind of confirmation that it will work.
Hello everybody!
I have been also dreaming about making it possible to run 2 VMs on one machine.
I can now finally say, I was successful with Ryzen 7 + unRAID!
I am now able to run 2 or even 3 VMs with dedicated GPUs.
My setup:
Ryzen 7 1700 @stock (planning OC)
ASUS ROG Crosshair Hero VI
16 GB Corsair 3200 MHz DDR4
Samsung 960 EVO 250GB (cache)
Seagate Barracuda 3TB (planning to get one or two more)
Radeon HD5770, R5 230 (guests); Nvidia GTS 450 (host, guest also possible)
- I am using these lowend GPUs which I had lying around, once all tested, I'm going to get pair of RX480/580.
Linux 4.9.19-unRAID x86_64
ACS Override Enabled (downstream,multifunction)
- note that multifunction option was MANDATORY for my setup, otherwise the override was working only for top 2 PCI-Ex16 slots.
Syslinux config:
cat /boot/syslinux/syslinux.cfg
default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
menu default
kernel /bzimage
append pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
kernel /bzimage
append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
kernel /bzimage
append pcie_acs_override=downstream initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
kernel /bzimage
append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
kernel /memtest
IOMMU Groups - these actually played out amazingly! I did not expect after lots of hate on the grouping, that it would separate so perfectly, see for yourself!
IOMMU group 0
[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 1
[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 2
[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 3
[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 4
[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 5
[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 6
[1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 7
[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 8
[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 9
[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
IOMMU group 10
[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 11
[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
IOMMU group 12
[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 13
[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
IOMMU group 14
[144d:a804] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
IOMMU group 15
[1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
IOMMU group 16
[1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
IOMMU group 17
[1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
IOMMU group 18
[1022:43b4] 1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
IOMMU group 19
[1022:43b4] 1d:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
IOMMU group 20
[1022:43b4] 1d:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
IOMMU group 21
[1022:43b4] 1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
IOMMU group 22
[1b21:1343] 21:00.0 USB controller: ASMedia Technology Inc. Device 1343
IOMMU group 23
[8086:1539] 23:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 24
[10de:0dc4] 25:00.0 VGA compatible controller: NVIDIA Corporation GF106 [GeForce GTS 450] (rev a1)
IOMMU group 25
[10de:0be9] 25:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)
IOMMU group 26
[1002:677b] 26:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450]
IOMMU group 27
[1002:aa98] 26:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 28
[1002:68b8] 27:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]
IOMMU group 29
[1002:aa58] 27:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]
IOMMU group 30
[1022:145a] 28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
IOMMU group 31
[1022:1456] 28:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
IOMMU group 32
[1022:145c] 28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
IOMMU group 33
[1022:1455] 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
IOMMU group 34
[1022:7901] 29:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 35
[1022:1457] 29:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457VMs:
8 CPUs (0-7 and 8-15)
Machine i440fx-2.7
BIOS: OVMF
Hyper-V is OFF
Virtio drivers: virtio-win-0.1.126-2.iso
keyboard and mouse passed through
Audio from USB double jack adapter
- I had to set sound quality to at least 48kHz, or there was crackling in the audio
or
Audio passed through from MB ([1022:1457] 29:00.3)
With this setup, I am running 2 GPUs with the Radeon cards passed through to VMs. I was, unfortunately, unable to passthrough the Nvidia card due to constant BSOD (System Thread Exception).
However, with newer Nvidia card, or another AMD GPU it is possible to run even 3 separate machines from this setup!
- unRAID does not mind at all that I am actually taking a GPU from it and it assigns it to the VM correctly - it is even able to boot to the VM, the BSOD appears after installing Nvidia drivers.
Hope I'll help at least somebody trying to achieve the same I was able to.
2 Likes
Hmm, I had similar luck using older radeon hd + RX480 but mixing fury + rx480 or rx480 + rx460 was much more unstable.
It would be interesting to test.
What are the simptoms of instability?
I have 2 r9 290 in main slots and passing one of them to VM.
I had no crashes but some games suffer performance penalty.
I had passed both of them but that had not resolved the problem.
heaven seems to just hang, but can be restarted. sometimes gta v just crashes back to the desktop. its more annoying than anything.
I'm looking for a new build with Ryzen that will focus on virtualization including GPU passthrough. I was wondering if anyone could help me out with a few questions. I am currently looking at using either the Asus PRIME B350-PLUS or the Gigabyte GA-AB350-GAMING 3. Does anyone have any information on how the IOMMU groups look on either motherboard? Is the ACS override patch needed for either one? Does anyone happen to have any information on how stable they are with the GPU passed through?
Second, does anyone have any information on whether it is possible to do GPU passthrough while using a NVM-e SSD? Is the NVM-e slot in a different IOMMU group?
Thank you very much in advance for the information!
2 Likes
Hi all! I went the road of building a Ryzen system using ASRock x370 Taichi and 1800x with the hopes of having a two GPU setup where one of the GPUs would be passed through to a Windows 10 VM. Unfortunately it seems that both PCIe GPUs show up in group 2 as @wendell had mentioned.
I submitted a ticket to ASRock late last Friday to see if I could get a response from them, but haven't heard anything just yet. If/when I hear anything I will report back!
MB: ASRock Taichi x370
PS: Ryzen 1800x
OS: Ubuntu 17.04
Kernel: 4.10.0-19-generic