Virtulization+PCIe Passthrough questions

Hardware Build a PC
1

Hey guys, I have a few questions about an upgrade that I’m looking to go ahead with, and I would like like to check a few things since I am new to this passthrough stuff and this particular side of virtualization.

In order for me to achieve a passthrough I need to upgrade my PC so I’m looking at getting:
Mobo: Gigabyte Z370 HD3 OR Gigabyte Z370XP-SLI <-- How can I be sure it supports IOMMU? If it doesn’t can someone recommend a decent one?
CPU: i7 8700K
RAM: 16GB of the cheapest stuff since this is hella expensive atm so, Crucial (CT16G4DFD8213)

Hopefully these specs allow me to do CAD stuff/gaming sufficiently.

I have a r9 380x that I’m currently using and would like to use it on my windows vm. The reason why I am looking at a 8700K is due to it having integrated graphics (for host) as I don’t want to buy another GPU. So I’m thinking about having the GPU display directly to my 144Hz monitor while the host displays to the other 60Hz one, I don’t mind doing it this way (rather than looking glass) as I have two monitors.

Now here’s the fun bit, audio and usb stuff.

  1. USB:
    I would like to have a USB expansion card in a pcie slot (passed through to the VM) but here is where things confuse me, if I do have a usb expansion card in there, will it take up pcie lanes? As the CPU says it can have up to 16 lanes, but that will be already taken by the GPU to my understanding? What if I want an NVME SSD in the future, would this also use lanes?

  2. Audio:
    I imagine if I have audio coming from the host, I can use it via hdmi connection between the motherboard/monitor, but from the VM would it be ideal to pass through a sound card or can I just pass through devices connected via the motherboards audio ports?

Thanks in advance! :smiley:

2

Get R7 1700 instead, the extra cores will be handy in VM, and with the extra $100 you saved get dGPU like RX 560 or used one for your second OS.

3

GI have no need for a dedicated gpu for the host, that’s why I’m looking at 8700k since that solves the graphics issue and it’s cheaper than 1800x (I would ideally get that) but $$$

E: Isn’t there bugs still with Ryzen or is that sorted out?

4

Looking back at what you said, it isn’t a bad idea, I do end up saving a bit, all I have to do is use some $35 nvida card for the host. I can also OC the CPU too, and that will give similar performance to 1700x/1800x stock.

My main concern now is which motherboard to go with, as it needs to support IOMMU correctly, according to https://www.youtube.com/watch?v=aLeWg11ZBn0 the Gaming 5 and Taichi seem to be ok, can someone confirm? Also can these mobos pass through specific usb ports?

My next concern is sound (as I mentioned above)

5

The main advantage of 1700 is 8c/16t vs 6c/12t plus OC.
Sorry i can’t recommend any MB or sound issues because I’m not experienced in VM, If @wendell would step in that would be nice.
Try asking in here, or search in the website for related issues.

6

I have done a bit of research and have found a few things out.

For AutoCAD workloads, It prefers higher clock speeds as the majority of it is single threaded workloads (revit/robot) so the 8700k is looking like what I’ll try to base things around. Second, the Asrock z370 Fatal1ty Gaming K6 seems to have good IOMMU groups (confirmed by Wendell’s video and other people on reddit1 reddit2), so GPU passthrough is possible on that board, when using an iGPU to run the host (Apparently it becomes a bit weird when trying a two GPU setup). In addition to this, since it has good groupings, it is possible to pass through a USB controller (or USB PCIe card) and a SATA controller.

For the sound, there are a few ways to go about it, but the one that might work for a noob like me is to use a USB to audio converter and make use of passing through a USB controller. If I do go ahead with a build like this, I’ll post to let people know how it goes. I think that’s everything considered :thinking:

7

I’m running a 1700 and Taichi for passthrough and they work wonderfully.

All the Ryzen bugs have been ironed out.

The Taichi has issues with USB controller passthrough, I believe, but let me share with you the IOMMU groupings as they show up on my system.

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6 [1022:1466]
IOMMU Group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 11 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 11 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 11 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 11 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 11 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 09:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 11 0b:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 12 0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580] [1002:67df] (rev e7)
IOMMU Group 12 0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 13 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [10de:1b82] (rev a1)
IOMMU Group 13 0f:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 7 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 7 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 7 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 8 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 8 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 8 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 8 12:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)

The only warning I’ve got for you is this: You still have a risk of encountering the segfault bug with your CPU if you go ryzen. You’re going to have to do a 8ish hour burn-in test to find out if your CPU suffers from it (it’s a manufacturing flaw).

Ah, just noticed you replied and are going for hte 8700k. That’s also a solid choice. The issues with a dual gpu setup is the way that the motherboard groups the GPU for SLI/CF.

I don’t personally know anyone using the Z370 series, but I’m sure it’s got support as good as skylake.

1 Like
8

Cheers! Hmmm I haven’t heard of it, what is this segfault bug about?

Edit: Ahhh yeah, i thought you were speaking about the 8700k for the segfault :smiley:

9

Ahh, you were on the discord yesterday, huh!

I don’t know too much about the bug, but it can cause issues with compilation and might cause issues in a couple other circumstances.

Edited the previous post to clarify what CPU I’m talking about.

1 Like
10

Yes that was me! Thank you for your help there, you guys were very helpful! All this stuff was a lot for me to take in but I think I have a much clearer image of what I would like to do (and how to achieve that).

1 Like
11

In regards to sound - Passing through a separate audio device is the best solution.

Initially I passed through a very basic USB sound card, like the one you linked above. If you want both your guest and host audio coming out of the same channel then you can plug a cable between the output of the USB card and the line in on your motherboard.

Later on I bought a PCI sound card as I wanted features such as Dolby headphones for gaming. I don’t have host and guest audio through the same channel now, but this is a trade-off for sound quality.

Both work perfectly, with no crackles or pops. Just depends what features you want from your sound.

Here is a thread I found useful:

https://forum.level1techs.com/t/gpu-passthrough-audio-solution?source_topic_id=122681
1 Like
12

I see, thanks for that! I’m thinking of just passing through the USB one, since I do not think I’ll have a need to hear both audio lines at once, but its nice knowing that’s an option.

13

We always strive to be. Glad we could help.

1 Like
14

GUYS!!! :smiley:

I finally got it to work, I’m so happy!!! It was pretty difficult for me at first, since I have only used ubuntu prior to this (and minimal usage of the command line), but after a few installs I got it working on fedora 27!!!

A huge thanks to the guys on the VFIO discord, they explained a lot to me and I am very grateful for that as there were a ton of things I did not understand, but this was well worth the learning experience.

The hardware I ended up going with was:

Mobo: ASRock Fatal1ty Z370 Gaming K6
CPU: i7 8700k
GPU: R9 380X Gigabyte G1 Gaming (I was using this previously)

The IOMMU groups were awesome for this board, I was able to effortlessly pass through the ASMedia USB controller (the two USB 3.1 Gen2 ports) and the ASMedia SATA controller (two SATA ports). The GPU passthrough was a bit of a challenge as I had to reset my BIOS (I updated it as soon as i got it) as even with the “use onboard graphics” and “iGPU multi-monitor” options enabled, it was still outputting to the GPU as the first device, so I think that was a small bug that causes a lot of issues and stress.

So the main guides I used to get this working on fedora 27 was:



https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

And the help of the awesome people on the VFIO discord!

Performance is really good, a noticeable increase with a better CPU and the same GPU. One thing though, I believe the R9 380X suffers from the reset bug, as I am able to restart the VM with no problems, but I don’t get any output when I shut down the VM completely and go to start it up again, I have to reboot the host, in order to boot up the VM again, other than this, it works very well from what I have experienced so far. I hope this info is useful to some people later on. :+1:

Edit: One thing to note is that on fedora 27 I don’t have sound coming through the HDMI speakers for some reason, it may be due to it running on an iGPU and the 8700k is fairly new, so drivers might not all be there atm (that’s what I am thinking)

15

Hi FluxBoi,
I am also trying to use Linux as my host an MS Windows as guest. My build will be 9900K + AS Rock Z370 Gaming K6 + some NV card for passthrough. However, I will use Ubuntu as Linux Distro.
Do you have some tips regarding the order I have to stick to get everything run properly?
What to do first and whats last.
Thanks

16

Hey, its 2019 now and guides a way more abundant. Ubuntu and most other distros now will have all or most of the required stuff in the kernel to enable VFIO and related tasks. In saying that you use Ubuntu, you might be able to get away with a Debian guide, but I’m not too sure, as I have only ever done this with Fedora and still use it to this day. It’s been rock solid and very reliable for my uni work over the past 2 and a bit years.