Level1 Linux: Livestream (Setting up PCIe Passthrough on Fedora on X299 and Threadripper systems) | Level One Techs

We've got two systems: i9 7900 and two Threadripper systems we're working on testing IOMMU. Come with questions.


This is a companion discussion topic for the original entry at https://level1techs.com/video/level1-linux-livestream-setting-pcie-passthrough-fedora-x299-and-threadripper-systems
6 Likes

I missed the live stream for this but I am watching it now. When I think of Linux I donā€™t expect to see Windows at all. I guess this was Windows being run in a virtual machine. The thing is though why should people like me try to learn to use it if we are going to end up using Windows for some things anyway? I donā€™t understand also as if you use the net under Linux doesnā€™t the government, your ISP and whatever still collect information or is that wrong? I remember this talk about virtual machines not being real pcs and have seen some funny videos on the net of it. Finally I am reading up on IOMMU, Grub and also sorts of other things you are saying and I am doing my best to try and understand some of what I am seeing but really for someone like me I need a Linux For Dummies : Real Dumb Personā€™s Edition. Is there such a thing? I hope the intro. to a new OS would be better than that I used long ago.

Other things ā€¦

I will simply say I enjoyed being exposed to a different OS, enjoy your videos at Level 1 Tech and even if I never understand Linux or another OS other than Windows I can still enjoy this content. I am glad I looked today in the subsection of Level 1 Tech.

Now about orange pop eh I try to drink it once in awhile but it always ends up burning my throat or well it feels like that and so I donā€™t go back to it for a year maybe.

PS: Donā€™t call yourself names for making mistakes. You are going to make them. Also I donā€™t like hearing one of the few smart people I enjoy watching rip on himself. You made a really good, fun video as it is just real and loose. It is not packaged like some ad.

Missed the live stream, but i wachted the whole thing, i like this format of video for showing/telling people what they can do with there pcā€™s and osā€™s
@wendell donā€™t think that you are the only one thats not good at doing things while people watch you and expect interaction, // statement from wife and kids :fearful:

1 Like

Sure data is always collected. But the difference lays in the fact that you control your own computer (when you run linux) vs a company controls your home computer (when you run something else). One doesnā€™t have a keylogger, the other one does by default. Which one do you feel more comfortable with for general computing?

For alot of us, windows is now just reduced to a gaming console. Everything else (web browsing, basic office work), we can do comfortably with linux. If you want to have both the possibility to play video games and also have total control over the hardware you payed for, doing what wendell was showing is an alternative. Basically, you run linux natively and windows as a virtual machine under linux. Windows is then technically under the control of linux, whatever happens inside the VM will not disturb the host. You can assume it is safer for your data to have that kind of setup.

What is special about this particular setup is that you are capable of letting the virtual machine use hardware you have inside the computer. You can actually tell the VM ā€œHey, hereā€™s an expensive GPU, use itā€. What wendell is doing in the the video is basically 2 things

1- Tell linux to ignore the GPU he intends to give the VM
2- Tell the VM to use the GPU

You can also dedicate actual hard drives, usb devices, audio card, etc to the VM. If you think about it, you basically have 2 computers in 1, one running windows, one running linux, at the same time. And windows stays contained and isolated, while permitting you to do whatever you usually do on windows, which include gaming, photoshop, etc.

1 Like

Since I never tried it with VFIO (because I had it previously work with pci-stub)

Here is what I gathered you need to do on Fedora 26. I just tried it and it worked VFIO style, but I might be missing something that would be useful to do, so if any of you notice something that should be done, please do tell. So after dnf update and dnf install @virtualization

  1. Add to kernel line in /etc/default/grub

amd_iommu=on iommu=1 rd.driver.pre=vfio-pci

  1. grub2-mkconfig > /etc/grub2-efi.cfg

  2. To get hardware IDs

lspci -nn

  1. Add in /etc/modprobe.d/kvm.conf

options vfio-pci ids=1002:6899,1002:aa50,1002:4383 (my own hardware IDs)

  1. dracut --add-drivers ā€œvfio vfio-pci vfio_iommu_type1ā€ -f

Reboot

I have a question, and this may seem dumb but i was wondering would it be possible to have the VM us a raided disk? I have three SSDs in my system two are raided together and I have windows installed on it. On my other SSD I have Antergos installed on. So would it be possible to take another GPU and throw it in my system like an old 270x and have Antergos use it while the VM uses my 1080ti and my raid 0?

This right here is what scares me. Hell I donā€™t even know the English language well enough yet and I have been taught English since birth. LOL!

Btw @useraddf I understood what was going on in the video and I appreciate the explanation you gave to hammer it home but I just donā€™t understand how this ever gets simple using this OS. I get that I may never understand how to use the OS but I want someone to point me to a good starting point where everything I need is in one location. I donā€™t want to be going all over the net looking at this and that to try and get things done and piece the OS together. I guess with it being open-source though that is the way it is. If I am wrong please tell me. Finally ā€¦ like I said before in other posts related to Linux conversations / topics I had installed it in the past and couldnā€™t get the bloody thing off my hard drive easily. It was a royal pain in the ***. I donā€™t want to put myself through that again.

so is this not working? is vfio bound to your pci IDsā€¦ there are 3 hardware IDs? that seemsā€¦ odd?

Oh, with my semi-old hardware, pci-stub works. VFIO also does according to the steps I posted. Iā€™m just saying that people that have issues with VFIO could try with pci-stub? But Iā€™m not sure how that will differ.

3 hardware IDs, because I also passthrough the audio from the motherboard to windows, as I have an extra sound card in the system. This actually fixes audio issues, btw.

edit:

For those of you thinking about this, Iā€™m using the DGX from Asus on fedora, and I have it working

I always follow those steps when I fresh install and it always works:

sudo dnf install gstreamer1-libav gstreamer1-plugins-good gstreamer1-plugins-ugly gstreamer1-plugins-bad-free gstreamer-ffmpeg gstreamer-plugins-good gstreamer-plugins-ugly gstreamer-plugins-bad gstreamer-plugins-bad-free gstreamer-plugins-bad-nonfree

sudo dnf install clalsadrv.x86_64 alsa-firmware.noarch alsa-lib.x86_64 alsa-utils.x86_64 alsamixergui.x86_64

For DGX Card

sudo modprobe snd-oxygen
sudo modprobe snd-pcm-oss
sudo modprobe snd-mixer-oss
sudo modprobe snd-seq-oss

alsamixer

***Set multichannel
***Analog Surround 4.0

1 Like

my monitors have audio out, have always used that to get no stutter audio. There is a way to fix the audio with ac97 drivers but itsā€¦ not newbie friendly

Looks like you need plenty of patience for this. :smiley:

Cool, good to know. Fedora really makes it easy, they sold me as the distro of choice. The only issue I have is that my old 5850 gives a white screen on fedora install usb and a black screen with no way of getting a graphical interface when logging into fedora installed with another gpu (R9 280). Iā€™m passing through the 5850 and it works, canā€™t say much about performance but looks good to me for basic office work.

@wendell watched the video and i am curious if you might be having problems with reboots due to using a 20 pin display port cable. this is a problem i had with an rx460 and stumbled upon the fix by absolute dumb luck. a 19 pin cable solves this problem. apparently the 20th pin feeds voltage to the video card and causes problems.

I enjoyed the format of the video, (I missed the stream, different TZ, sorry) ā€¦ yes it was longer than it strictly needed to be, but it has that extra sense of reality. Realtime with @wendell :slight_smile:

Any chance of seeing you spend an hour or two trying to setup Gitlab a few weeks from now? Or trying to use Glade or qt creator , or writing a webserver in your language of the day or using puppet/chef/ansible across VMs or just to rebuild a machine?

@wendell just wanted to say I loved the stream!

Donā€™t worry about being too quiet, I know it can feel like every pause is hours when youā€™re ā€˜performingā€™ but itā€™s never actually that long, so feel free to shut up and finish what you were doing even if it means not interacting for a a few minutes.

Also, please, PLEASE, feel free to waffle/ramble because the stuff you talk about when you go on a tangent is usually very interesting.

Hope you do more livestreams in the future!

@wendell , First off I want to thank you so for the stream and your interest in this area, very informative!

I am eager to setup a 1950x system with Linux the primary OS as soon as this VM performance can be fixed. (And hey a re-init bug fixed here or there wouldnā€™t hurt either!)

You said that Xen doesnā€™t have the problems with the npt bug, but said there was some issue with the PCI passthrough. Can you clarify what that is? If I can use Xen to get my desired setup I donā€™t mind going that route either.

Working on xen myself but Iā€™m not there yet. Anyone have any experience doing xen on fedora want to chime in here?

1 Like

so, i managed to get virtualization going on i7 6700k ( i know i am late to the party)
and got someone offering me buying me probably anything under ~2.5-3k euros (i know insane) but he wants a rather quick response and list

@wendell would you still recommend the Taichi board for virtualization most and which memory to go with it ? i would pick the trident z rgb 32 GB 3200 kit

so basically thats my current list: https://de.pcpartpicker.com/user/Achati/saved/NYjjXL

any ideas what else would be useful for such a project except a kvm switch and loads of tme ?

PS: sorry if this is kinda offtopic and if i should repost in a seperate thread then just tell me to or move the post (no idea how that works behind the scenes here)

PPS: the mentioned working virtualized system is running manjaro-xfce and virtualizes enterprise win10, uses virt-manager, passes through a rx480, the host runs on the iGPU on the 6700k
your recent videos on the ryzen virtualization led me to your older stuff about doing it on skylake and i got around learning it all and managed to do it ā€¦ dream-ish system yay

Hey there! Sorry for not answering any questions but Iā€™d like to know what the situation is with IOMMU and threadripper. Iā€™m thinking of building 2 gamers 1 CPU type rig with an extra NAS with Plex running on itā€¦ I know Iā€™m asking for a lot but having that many cores at a low price are interestingā€¦
Here are most of the parts I was thinking of using :
https://pcpartpicker.com/list/RBMZNN
Player 1 would have a 1080.
Player 2 a 970 I already have.
The host would be the cheapest GPU I can find.
There arenā€™t any drives but both players will have an SSD as a boot drive and the NAS will be a 5 4TB drive Raid5.
If the IOMMU situation is too chaotic on threadripper, can it be done on ryzen 7 or X299?

Iommu on TR is not ready yet. X299 has been a dream but $$$$

1 Like