Would like some expert opinions regarding a system for VFIO / Looking Glass

Hello.
So you probably guessed from the title but i am a Linux user and over the last couple months i have been experimenting with VFIO and Looking Glass.
So far i am very impressed with the results but my current 6700K micro ATX system just does not have enough cores or PCIe expandability to do what i want. So i have decided to use this as an excuse to upgrade.

This is far from my first build but i am still having trouble deciding on what platform i should go with.
My budget for this system is around 1200-1400AUD but i can push that a bit higher if i wait a bit longer. (And yes that is Australian dollars…)

I already have a GTX 1080 (host system) and a Radeon R9 290 (guest system). I will eventually upgrade the 290 but for now it will do.

I also plan to pass through a dedicated Ethernet controller, USB 3 controller and possibly a sata controller so the guest can have direct access to its boot drive.
It would also be nice to upgrade my Linux boot drive to an NVMe M.2 but that will depend on how many pci lanes i have left over.

So I’m going to need a processor with at least 20 pcie lanes which instantly rules out Intel’s entire consumer line including the 9900k which would have been at the upper end of my budget.
That leaves me with a choice between Ryzen 7, Threadripper or an Intel X series processor like the i7-9800X.

I would like to go with Ryzen but i am a little concerned about single thread performance because i do still play a lot of single threaded games. I know its not a massive difference between Ryzen 2 and Intel but i am considering waiting to see how Ryzen 3 performs. But then there’s no guarantee Ryzen 3 will work well with VFIO on launch.

And even after all that i still need to make sure i go with something that has popper IOMMU support so i can hopefully avoid having to the ACS override patch i am currently using.

So hopefully now you see why i cant make up my mind. I would love to get some suggestions / Recommendations from anyone who has experience in this area.

Thankyou.

p.s. To clarify im not looking for help parting out an entire system that fits within my budget. I just need help picking a CPU and motherboard that support everything i need. These are the 2 most expensive parts and my budget is flexible so as long as they are not too expensive i can figure everything else out myself.

i7-6900K might be worth looking into. Intel HEDT tends to have better hardware ACS.

If you need even more single threaded performance and don’t mind only 6 cores, the 6850K is another option.

Aussie dollars are similar to Canadian dollars, so Threadripper and Skylake-X are instantly getting too steep in terms of pricing.

Older platforms tend to be more mature for IOMMU, and Broadwell-E is what I recommend right now for those seeking good single threaded performance without going Skylake-X.

That could work. But the cheapest i can find it is around 1100AUD from an AU store or 1400AUD on amazon US.
the 9800X is currently going for 970-1000AUD. Unfortunately its not currently in stock anywhere…
On the AMD side a TR 2920X would also be cheaper at 950-1000AUD and according to userbenchmark it actually has better single thread performance at stock speed. And its not far behind when comparing overclocked scores ether. Link

I just dont know how manure IOMMU support is for something as new as the 2920X. It is also at the far end of my budget. I was hoping too find something in the 500-800AUD range but i would rather wait a bit longer than settle for a lower end processor.

I wouldn’t be too worried about single threaded performance. Usually the GPU will be the bottle neck in either case for gaming, when talking about same-gen CPUs. And even if it is CPU dependant, we’re talking less than 5% difference here.

It’s like owning a car that can go 200 mph vs 190 mph. If most speed limits you are experiencing go up to 150 mph, does it really matter for the one or two use cases per year you will be able to go max speed? Both are Freakin’ Fast regardless! :slight_smile:

Add to that the AMD offerings in general offer more performance per buck at their various price segments (though perhaps not the performance crown), this makes me lean towards AMD more. Of course, Intel is better at performance per watt and are often more quiet, too. But we all have different things we consider important…

1 Like

True. Considering the options available single thread performance probably isn’t really an issue. I think i am going to try to stretch my budget around a 2920X. The question is whats IOMMU support like for the 2920X?

Edit: Considering the cheapest TR4 board i can find costs more than most processors this will be quite the stretch. But i will make it work if i can…

Don’t put PCIE devices on the chipset for X399. Right now the actual X399 chipset has no hardware ACS.

I meant looking at the used market for the 6900K, because that would fit your budget better. For this case, you probably won’t be overclocking, so any used 6900K can work.

Bryan from Tech YES City gets processors from Aliexpress, and he hasn’t had any issues (He lives on the Gold Coast BTW) but eBay would work too.

1 Like

Rite. I wasn’t planning to run anything through the chipset if thats what you mean. But as far as devices using lanes from the CPU. Whats the IOMMU support like in that situation? And does the motherboard have any effect on IOMMU support for devices connected to CPU lanes?

I had a feeling you were talking about a second hand 6900K. I am not to keen on the idea of using second hand parts for my main rig. I know the CPU is probably one of the least likely things to just randomly die but i still like to have warranty in case it dies 6 months down the road. I also just don’t like the though of upgrading to a 3 generation old discontinued part. That said. As far as budget options go i think that is the most affordable option that meets my needs. So i will be keeping it as a fallback option if it turns out i cant afford something better.

At this point i am leaning towards a 2920X system but its going to take at least a couple more months of saving to make that happen. And im still not sure if it has the IOMMU support i need.

If i do go with a 2929X i Was considdering going with the ASRock - X399 Taichi with the intention of passing one of its 2 gigabit nics though to the VM but based on what you just told me about X399 IOMMU support i suspect that wont be possible. So i may go with the Gigabyte AORUS Gaming 7 Though i still need to look into the PCIe lane assignment because i would want to use the first and 4th slots for my GPU’s so i still have the other 3 slots available for expansion.

p.s. I am a fan of overclocking so i will be trying to squeeze as much performance as possible out of this system. But without pushing it to the absolute limit because i do still want it to be a stable 24/7 system.

The Gaming 7 does have a Killer NIC rather than an Intel one, though ideally you want 1 Intel NIC and 1 of another brand like Aquantia/Realtek/Killer. And that’s if the chipset has proper ACS. X399 currently does not have chipset ACS.

It would work if you bought a Intel PCI-E NIC, but I highly recommend a board with PCI-E Bifurcation.

Eventually you need to figure out how to ensure the guest and it’s PCI-E devices are all running off of the secondary die. You don’t want anything to travel over infinity fabric and UMA mode might help when running your VM.

1 Like

Thanks i completely forgot PCIe bifurcation was a thing. I also forgot to consider PCI device => Die assignment. I still need to figure out how bifurcation would work as far as the hardware side of things. I assume i can get some sort of adaptor that splits a x16 slot. I would then need to figure out how to mount the devices attached to said adopter but that sounds like a fun 3D printing project so i’m fine with that.

In any case it seems i may want to choose a different board because it does not look like the Gigabyte Gaming 7 supports bifurcation. Based on this post it sounds like the Taichi is the board to go with.

Edit: So here’s a thought. According to the level 1 review of the Micro ATX version of this board 2 of the M.2’s are in their own IOMMU groups. Assuming this board is the same is it possible to pass through an NVMe drive as a VM’s boot drive? I assume M.2 pass through would only work with NVMe and not sata drives.

Still depends on topology so that you don’t have to go across infinity fabric. You’ll have to find someone with the board to do an lstopo with the NVMes populated.

Im still not ready to pull the trigger on this system (still saving) but i have been looking into ram options. I know Ryzen’s memory support has been improving but is it still worth going with certified memory? (aka anything with Samsung B-Dies)

The cheapest 3200MHz option i currently have is kingston HyperX fury at $176 AUD (Including shipping)
The cheapest kit i can find at that speed with B dies is G.Skill Flare X at $282 AUD

I am looking for the best performance i can possibly afford but is it really worth the 60% price increase at this point?

p.s. i would be using at least 2 kits for quad channel 32GB 4x8 with the possibility of doubling that in the future.

Edit: Did a little more digging and found this Ballistix Sport LT DDR4-3200 4x8GB
That’s obviously not using Samsung B dies but it is listed a “Ryzen Ready” i have no idea how close it would get to its advertised speed on Ryzen but at $332 AUD for 32GB vs $538 AUD for the
Flare X… Its kinda hard to pass up. Though one down side. Its Amazon global so if i ever have to RMA it that could be rather painful.

B die has tighter timings, and is more expensive because Samsung ramped down production while demand is still high.

The question is how much of a performance hit am i going to take going with something like Ballistix Sport LT? And how close am i going to be able to get to 3200MHz?

All it is is latency. It’s important if you have operations over infinity fabric, but if it’s in UMA mode, that doesn’t matter as much.

1 Like

Hey Brandon, would it be possible for you to post some links you’ve been reading about VFIO / Looking Glass? I’m also gearing up to build a second case with a gfx card to dedicate to a guest system. I’ve found a few posts but they span from layman to expert++ so I suspect it will take a few weeks before the dots start aligning. Anything would be welcome.

There are a lot of random little bits an pieces all over the the pace if you google around. But this would have to be the single most useful source i have found so far.


My current test setup is based entirely on that guide.

iirc this page has some useful info in IOMMU and applying the ACS override patch.


Though from what i understand the acs override patch should really only be used as a last resort. Your best bet is to go for a system that has good IOMMU support out of the box.

Thank you, everything helps.

we consolidate a lot of these random little bits here: