This would be an awesome collab with Linus and Anthony @Wendell, because simply using unRAID is kinda the “pre-packaged VFIO” solution. If the next 7 Gamers 1 CPU system was based on 7 6800 XTs and used a server Linux distro instead of unRAID, that would show how people can break free of unRAID to do everything it does, except you’re in control. That would be more useful info as opposed to how to modify unRAID to do what you want. (6 editors 1 PC had to get a special Hugepages build just to get working with proper memory bandwidth… Yeah…) The IPMI with GPU would mean the host OS can run using the IPMI interface. (I believe 6 editors 1 PC did this)
The old 7 Gamers 1 PC if anyone is curious:
and The 6 Editors 1 PC series:
BTW, Linus used (basically) these two things from One Stop Systems:
But all that stuff runs off of PLX chips. There’s even Gen4 versions now, but again, it uses PLX so the ACS override is unavoidable.
I loved your series with Gamers Nexus making a custom NAS system, so taking it to the next level for a ultimate Anthony, Wendell and Linus collab for the return of the 7 Gamers/Editors 1 CPU project, except avoiding the use of unRAID, would be pretty sweet.
And to take ZFS out of the equation, use BTRFS instead. Same features (mostly, anyway) but native Linux unlike ZFS.
Having said that, I don’t think LMG needs Wendell to make this project work. Linus may not know much about Linux (although he knows more then he’s showing off) but Anthony is well versed and would be able to finish this project on his own. Still, another LTT collab with L1T and/or STH would be nice, if only for that d@nged virus
Well, doing it fully on their own with zero unRAID needs Wendell. unRAID is basically a compilation of quality of life Linux tweaks to get QEMU user accessible. The next step is doing what it does from scratch, so you don’t have to pay for a commercial shareware Linux distro.
No I don’t think so. You may be underestimating Anthony’s capabilities here, and Jake also knows a few things Linux too.
I also slightly disagree on your comment about unRAID and QEMU. The main thing unRAID does, IMO, is providing a GUI for the hypervisor, on the cli one could use QEMU just fine as regular user. At least, that’s how I understood the QEMU docs when I was trying stuff out. (quite some time ago )
No, actually there are some special sauce patches included with a paid version of unRAID. Most importantly better downstream ACS support than anything public right now.
The fix for ACS dependencies is to have direct lanes you can bifurcate, and avoiding use of PLX chips at all costs.
…to @wendell. If he can get Gigabyte ARM server boards, I’m certain he can source this board to kickstart things with Anthony and Linus.
What I want to see is scrapping all ACS Overrides and PLX chips for the Threadripper PRO iteration. 128 lanes is enough for both Optane 905p arrays and 100Gbit networking.
You can easily switch between different ACS modes in unRAID whereas it requires a lot of work on something like Fedora 33.
Since Wendell is already trying to work with Fedora 33, getting this board working with bifurcation on the final slot, then M.2 to PCI-E for the rest of the lanes, you could theoretically bypass using ACS on a multi PLX chip “external PCIE chassis” idea using just the available 128 lanes.
Might be a chance for Linus to learn “Cockpit” to manage VMs.
I wouldn’t call it a lot of work, switching modes only requires a change of kernel parameter, update your bootloader config, and reboot.
If you commonly need to switch, make multiple bootloader options each with different ACS patch options.
And even if switching between modes on unRAID is easier because the options are built into the GUI, that does not mean that the patch is any different than the one generally used on Arch/Fedora/Ubuntu/ETC. It just means that they have some GUI stuff hooked up to switching kernel parameters.
So I still have not heard any proof that the actual ACS patch used on unRAID is different than the public one generally used.
This board is so nice! UnRaid has been fun for me (limited Linux experience of that level caps me from creating my own at this time) so i didn’t mind paying the licensing fee for ease of use. They have clearly put their time into 6.9.
This board will be my next upgrade for my house unRaid. My current X79 and i7 3820 finally hit a performance wall that justifies new system $$. I make my stuff last. 7x PCIe with 128 lanes, what more can i ask for?
Heres my question: With custom liquid cooling i can probably fill most of those GPU slots, assuming anyone can ever get one, with all those lanes and the 2x M.2 populated would i even come close to saturating bandwidth? Assuming the unRaid is streaming to multiple users simultaneously.
I figured the PCIe to 4x M.2 drives being hit, with the Graphics cards, might do it🤷♂️
)
