Running on Linux using KVM and am trying to work out how I may handle SATA drives on a potential X399 build.
On my current x370, after fighting it for a few hours, I finally realized the two SATA controllers are split in a way not conducive for PCIe passthrough of SATA drives to a VM. One controller hosts all the normal SATA ports and the other appears to support the U2 and SATA Express ports. Doesn’t look like I can change that.
Now that Ryzen 3000 is announced I’m feeling a bit better about upgrading to Threadripper a bit sooner (I’d rather not wait six months if I can avoid it) but I’m wondering if the Taichi, with it’s three SATA controllers, may have the same issue? Been poking around here and the internets at large for an answer but am coming up short.
I’m trying to setup a Windows VM for all my rich media needs. I’m passing in a video card, a FW card, my entire SATA controller (my Linux host runs of NVME) and one of the USB hubs on my current setup, but am using VFIO for networking (it gets me gigabit speeds so that works out for me and no need to both with more network cables, etc.). My hope was to do the same thing on x399 without having to do the ACS patch. ACS works but it’s just one extra thing to bother with.
I know of no combo USB/Firewire/SATA (and while shooting for the moon maybe audio) all in one card. That would be awesome. Otherwise if I have to get a SATA controller card, I’ll have 3 (out of 5) slots of the mobo just for the VM. I can probably live without the SATA controller on the host since it’s NVME but it means I can’t use a larger SATA drive in the host for things like VM images (of non PCIe passthrough VMs, for testing, isolated windows apps, etc.).
I thought about doing by-drive passthrough with VFIO. The last time I did that, I was getting silent data corruption. I think this was happening when booting into baremetal after running in a VM but it was still cause for concern. Passing through the SATA controller on my current setup works quite well (apart from not being able to use SATA drives on Linux) so that would be awesome if the SATA ports with the Taichi were spread across more than one controller.
Anyone know?
Aha, finally searched for the right keyword in Google and found this:
Seems to indicate that, no, all SATA connections hang off the x399 chipset and, I’m guessing, in a way that I can’t split them via IOMMU (without the ACS patch, and probably the entire SATA controller over to the VM with ACS).
Yeah I have one and I believe that is the case – they all hang off the chipset.
snaps fingers I was afraid of that. Curious then, what is your preferred approach? Do you have multiple NVMe’s (one for Linux, one for Winders, e.g.), virtualize the storage, use a NAS or?
MDADM + LVM + LUKS exposed as raw block storage to the VM.
I’m still trying to figure out good way to backup though. Also I haven’t run any tests on my storage as currently I’m not running anything IO intensive. I also have a SAS card where I have all my storage except for OS which is running on NVMe.
Hmm yeah I’m pondering my options here. Since I’ve started doing PCI passthrough, I haven’t had a need to baremetal boot into the system. Benchmarks show near metal performance and likewise even my latency sensitive apps (like Ableton Live) work really well so far. But there is always my safety net option there (to boot directly into Windows).
Linux is on an NVME, Windows is on an SSD and I have an HDD for bulk storage (for Windows). If I case aside my safety net, in reading up on things, it looks like ZFS + qcow2 seems to offer good to great performance (with raw instead of qcow2 as a simpler option). If I move to virtual storage, I could multi-purpose the SSD and HDD (not sure for what though since I don’t really need a ton of space on Linux - when I do, I just use my NAS) and house Winders and my sizable data on virtual files.
But that way it’s certainly a one way trip. No more bare metal booting that way.
You could put in a Dell Perc H200/H310, and pass that through to say a FreeNAS VM
Well… it is even worse.
Even if you do use the ACS override patch, you can’t practically pass through either of the two sata controllers.
There’s some chipset magic going on that makes doing so unsafe, and while it appears to work at first, it leads to instability. Like… consistently hard freezing the entire system in a manner of days, sometimes even in less than 24 hours.
I tried to do this sort of setup on my X399 Fatal1ty.
Just get a sata controller of some sort. Even those dirt-cheap asmedia ones would do what you want. And there’s always those cheap LSI cards that you can find everywhere.
I had my finger on the Buy button for a few days but decided to go a different angle in the short term and wait just a little longer. I can avoid ACS if I can switch from a firewire audio device to a USB one, and since I’ve been needing to upgrade my audio situation for a while (I need more I/O), I decided to upgrade this side of the fence first. Passing through one of the USB controllers has worked quite well for other things so we’ll see how it goes.
That doesn’t fix SATA but so far I haven’t had a need for any SATA devices on Linux. So passing the entire controller isn’t ideal but it’s passable - for now. I still plan on upgrading to Threadripper for my next upgrade. It’s just a question of when.
I can solve the SATA thing if I go back to trying directly passing the drives through via VFIO instead. Or I can move to fully virtualized storage (if I’m willing to give up bare metal boot). Or I can go with a SATA PCIe card as you mention (though not without ACS on my current box).
That said, I’m really hoping either Ryzen 3000 or, particularly a Threadripper revision improves IOMMU grouping for chipset devices. I think I would still want to go Threadripper but fixing IOMMU that would open up virtualization to a lot of folks. I dunno if it’s a hardware limitation or what but it would be nice, especially since the chipset seems to be more important for Ryzen 3000.
Have you tried passing the drives through using their path? Rather than the whole controller (which would be easier if it worked) you could add each of the drives using the add hardware, storage, then giving the /dev/sdxN or IIRC the /dev/disk/by-id?
Yep I did some moons ago. It definitely worked but I was getting silent corruption between bare metal and the VM. I think this may have been because I was sharing the same data disk across two different Windows installs technically. So indeed this is my next step to try again. I just am slightly wary because the data corruption thing did cause me some pain last time.
1 Like
