A new set of fans arrived this week:
Thanks @Molly for recommending the Pn MAX series, really like the blade design on those.
I ended up going with their server fans for some reason instead of P8s though. For now they seem to be better at being quiet than the San Aces, but in an A/B test at “normal” RPM I wasn’t sure which is which.
Didn’t test them too much yet, barely finished replacing 3x Eiswind with 5x P12 MAX:
The P14 Slim is going to help cool the AICs, I still need to print a mounting bracket though:
On the software side I’m still in a bind, because the original plan (detailed below) is not going to work, sadly. I think. I didn’t manage to get a PCIe passthrough to an HVM and couldn’t get OPNsense to run on a PV, even after installing Xen “additions”. Tough luck.
The original plan was to get a Xen hypervisor and a lean Arch distribution (or something different, but highly customizable, none of that Ubuntu crap) to serve as minimal dom0 (for those unfamiliar with Xen speak, a privileged VM), plugged only to a “management LAN” with dedicated physical connections, and only on-demand internet access. Working hostname for the dom0: “vmserver”.
Along the dom0 there were supposed to be at least 3 different virtual machines (domUs) running on the server:
- A dedicated router OS, most likely OPNsense, to manage the 10G NICs and switching on them; working hostname: “opnsense”
- A storage server, likely another Arch installation but I wasn’t hell-bent on that. ZFS management, storage passthrough (HBA + NVMes), NAS servers (SMB, NFS, iSCSI), this kind of stuff; working hostname “zfserver”
- A services server. All the other junk that I want/need to run, like the internal certbot, pihole, lancache, etc. The only vm allowed to run somewhat unvetted software in containers (like the pihole or lancache); working hostname: “svcshost”
As you might have noticed, all the hostnames are 8-letters long. The word “hostname” also has 8 letters. Coincidence? ( ͡° ͜ʖ ͡° )
Initially I wanted the domUs to have direct access to the router VM bypassing dom0 for additional separation, but that seems to be impossible, at least for now. The closest thing I have achieved was VF passthrough from the NICs, as the dom0 can stay disconnected from those, and likely this is going to be the way forward.
Ideal network separation diagram
+------------------------------------------------------------+
| Xen Hypervisor |
| |
|+----------------+ |
|| opnsense [nic0]+-------------------------------------[nic0]
|| [nic1]+-------------------------------------[nic1]
|| [mgmt]+---------+ |
|| [vif0]+----+ | |
|| [vif1]+--+ | | |
|+----------------+ | | | |
| | | | |
|+----------------+ | | | |
|| zfserver [vif0]+--+ | | |
|| [mgmt]+----)--+ | |
|+----------------+ | | | |
| | | |+------------------------------+|
|+----------------+ | | ++[vif0]-----+ vmserver ||
|| svcshost [vif0]+----+ +--+[vif1]-----+-[mgmt-lan] ||
|| [mgmt]+----------+[vif2]-----+--------------[eth0]
|+----------------+ +------------------------------+|
+------------------------------------------------------------+
Non-ideal (SR-IOV based) network separation
+------------------------------------------------------------+
| Xen Hypervisor |
| |
|+------------------+ |
|| opnsense [n0p0v0]+--------[passthrough]------------[n0p0v0]
|| [n0p1v0]+--------[passthrough]------------[n0p1v0]
|| [n0p2v0]+--------[passthrough]------------[n0p2v0]
|| [n0p3v0]+--------[passthrough]------------[n0p3v0]
|| [n1p0v0]+--------[passthrough]------------[n1p0v0]
|| [n1p1v0]+--------[passthrough]------------[n1p1v0]
|| [n1p2v0]+--------[passthrough]------------[n1p2v0]
|| [n1p3v0]+--------[passthrough]------------[n1p3v0]
|| [mgmt]+-------+ |
|+------------------+ | |
| | |
|+----------------+ | |
|| zfserver [vif0]+---------)----------[passthrough]--[n1p3v1]
|| [mgmt]+-------+ | |
|+----------------+ | | |
| | |+------------------------------+|
|+----------------+ | ++[vif0]-----+ vmserver ||
|| svcshost [vif0]+--+ +--+[vif1]-----+-[mgmt-lan] ||
|| [mgmt]+--)-------+[vif2]-----+--------------[eth0]
|+----------------+ | +------------------------------+|
| +-----------------[passthrough]--[n1p3v2]
+------------------------------------------------------------+
The ideal layout assumes a full NIC can be passed; I’m not 100% sure that’s the case, but I didn’t try passing all 7 functions at once (4x PF for the VPs, 1x general NIC at .4, and 2 storage offloading functions). The non-deal assumes SR-IOV passthrough of all the ports to the router + inter-domain connection via the last Virtual Port - all n1p3v* act as if they were connected to the same network.
Except for OPNsense, all the other vms could have been running in PVs as they are Linux. In theory I could just go with some Linux-based router OS like VyOS, the list is long. Which prompts the…
Solution #1 - Linux-based Router OS in PV
Since PVs work so far, this solution seems like the obvious choice. I do, however, have some reservations. What if that’s not the only broken thing that I’m about to encounter in Xen? What if it’s another piece of the puzzle that’s broken? We’ve seen pfSense running in xcp-ng and that’s also Xen, right?
If I go that route I’ll probably choose VyOS. Going with Linux has the additional benefit of drivers - while the kernel is a mess at times, the community and corporate support here seems to be above what FreeBSD can offer.
Solution #2 - Dedicated hypervisor distribution
See previous point - we’ve seen pfSense running as xcp-ng VM, at least on the Son of the Forbidden Router IIRC. With passthrough. Which means it should be possible. I should possibly at least try to set it up and check whether similar problems persist.
I’m not a huge fan of those dedicated hypervisor distributions because of all them hiding the details and doing things “their way” as opposed to doing them “my way”. Even something as dumb simple as virt-manager can be a pain to work with the moment you have to do something outside the box.
Just so we’re clear, I have never used xcp-ng and I’m not dissing on it; I just have a feeling it’s not going to be my cup of tea. And if you think about suggesting Proxmox…
Solution #3 - Just go KVM
… I’d rather just stay with the original plan, but switch to KVM and libvirt, as I already have a ton of experience with them. I just really wanted to go with Xen initially for the added separation of dom0, but the more I work with it, the less differences from KVM I see. For instance, I assumed I wouldn’t even have to deal with “passthrough” and I’ll be able to just assign hardware to VMs sort of like I do vcpus or memory; but at least from the tutorials and documentations I went through I see no way to do it properly.
Side note: It appears to still be possible though, see e.g. this slideshow.
Solution #4 - Ah, screw it! (go monolithic)
If all else fails, I could just go monolithic instead. All the separation and security is gone, but at least it works, right? Right?
For now I’m pretty undecided, so I’m ready to receive feedback.
Edit: sorry for the typos, my eyes hurt already 
. Just re-read the post on my phone and corrected some, but there may still be more left.