Hey all, looking for some advice as I think through my workstation replacement. This is probably going to be a bit wordy as I try to lay out what I currently have, and my current thoughts about what my goals are for the new system.
Currently I’m on the old X399 platform with a 2950X and use a HBA to provide connectivity to my drives. Currently all my storage is SATA and a mix of enterprise SSDs and spinning rust for around 6TB of total usable storage. I’ve been running Arch as my linux distrobution for almost five years on this machine with a storage stack of MDADM providing RAID1/mirroring of drives with LVM handling volumes and snapshots. I layer LUKS on top of logical volumes for encryption and I use borg to do block level backups of my volumes out to a NAS. This is a work machine with VMs for various clients and projects that I work on utilizing VFIO/Passthrough for my development environments.
My goal with my new system is to see if I can get more storage and have it be all SSDs, with a preference for NVME. I’m finding that I’m saturating my old SATA drives from time to time and that causes things to run slower than I would like. Since I am likely to stick with this build for the next 5+ years I would like to be using at least Gen4 NVME connectivity to the drives. I also would like to make sure the storage is solid (at least 1 DWPD) and has power loss protection. I’m also thinking about switching from MDADM/LVM/LUKS stack to a ZFS/LUKS stack. However there have been a couple threads about issues with performance around ZFS and I’m not sure that is the way forward. However a raidz1 pool would mean I could probably hit my numbers with around three drives. I would also have a separate set of mirrored storage that would host the base OS and be separate from the main pool.
Anyway I guess this is a long way of asking, if you wanted to have 8-10TB of (at least RAID1) NVME gen4/5 storage on a TRX50 platform how would you go about it? So much has changed and the fragmentation in interfaces across storage makes it hard to piece together something that seems doable.
I would buy one or more large capacity U.2 drives (6.4/7.6tb or 15.36tb). You can buy lightly used refurbs for around $100/tb or less. I’ve bought a bunch of 15.36tb drives for $1,500 each from this vendor before, they currently don’t have any NVMe 15.36tb drives in stock buy here is a 6.4tb (3DPWD) drive for $100/tb:
If you don’t want to mess with the MCIO or SlimSAS ports found most TRX50 motherboards, or M.2 to U.2 adapters (like the ones that come for free with retail packages Optane 905p U.2 drives), you can get something like the Sonnet dual U.2 PCIe card for $200. I’ve got one and it works, was able to stick 2 15.36tb NVMe drives and use them on an HPE ML30 server.
On board m.2 drives. These seem to range between 2-4 slots available, depending on the board. Depending on your use case, 2 slots with a dual 8TB setup could work, but 3 or 4 slots with RAID5 / ZFS z1 is probably what you want. Only problem is that 8TB m.2 drives cost €950 while decent 4TB drives cost €280. In USD that is $950 for 8TB and $210 for 4TB.
Buy a bifurb board like the Asus Hyper M.2 x16. Board is like $50, so if you are looking for $/TB this is probably the cheapest option. With this you can get 8 TB of raw storage for €600 / $500 and another €600 / $400 for 4x4 TB raw storage.
Buy a switched storage card, like the Apex Storage x21.
u.2 or u.3 route. You can get 16TB u.3 drives for €1.4k or ~€88 per TB. Biggest advantage to this, is that you will not be limited in size to the m.2 drives.
ICYDock route. If you have a spare 5.25" slot you can fit this little baby in it:
Best way I would say is to get four m.2 4TB drives in there and RAID5 them, that should give 12TB of storage which should be way more than you need.
