TL;DR - recommendation for storage config (type and layout of NVMe/U.2 drives to equal 8TB) with TrueNAS Scale for an engineering firm (85-100 workstations connected over network with primarily CAD files being stored/edited on the fileserver)
Hey everyone - I have an IT client that I acquired in the past couple of years. They had existing infrastructure and I am slowly replacing and upgrading their environment, as they are rapidly growing. I started out managing 25 PCs and 1 Windows server for them. We are now up to ~85 PCs, and a domain controller/fileserver, and some VMs for apps and a postgres database (Quickbooks, and a few other apps).
Their current Domain controller is running in Hyper-V with primary storage being a couple of Samsung 870 EVO SSDs in RAID 0, server up via SMB.
I want to move them to TrueNAS Scale for the storage aspect (with connection to Active Directory for permissions management), and something with lower latency and better durability than the 870 Evo drives. Any suggestions on a pool layout and type of drives?
Well, first stop is determining how valuable their files are. (I suspect to them, quite a bit ) Depending on their answers you may want to opt for a high-availability cluster or a simpler RAID1 server+failover setup.
Last year LTT did a 2-part series on NetApp stuff and how to circumvent their “NetApp-tax” by using regular hardware controlling the NetApp SAS-based disk-shelves. Might be an interesting refresher for an IT guy
Alternatively, consider NVMe-based hardware from the likes of SuperMicro, 45Drives and Mikrotik. In the server market, EPYC still has the upper hand, for now. Also consider switching to 10Gb/s networking and a 100-400Gb/s backbone between server/NAS and various switches.
If you only want 8TB then you probably dont need a real storage chassis with a backplane meant for a lot of drives. Something that can have a 9500-16i put in and connect to 4 Kioxia NVME drives would work. Probably something like four 6.4TB drives in a “raid10” type config in TrueNAS. Likely the KCD81VUG6T40 model. It gives you 12TB of storage but the next model down is too small.
Probably + additional replication. + a spare ready.
HDD (cold) backup + off-site/cloud storage. If assuming 3-2-1 principle. Depends on policy. But with existing 870 RAID0 I doubt there is much present
I’d certainly plan for scale-up and/or scale-out, so plugging in new drives is easy and doesn’t require acquisition of new hardware. Rapid growth can result into exponential increase in storage needs over time.
\is this file-only or how important are block and object storage or possibly will be? Not all storage systems can handle all of them and migrations aren’t fun or cheap
most IOPS get eaten up by networking. So keep an eye on that. Faster drives won’t solve network latency
Yeah it makes me wonder if an object storage system like Ceph or Minio might start to make sense at some point for some of this. Seems like it might be easier to scale. Maybe?
If data is important to you, never use a RAID level with a 0 (zero) in it. It gives you a redundancy of exactly one drive and only if you’re very, very, very lucky a 2nd failure doesn’t destroy your data. If you have 4 drives in an array, use RAID6, so you have a true 2-disk redundancy. On modern CPU’s with their enormous compute power, the write penalty is very much reduced over Xeon, Opteron and Celeron days. The 6TB Kioxia drives you mentioned still give 12TB capacity in RAID6, but any 2 drives can fail and you’d still have your data. Whereas in a RAID10, after the initial drive failure, you have a 67% chance the wrong drive fails out of the remainder so your data is lost.
Whereas in a RAID6, after the initial two drive failures, you have a 100% chance the wrong drive fails out of the remainder so your data is lost.
If redundancy is too low, use more redundancy. 3-way mirror/replication isn’t an uncommon config. But we homeserver cheapskates tend to go low on redundancy in general. I still remember that guy with 20TB HDDs and a 22-wide Z1
And that’s why you have spares ready. To never get into trouble. And NVMe mirror resilver are merely minutes. By the time the admin checked the mail, pool is healthy/online again
With Ceph it’s far easier. If something fails, the cluster automatically replicates the missing data to other disks as long as there is free capacity. So checking for free space is all you need on a disk level. And very fast.
8TB useable seems low to me. I cant go into details, but my fleet is at about double that and the incoming stuff is making the storage requirements tick up very quickly. Unless they already have a PDM going, expect one to suddenly appear and demand 1TB for files + 1TB for database
Having people work off of the server is good. Prevents all sorts of “coffee in the laptop killed my work, bring it back!”-requests
Since you are in reach of DFS, look into it:
Advantage in M$-land is that you can have multiple machines provide the same data in the background, meaning one can go down due to fault or maintenance and the users just keep happily working along.
Migrating into a DFS-deployment requires a lot of legwork and talking processes/workflows through with “people on the ground”. There is a lot of links (= folders on the network people point various applications/shortcuts to) that will suddenly stop working unless the change is well planned.
Thanks! One thing I neglected to mention is that I have a BCDR appliance in place currently which is taking hourly snapshots that can be virtualized locally, and is pushing those to the cloud as well, where it can also be spun up.
I did watch that video from LTT when they released it. Will go back and revisit it. Appreciate the insights!
The DFS solution is very compelling! I will definitely look into that! As for the 8TB, I mean to say starting with 8TB with room to grow. Trying to communicate I wasn’t looking for a 100+TB solution. I have a 12 x 3.5" bay server I can add spinning rust to for less actively used archive data, but I want the current project data to be running on flash.
Working with data locally is pretty much a non-negotiable for me. Even with endpoint backups, I can’t control the end users being morons. So, I know I have hourly snapshots of the server, and can easily restore files there. Makes life much easier.
I am meeting with my client on Friday, so I will bring up the PDM. I know that we could almost immediately implement Autodesk Vault Basic. I greatly appreciate you mentioning a PDM solution, as well as the other insights!
) Depending on their answers you may want to opt for a high-availability cluster or a simpler RAID1 server+failover setup.
It gives you a redundancy of exactly one drive and only if you’re very, very, very lucky a 2nd failure doesn’t destroy your data. If you have 4 drives in an array, use RAID6, so you have a true 2-disk redundancy. On modern CPU’s with their enormous compute power, the write penalty is very much reduced over Xeon, Opteron and Celeron days. The 6TB Kioxia drives you mentioned still give 12TB capacity in RAID6, but any 2 drives can fail and you’d still have your data. Whereas in a RAID10, after the initial drive failure, you have a 67% chance the wrong drive fails out of the remainder so your data is lost.