TrueNAS storage solution for virtualization environment

[Fabio_Bologna]

Hello forum,
I have some doubts regarding a new project I’m tackling at work this month.
We are running out of space on my old HP StoreVirtual SAN and I’m trying to figure out a budget minded solution to the problem in the topic title.
Basically what I have come up with is the following.

For the storage solution I wanted to go with TrueNAS, probably core because I don’t need anything exceedingly fancy).

I spec’d out a head server in the form of a Supermicro CSE-829U X10DRU-i+ equipped with 2 x E5-2667V4 CPUs because we use Samba shares in the office and I know good single thread power is basically required. I added 256GB of ram (16 x 16GB DDR4) so I can have a good ARC capacity. The HBA will be either a 9305-16i or 9300-16i, the first is like 200€ more then the latter but I heard the silicon is newer and way more efficient so I’m still considering what to do.
I will connect a JBOD box to the head server in the form of a HP M6720 24 x 3.5" box.

In total I will have 36 x 3.5" bays at my disposal.

I wanted to populate them as follows:
32x 4TB Enterprise SAS HDD to form into a single pool of 4 x 8 wide RaidZ1s
2x 960GB SSD for metadata stuff to make everything a bit more snappy
2x 480GB SSD for L2ARC

I also wanted to add a 16 or 32GB optane ssd for SLOG.

Everything will be connected to my hosts via a pair of stacked Dell PowerConnect 8024F so all 10G links.

In the pool I wanted to have 2 datasets, one NFS for VMWare and one SMB for our windows data sharing stuff.
Moreover I wanted to use the smb share as a backup destination for my VMs because as part of our working condition we travel a lot to startup the plants we develop the automation for.
So my idea was to have the developers work on the VMs in the VMWare infrastructure and once they need to travel abroad, grab the latest VM backup from the SMB share and play it with VMWare Player on their laptops so that they are always in sync with everything.

This is my plan at the moment.

I was also wondering about deduplication functionality but I have no idea what that would imply on the overall responsiveness of the system.
We normally have around 20 to 30 VMs active at any given time.

My question to you experts is would this solution be satisfactory for 10 PLC software developers and maybe 3 or 4 power users?
I want the machines to feel snappy but as I said cash is… well… not a friend in this period.

Thanks in advance for any suggestions

[gcs8]

Ohh boy am I about to rain on your parade. I assume if you are going after something as old as an E5 v4 you are getting it 2ed hand (eBay), so that’s fine, but what do you do if it fails? Does that HP StoreVirtual SAN have A/B controllers? Do you care if your whole storage/VM stack goes offline for 30 min up to 30 days? How critical is this? Also, unless I am blind or you swapped out the SAS controllers/expanders in that HP M6720 for SAS3 there is no need for the 9300 series of HBAs, You need an LSI 2308 with two SAS SFF-8088 cables for the HP M6720 and I don’t know what exact chassis you are looking at, but if it has the BPN-SAS3-826EL1 then you will also need a 93xx-8i for that and two Internal Mini SAS HD SFF-8643 to SFF-8643.

@Fabio_Bologna

32x 4TB Enterprise SAS HDD to form into a single pool of 4 x 8 wide RaidZ1s

I would advise against that, you would be better off with 5 x 6 wide RaidZ2s for a total of 30 drives leaving you with 2 bays for hot spares.

@Fabio_Bologna

2x 960GB SSD for metadata stuff to make everything a bit more snappy

Two is good, but 3 makes me less scared, if you oops your metadata vdev the whole pool is gone forever. Also, NVMe?

@Fabio_Bologna

2x 480GB SSD for L2ARC

NVMe? If you have to buy these new, I would push you to get more RAM instead, 32G DIMMs are not super crazy. It also takes ARC/RAM to manage an L2ARC.

@Fabio_Bologna

I also wanted to add a 16 or 32GB optane ssd for SLOG.

That’s a huge it depends, first off, needs to be 2 (mirror) for a ZIL (SLOG), If you lose writes in flight, you are going to have a bad time. Also, the Optane needs to be faster than your pool or it is a bottleneck and will slow you down instead of speed you up. If we do some quick napkin math, each vdev = throughput/IOPS of ~1 disk, and a 4TB 3.5" SAS drive is probably only good for ~150MB/s, so if you go 5x6Z2 your pool should hit ~750MB/s on a sequential write after you saturate the on-drive cache if there is no other reads or writes going on. So your ZIL should be good for at least 1GB/s to match that.

@Fabio_Bologna

one NFS for VMWare

If you are going to do NFS on ZFS, you are going to want a fast ZIL or it’s going to suck.

@Fabio_Bologna

I wanted to use the smb share as a backup destination for my VMs

What are you planning on doing here? Are you going to shutdown the VMs and copy them over to the SMB dataset? Or are you like using windows full disk backup to SMB?

@Fabio_Bologna

So my idea was to have the developers work on the VMs in the VMWare infrastructure and once they need to travel abroad, grab the latest VM backup from the SMB share and play it with VMWare Player on their laptops so that they are always in sync with everything

Ideally, when you are at a version/checkpoint that everyone should be synced up on, you should have a version controlled OVF/OVA that they are using.

@Fabio_Bologna

I was also wondering about deduplication functionality

Don’t bother, it is more involved than it should be. Though I have not tried using a dedicated vdev for the DDT.

@Fabio_Bologna

would this solution be satisfactory for 10 PLC software developers and maybe 3 or 4 power users?

That depends on what they are use to now, it might be ok or you may have to get SAS3 SSDs to put in the chassis with the BPN-SAS3-826EL1, so maybe don’t fill those bays up? It is a big maybe. I have no experience with PLC devs, but I would assume writing/compiling ladder logic is not anywhere as bad as some modern code compile requirements.

[Exard3k]

@Fabio_Bologna

I also wanted to add a 16 or 32GB optane ssd for SLOG.

This is old crap 1st gen Optane. Get a 800/960GB Enterprise NVMe for mixed use.

@Fabio_Bologna

I was also wondering about deduplication functionality but I have no idea what that would imply on the overall responsiveness of the system.

Don’t. It will just use memory for minor gains. Dedup ratios <4.0 are usually considered underperforming in terms of cost. You can activate it on specific datasets if you know the ratios are really good, but never do this pool-wide.

@Fabio_Bologna

32x 4TB Enterprise SAS HDD

4TB drives are a bit yesteryear and really expensive per TB. Unless you can get them @~15$/drive and get a handful of spares, I wouldn’t bother. 16-18TB is the sweetspot atm. And SATA work just as fine and are cheaper.

@Fabio_Bologna

to form into a single pool of 4 x 8 wide RaidZ1s

4TB isn’t much, but I think this is too greedy especially when using old 4TB drives. I recommend a good backup server. Otherwise use something like 6+2 RaidZ2.

@Fabio_Bologna

In the pool I wanted to have 2 datasets

You want more. Someone deletes something on his share and you rollback the entire dataset for everyone? No, you make a lot of datasets and rollback only what is needed. And also to tailor dataset properties to your needs and optimize

@Fabio_Bologna

So my idea was to have the developers work on the VMs in the VMWare infrastructure and once they need to travel abroad, grab the latest VM backup from the SMB share and play it with VMWare Player on their laptops so that they are always in sync with everything.

Copying a VMDK from dataset A to B isn’t a backup. Snapshots are what makes ZFS so easy when it comes to backups and rollback. Just send them to the backup server for a backup.

If VMWare requires having templates/copies in a shared folder, go ahead…no limit on SMB shares or datasets. Also hit the checkbox for “Shadow Copy”.

@Fabio_Bologna

My question to you experts is would this solution be satisfactory for 10 PLC software developers and maybe 3 or 4 power users?

Depends on what the usage is. But memory helps quite a bit with reads. The server uses like 4-5x the power of a modern low-cost/homeserver and is probably slower, so check ROI.

Performance relies on drives and what else is running on the server. If other stuff is using most of the CPU and memory, it won’t be great. Otherwise the network will be the bottleneck for sequential reads and writes. Which is all a server can hope for
Random I/O depends on drives and how good the cache hits (reads) and SLOG (writes) are.