ZFS NAS + VMs - Please help to reach all feature goals with a given hardware configuration

Operating Systems & Open Source
1

Hi,

I’m looking for help squeezing every feature out of a given hardware configuration (every part is already at hand):

Its goals:

  • Saturate 2 x 10 GbE ethernet connections
  • Provide a Mirror on SSDs for the Host OS, ZFS “Caching”, Guest VMs (Router, VPN, “light” stuff) and around 500 GB of space as a network share
  • Provide a RAIDZ3 on 7 mechanical HDDs

3700X_64GB_X470D4U NAS_Overview
3700X_64GB_X470D4U NAS_Overview1766×1276 150 KB

  • It would be great if the VM Host could support AMD’s Secure Memory Encryption feature where the memory of host and guests is automatically encrypted separately to achieve a bit more separation, HWiNFO says the feature is active on the system.

Can you guys help me achieve all points?

  • I hate to admit it but I still “feel” most at home with a hardware RAID controllers and never really trusted myself going full ZFS

  • If I’m able to reach all of the goals (see a little more details in the graphics file), I’d be finally content.

1) Do any of you know “for sure” which OS makes to most sense to get everything done “the most proper” way on a single system?

Thanks for any input!

1 Like
2

FreeBSD 14 should do most of it fine, SME/SMV isn’t supported but it seems like Linux have some support at best and it doesn’t seem to be complete at least not in mainline.

3

I guess you also need a pcie slot for gpu to let the system boot.

What motherboard are you going to use?

4

Nope, the system has a KVM IPMI. Otherwise the dGPU would have been in the diagram.

That’s another reason why the dedicated HBA isn’t planned to be used for the fixed internal storage, makes it painless to swap it for a dGPU in case an actual modern GPU feature like media transcoding is needed for one of the VMs.

5

I’m aware that SME in the home user environment might be a bit much to ask but maybe it’s doable. Otherwise it will always annoy me at the back up my head :wink:

6

Just search for working examples and/or tutorials. Having a brief look turned up with very sparse information about it which makes me doubtful.

7
  • Not thinking about AMD SME for the moment, could Proxmox be doing everything else?
8

At present, I run TrueNAS Core on top of ESXi, passing through PCIe devices as necessary (HBA’s, NVME SSD’s, etc…).
I am looking at migrating TrueNAS Core and all of my other VM’s to Proxmox VE or setting up a dedicated TrueNAS Core box in addition to Proxmox VE. I will be swapping out hypervisors due to Broadcom’s acquisition of VMware. It is a complete fiasco - One where several folks are even calling them a ransomware gang.

Something to toy with would be to have TrueNAS Scale to do all of that. Scale can do file server, VM’s, and docker images too.

I do not know if this is still the case, but a while back someone on another forum posted their findings on the performance differences between TrueNAS Core and Scale - Core was faster at the time.

Some things to note:
-TrueNAS Core & Enterprise are both based off of BSD.
-TrueNAS Scale & Proxmox VE are based off of Debian.
-People have used Proxmox VE as a file server (I believe TrueNAS is better for this role. Others seem to agree too).
-raidz3 is similar to raid5 and raid6 except instead of one or even two drives worth of parity, there is three. (Parity is spread out and not dedicated to just one device.)
-ZFS has compression built in and can be changed on the fly.
-When using ZFS, if you want more speed, skip the cache and add more ram. Cache costs ram. (I just put up a bare metal TrueNAS Scale server for a friend with an Xeon E5-2680 v4, 4x 12TB HDDs in raidz6, and 128GB ram - it could handle pushing over 600MBps over 10Gb fiber without compression. It’d prolly go way faster with more drives and a slightly different drive config.)
-Using your boot drives for OS, Cache, Slog, VM’s, and as a share is possible - however (AFAIK), setup not easy and this not recommended. (Using them for OS, VM’s, and as a share should be fine though.)

This link has been very helpful for setting up Proxmox VE.

TrueNAS & ZFS Hardware Considerations.

Questions for you:
-What will the workload for this server be like? Is this going to be mostly sequential or random 4k? Read or write intensive?
-How is this data going to be accessed or exposed? Eg.: Shared out to other devices (like windows) through file shares (SMB) or mostly accessed by linux/unix/apple/other-devices over a network (ssh, NFS, iSCSI, etc…)?
-Which is a higher priority for you - Speed, capacity, or somewhere in the middle?
-Are there plans to increase the number of perma-connected drives in the future?

**I have had my setup for a while and there are experts on the subject of ZFS over on the TrueNAS forums.

1 Like
9

AMD SME x Proxmox:
https://www.reddit.com/r/Proxmox/comments/yr1r9s/is_anyone_using_amds_secure_memory_encryption_sme/

and

1 Like