I’m going to build another proxmox server, maybe to run VMs to experiment with such as NixOS, BSD, maybe OpenClaw, a bit of a dev machine. I have some 1ltr machines for running k3s and my home stuff, so it’s something to run chunkier workloads / desktops on.
I have spare
2 of 1TB Samsung NVME
2 of 1TB Corsair SSD
2 of 8TB Seagate Barracude 7200rpm HDD
and the machine has 64GB of RAM. I already have a TrueNAS server for actual network NAS usage.
I figured use one NVME for Proxmox, storing ISOs, I’m not entirely sure what else.
One NVME for the VMs OS to be stored on?
Then combine the two SSD for L2Arc and two HDD into an ZFS cluster?
I’ve gone over the L1T vids, some Alan Jude vids, asked Claude (it gets it a bit wrong it seems). Does anyone have a good plan for the best way to use all/most of these discs please? From what I was reading the 1TB SSDs maybe too big/wasted?
1TB is extremely overkill for just an OS drive. I’d setup a ZFS mirror of two SSD drives for boot, os, ISO storage and all other storage options. I’d also limit some of Proxmox logging and writing, to prolong the SSD usage (my 256GB boot SSDs used to average 2% usage per year in Proxmox before I limited writes).
Consumer M.2/SSDs are not good ZFS arc/cache drives. They help for small bursts, but you can destroy an SSD with all the writing.
With that said, I’d make another 1TB M.2 mirror to use for VM//Container storage only. That way every launches and updates fast. The M.2 interface is what shines here for VMs.
Lastly, ZFS mirror of the HDD big storage. You double your IOPS for lookups, searches, and file i/o. While write I/O speeds do not improve, reads will double!
Then keep adding HDDs in pairs as your expand the data storage.
Thanks for the heads up on logging, I should probably go adjust my other proxmox servers in that case.
So basically set up three mirror pools, ssd for host/iso, nvme for vms, hdd for anything that doesn’t sensibly fit on the vms. Nice idea, thank you for replying.
If they are consumer (EVO or PRO) drives be careful. Lots of reports of those choking on servers with long uptimes. The firmware seems to crash requiring a power cycle to come back online.
It’s not every drive, and it’s not all the time, but the incident rate is rather high.
I had this happen with one of my two 512GB Samsung 980 Pro’s. Once every 90-120 days it would just “disappear” from the host and would not come back without a proper power cycle. (a warm reboot wouldn’t do it)
See here for more details:
This particular thread was started by someone who had issues when using Windows Server, but mine had the same issue under Proxmox, so the OS doesn’t seem to make a difference.
The funny part is, I had a LONG history of using Samsung SATA SSD’s in server / ZFS applications without a single problem, but something seems different when it comes to their NVMe drives.
If the Samsungs you are talking about are actual Enterprise drives (usually models starting with PM, like PM1743, PM9A3, etc.) - however - disregard the above. Those are validated for server applications, and I have not heard of any problems with them.
nvme is the fastest storage you have, and you want to use it for bulk data storage and OS? It makes more sense to use it for VMs/containers, or at least the root/C drive of the VMs so it’s fast.
L2ARC is something that is very rarely necessary or worth it at your scales. It also requires some setup to avoid blowing up your system in creative ways. See ZFS L2ARC: The SSD Cache That Helps… Until It Hurts – cr0x.net
It is also going to chew the SSDs.
Use these for OS and iso storage (possibly together, in a raid if you want, the OS is not going to use a whole lot of space anyway)
The hdds can go in a zfs mirror and will be fine for data storage or for the “data disks” of the VMs/containers.