OS: Proxmox
First, I’m fairly aware this is overkill. I got the Optanes because they are on fire sale and I will build a proper setup at some point. This is currently a mix between home services and a learning environment. I can schedule backups to the Synology.
Equipment
Intel 12600k, 64GB (non ECC)
Two drives plus 10Gbit ethernet on a Qnap card in the primary video slot (8x), 2 in PCIe slots and 4 drives supported by the motherboard (8 drives total).
Optane 1600x 58GB, Optane 1600x 118GB, Optane 1600x 118GB, Optane 905p 960GB, Optane 905p 960GB, (5) Samsung Evo Plus 2TB.
NFS storage: 45TB on a Synology 1522+
Thoughts:
2x Optane 118GB OS - Keep 60GB or so as a separate partition for metadata
2x Optane 905p - Stand-alone drives minus 60GB for 3rd,4th metadata and 2x50GB for ZIL
First question is about your use case(s). Depending on what you plan to store and how to access it can have a relatively big impact on the storage setup.
Second question is about the motherboard that you didn’t mention. The 8 NVMe drives that you can connect can at max use 30 PCIe lanes (7x4 lanes for 905s and Evos +2 for a 1600x), no consumer mobo (to my knowledge) has that many independent PCIe lanes. Let’s find out how to best utilize the PCIe lanes of your mobo.
General thoughts pending feedback on use cases and mobo:
Optanes are overkill for OS drives
I personally don’t raid 1 my OSs at all. The reliability of Optanes may even convince you to forego that. But that is ultimately your choice.
If you only plan to use 58GB of your 118GB Optanes for an OS there seems to be a good candidate for a use case for the 58GB Optane…
Test data transfer speed with real data between this computer and your Synology. Size your ZIL for about 5-10s of that. Chances are that you may get away with a 10th or less of what you budgeted.
Nonwithstanding mobo and use cases, consider:
5x Samsung Evo Plus 2TB in RaidZ1
1x Optane 58GB as OS
2x Optane 905 partitioned as ZIL, Cache, Special device in to be determined sizes - the remaining capacity as stand-alone (or maybe raid0?)
Thank you. Yeah, there is probably some sharing on some of the slots. The Qnap card has a PEX controller, but it is still using 8 lanes. I have a Z690 TUF D4 motherboard. I’m probably going to do only 4 of the Samsung Evo drives and put in another computer. To be honest I can probably fit everything I want 100% on Optane drives but want to see how fast I can make it go. There is the possibility I’ll remove a few of those drives and repurpose elsewhere when I’m done testing. I’m doing mostly containerization with a few virtual machines.
What I want to do maybe when a decent Lenovo SystemX server hits the recycling pile at work is put out the 7.68TB 9A3 in my main machine, get 3-4 more and build a system off that. We can take anything except drives which they send out to be destroyed. I’ll probably want at a minimum 25Gbe to consider that usable over the network. It takes about 4-5 years for machines to be retired most everything has DDR3 even…
I think I’ll do
58GB Optane (or 118 and use some of the space for something else)
2x905p for ZIL/Special (rest stand alone)
4xSamsung