Liar!
Calling 6 nodes with 35TB of raw Enterprise Flash less fancy than my 2 node 12TB flash setup is an understatement to say the least
42TB going to 50TB actually. That’s just buying some storage. Starting from scratch makes it easier. You are getting similar performance with a 2 node cluster and it does scale up with nodes. You’ll see a big increase when you add the third. I wish I recorded the benchmark numbers, but adding a 4th and 5th made a big difference and I’ll see when I get to 2 OSDs in all 6. You’ve clearly given more thought to the details of the network adapters you’ve picked and maybe the NVMEs themselves make a difference than I’m accounting for. Compared to your latest machines I think I bought and outfitted all my Nuc 9s with memory and network for less with probably some left over. That went to GPUs and Storage.
I really like you using the long 22110 M.2 slots for the M.2 Enterprise drives. I played around with that idea too.
It’s not easy to find systems that take them. It’s one of the large advantages of the MS-01 clusters, although many people I see don’t understand that PLP add performance and it’s not just there for power loss.
I’m really jealous on the first benchmarks, especially the reads, both 4k and 4M. More drives and especially more nodes seem to just make things fly. 4M reads is what I expect bonded 10Gbit links to more or less max out on.
The reads can go higher, but to keep the 8x PCI slots open for GPUs or more storage I’ve sacrificed network performance somewhat. The dual SFP+ x520 adapters are running on PCIE2.0x4, so they can’t fully saturate 10gbe. This is where I’ve made some tradeoffs, as I think the performance is good enough for my goals. The problem with a homelab for a homelab;s sake is that you can chase numbers without any real need and I’ve tried to make some balanced choices for my project.
My thoughts of Ceph keep changing. I’m more of the opinion now that running it in a homelab for more than learning, four nodes should be the minimum, not three. That later is more like running raid5 and you need to keep enough free space on your other nodes to accomodate the storage if one fails. I think the performance can be ok with SSDs, and certainly you’ve shown me that making considered choices on hardware can provide better performance. There are details you’ve mentioned about network cards I didn’t really know about and I wonder how the ping times of my network affect my overall performance compared to what you are getting with the connectx-4. How much of it is the cards, settings, switch? I feel like that would improve my 4k numbers more than the 4M though. (still a mystery). Handling the cooling of 25GBe+ cards in small chassis like the MS-01 and Nuc 9 Pros isn’t easy.
That’s some awesome stuff. 250W isn’t bad for all the stuff you have mentioned. People here run single old Xeon servers that use more
I haven’t built a PC since 1997 and arguably still haven’t. Modern consumer motherboards left me really disappointed. Not enough PCIE channels to really be able to run Ceph well, and going to server oriented boards results in a large increase in power usage (mostly). If I did it again I might consider a single Epyc system. I didn’t want spinners or ZFS and finding something that would give me durable storage, GPUs etc wasn’t easy. Seeing your choices specifically for Ceph and considering power and noise is really useful.
Ceph does scale out, even in a home lab and adding more nodes will increase performance. OSDs to a point, but as you discovered with 4k IOPS CPU can be a bottleneck with fast NVME storage. More nodes helps a lot with data durability, as I can utilize my storage to a higher percentage and handle the loss of a OSD or a node easily. I expected to only use 3 replicas, but I’ve learned more about EC lately. I’m going to be playing with MSR and trying something like a 8+2, 7+3 or 8+3 pool. MSR can keep the number of OSDs used in a node to a max of two. So even if I have 3-4 OSDs in a node, if the node goes down, it won’t go past the limits of 8+2 and recover pretty quickly. That takes the 50TB of storage I’ll have with 8+2EC to 40TB usable rather then 16TB. (of course some will be 3 replicas for VMs). More OSDs on fewer nodes means one would need something more than +2. I’m generally a fan of using fewer machines if you can, but ceph does make having more nodes attractive.
Downside, you need a lot more network connections. My 10GBe switch does have 4x25GBe ports. If I do build a GPU specific box someday I’d like to see what type of bandwidth I can get with a native client accessing all six nodes.
I’m going to be quite interested to see when you add hard disks. I’m not really convinced you can get good performance without going with quite a few drives, but it always does depend on how it’s planned to be used.
I just use a decibel meter app on my phone for noise. Close enough I guess. As far as switch noise, you might have been better off with the CRS504 and using break out cables, if that was an option. You end up getting the same number of 25gbe connections, but cooling is better done in the CRS504 and by all accounts more quiet.
Hope I’m not hijacking your build thread with these two posts. I can most future posts elsewhere, but not sure I have much else to say. Hopefully someone will see different approaches as useful.