So I have a large number of samsung pm9a3 drives that I am trying to optimize for 4k r/w iops but would also like some durability with raid 10 and finding that iops simply doesn’t scale. Here are the scripts I used (ansible doing mdadm stuff, running fio, etc) with consolidated results via gpt.
Point is, raid 10 doesn’t scale with high performance nvme and raid 0, with the obvious practical limitations of having large arrays due to failure, also doesn’t scale super well but is more inline with what I was expecting from raid 10 on the read side. I’d imagine it is only worse with raid 5. Anyways, what would people recommend for optimizing 4k r/w if I am able to get up to 16 of these drives with a 64c cpu?
SW raid like xinnor / raidix - Don’t know pricing / if I could even afford them. Bootstrapped startup here…
HW raid card like Highpoint - ssd7580B or one of those broadcoms (they seem lower performance)
Graid - Not considering for this setup but maybe in the future
I have 6 of these nodes running k8s in 2 clusters of 3 nodes each where I want at least one node to be as fast as possible since it will be the master node for a postgres cluster. Anyways, willing to get a card and rewire the server if I know it will get me some more iops…
Consider pull in each disk into individual tablespaces. That provides nice lineal scalability. Consider a stack of raid1 configs for redundancy.
The downside is a little scripting, e.g. to partition tables where each partition lives in a different tablespace.
Tablespaces are a good idea but would not work with my current plan of using the zalando postgres operator which doesn’t support that. If I end up running the DB in a VM then that is definitely what I would want to do though.
Yeah I saw that. In my use case my data can be recovered and so having raw performance does make sense even if it is a little unsafe. For now it is not an option but maybe later.
I’m honestly thinking I am just going to do raid 0 on groups of 4 drives at a time. Seems like the best value as far as getting near linear performance gains in iops while also fitting some of my application requirements for sizing. I was hoping to have durability with raid 10 but I am a small startup and can backup my data / have it running in HA configurations and make due with the small chance one of these nvme drives fails - albeit them being purchased off ebay from china - yolo startup life stuff…
