the nodes are same layout 4545p EPYC4000DU4 the ram will be 2 to 4 sticks either 48g or 64g sticks of ecc ram. The hard drives will all be 1.92tb with dual sata ssd boots zfs mirrors and 4 nvme ssd. For the 8 remaining lanes I’ll be throwing in dual port 100g nics.
My main struggle is can I break up the boot drives and mirror them across all nodes for HA and can I just use ceph for the two data bases. Do I need to seperate it at a hardware pool lvl IE two osd per db per node or just add separate “folders” mounts? and do redundant sql servers across the nodes.
I like ceph and I like it even more if when I drive fails you just hard remove the old osd once its flagged/locked out. I can drive update easy enough by just dropping them one at a time. If a node goes down and you fix say the network cable whatever. Do you have to drop all of them on the node and redo to bring it back in once its flagged as gone? They dont seem to be answered. It might be but I’ve read on here were someone does it as it is easy for them?
Ever used PXE / BOOTP? I set that up on my router VM for my local Proxmox Ceph cluster years ago. Note, you want a dedicated VLAN/NIC for this, and a dedicated RPi if possible so there’s always uptime even when your router reboots. Sounds like you have secondary NICs too, so just use the 1 Gbps nics for the PXE boot.
Yes and no.
Yes that you should be able to taylor the policies to meet your write and replication/read status.
No that a default cluster most likely isn’t the best config for this. You’ll need to dig into the best advice for caching and write-thru configs for whatever particular DB you want to host (rbd_cache_policy vs persistent cache, and on what SSDs vs enough of a memory pool? etc). Some relational DB engines don’t like delays in rowlocks, and want immediate reads upon writes.
Relational databases need very special handling on a CephFS cluster. This is where I liked GlusterFS because of it’s localization of data writes (which you can force in CephFS as well) - but now it’s abandoned and therefore deprecated from Proxmox.
I had a pool of 4x SATA6 SSDs per Ceph node and it screamed, once configured properly.
Nope green but old green. I remember the days when DOOM demo was on floppy and what not. My life took me down a different path but its starting to come around to where I get to do both now at least at an inhouse none critical lvl. I’m more hardware decent easier to get then the software side. Yes the dual 1g nics will be to separate the two areas IP wise and allow them to float around if they need to be moved for whatever reason. The two 100g ports “add on card” is for the back end stuff. They are mainly point logging databases. We had issues with some old spinning disks and we went consumer nvme just to test the theory and well now I am putting the next stage in place for it to hopefully run for another 10 or so years. Total data wont be more then 1tb if that each db. The stuff is overkill and more for ease of use on our end.