1: I want to create a HA cluster that will run some VM without paying.
2: I need automatic failover.
3: Share storage between nodes (2 nodes in 1 DC and 1 node in another DC. Latency between 2 DC is 30-50ms and only 1G fiber so don’t suggest iscsi/NAS).
4: Consuming lowest power possible.
5: Periodic automatic backup of VM.
6: 32gb ram , 20 thread and 1tb storage and 1G network is good enough
Preferred solution:
1: Using Proxmox and ceph.
2: Using MINIPC intel 13 gen with 2 2.5g nic
3: 2.5G direct connection between all the Nodes.
Alternate Thoughts:
1: Using XEN but not sure XOASAN (as far I know it’s paid)
2: VMWARE is not free.
What is your thoughts?
Why do you think it is a bad idea?
How would you do it more better way ?
Particularly why won’t you choose MINIPC and go for a server? (I don’t need PCI lanes. I just need 32g ram and 1tb storage)
I’m planning to do the same. Waiting for funds to get this going, buying multiple nodes + proper networking ain’t cheap.
Easiest way to deploy ceph and you get a hypervisor too. And File+Block+Object storage in a single package.
You want Enterprise drives for Ceph. There are models that have M.2 variants but they are a bit slower because smaller. And the OSDs generate quite some load and I’m not sure if a MiniPC can handle this without bottlenecking IOPS on the storage. But with a single 1TB OSD this should be fine.
Don’t expect much performance out of the storage, Ceph isn’t known for speed even with 100 OSDs and 64 core CPUs. With consumer NVMe drives I wouldn’t expect more than 100 IOPS on small random IO by default. But depends on a lot of factors, especially network (latency).
MiniPC limits expansion somewhat. With 3x replication on 3 nodes you get 1TB usable space. And you need some kind of boot drive for Proxmox. Ceph also needs some GBs for the disks each. May fit into 32GB (don’t know whats running), may be too little.
I suggest looking into “just Proxmox” or “Proxmox + dedicated storage server”. The VMs will still be HA (high availability), but the storage isn’t. Makes things way easier and faster without Ceph. You can use an external (or virtualized) storage server as shared storage for the Proxmox Cluster.
Otherwise you get a nice hyperconverged cluster just ready for new nodes.
If this is a node…forget about HA clustering. Network latency will kill everything. That’s 100ms for a round trip. That’s 10 IOPS assuming compute and writing to disk is instant. It will run as fast as 90s Floppy Disks.
A better solution than typical HA would probably be the master/slave model if we are dealing with two DCs that will talk via the Internet uplink and not a dedicated channel (even ignoring ms).
A lot depends on what the OP wants to do with this creation. Should it be efficient as one organism or just a matter of availability in case of emergency?
Certainly, building an organism based on various DCs that will talk via the Internet will cause lag and some instability in the worst case scenario.
It is possible that a better solution will be to have single servers or ecosystems per DC and keep everything as a VM and maintain synchronization, and in case of a problem, transfer the mass to the other ecosystem. It would also be good to have some form of loadbalancer on the front.
You need a 3rd node for quorum. 2 min replica will work but they are still distributed across all nodes in a balanced fashion. And all nodes have to wait for the slowest node to finish.
There are normal NVMe drives you get at any gas station these days, but they don’t have powerloss protection aka supercaps. As Ceph is syncing after each write (and is only fully committed once each nodes reports success), normal drives get a huge penalty because of this. Even enterprise SATA SSDs are way faster than NVMe consumer drives. A shame, but with the recent drop in NAND pricing, enterprise drives became very affordable. Still a gamebreaker for most.
You can mirror the cluster to an offsite location asynchronously, but a cluster has to be in close proximity between the nodes. This is for connecting multiple clusters.
Dc1 is the primary and as dc 1 has 2 nodes so they have HA. Dc2 node is only for backup in case there is fire or something in DC1.
Between dc1 and dc2 there is a direct l2 connectivity but rtt is around 30-50 ms.
In short if im not creating XY problem,
I want to have HA in dc1 and if dc1 goes down due to natural disaster i want dc2 to be active.
How to do it?
One idea just came. In dc1 a nas1 that will have iscsi between node 1 and 2 and in dc2 nas1 will be synced with nas2 that will provide iscsi to node3.
Without ceph.
Node 1,2,3 will be in a quorum with same storage path but different nas
Is it a good option?