Distributed file systems and Ceph

Operating Systems & Open Source Linux
1

So I am still in the process of redoing my homelab setup and I am still trying to work out how I am going to do storage. I want to have my storage related VM’s and my workload VMs separate thus NFS will be the backend. Additionally, each deployment should have its own storage space with no global permissions.

As far as I can tell, Ceph is a poor choice for a small 3 node setup. It has a lot of complexity for something small and the overhead is likely way to much. I want something that can mostly minimize network activity and that is simple to setup. I don’t mind a little data loss as long as I am not completely pinned by network overhead. I want something that can have a VM on each node and then maintain mostly up to date copies on each node. I am looking into possibly using btrfs send receive with just two nodes but it is tricky to do correctly as btrfs requires that you snapshot first then send.

Is there any software that is like what I am describing? I want modularity and portability.

2

Well ceph is exactly what you are looking for, all bells and whistles included. 3 node setup would have 1 node loss tolerance without issue, and data replica count is customizable.

If you running kubernetes then there are automation tooling available for deployment (rook.io).

3

If Ceph is too complex and you dont have the hardwaare, check out Longhorn if you use Kubernetes. Not that sophisticated as Ceph, but I heard about it being way easier and it still is distributed storage and a known name. I don’t have personal experience running Longhorn though, but people apparently like it.

Getting Ceph running is dead simple on Proxmox btw. Just point and click basically. And no need for NFS as your VMs run on RBD and you get CephFS for file(sharing). But all cluster storage has lots of network traffic because cluster, thats not overhead but necessity.