Multinode Storage System

Software & Operating Systems
, ,
1

Hey Yall,

I’m looking for a maybe different solution for my current setup.

Currently I’m running truenas with a raidz3 setup with 12x1tb drives and this worked great when I was just running a single server running some vms but now I have a 5 node kube cluster( Beelink SER5 Mini PC, AMD Ryzen 7 5800H x 5) and I want to decommission my big storage server while still retaining the reliability/redundancy of the truenas server. Is there a better way to store my media/files across the cluster without having to get a large drive in every system? Is there a raidz type of replication/parity for kube storage? Or should I just stick with nfs from the truenas server?

2

Linus did a server room update not that long ago that mentioned a server cluster.

You may want to read up on the various file systems mentioned there.

3

Have you looked at ceph? Openshift Container Storage Openshift Data Foundation IBM Storage Fusion Data Foundation uses Ceph on top of LVM to provide Block, File, and Object storage across the cluster. There’s operators for it and you can tune things like replicas and sharding to your liking

1 Like
4

ZFS itself offers no distributed filesystem. People usually go for GlusterFS or Lustre on top of ZFS to achieve clustering.

Ceph is a totally different beast, but will get you clustered storage. I considered Gluster, but will migrate my storage to a Ceph cluster within the next year.

Kubernetes Cluster and Ceph? Check out Rook, it’s a storage orchestrator for Kubernetes. Only heard of it so far, but looks like a good combination and easier migration with existing Kubernetes.

Latency and IOPS won’t be anywhere a single server, but that’s inevitable when doing things via network. But it’s scalable and HA, where single server isn’t.

Ceph has all the same goodies ZFS offers. replication/mirror or RAIDZ/erasure coding. Many other things will be familiar and serve similar purposes. WAL disk is kinda like a SLOG, MDS is like metadata special, RBD is where you get your block storage, and CephFS is for filesystems. But Ceph is object storage, so you’re very flexible in terms of object, block or file storage, be it S3, block storage or CephFS.

I’m in the planning phase and only ran Ceph in VMs for testing, but I’m going to attach nodes (money has higher latency than networking) to my homeserver and will eventually move my data from ZFS to Ceph, although I keep my TrueNAS VM and some disks for backup purposes.

5800HX doesn’t seem too bad, but Ceph is rather demanding for “just storage”, so expect some CPU load and memory (2-8GB per disk) for your OSDs (disk babysitter daemons). CPU is fairly moderate for HDDs, but you need a lot of CPU for SSDs (e.g. 2-8 cores per NVMe) and you want enterprise drives and PLP to get good IOPS. Good networking because every IOP and every byte of bandwidth is network traffic.