I plan to migrate my home server due to rising energy costs into a data center (we all know why…). I want to use some cheap VMs and slap them together to create a Kubernetes cluster. My only problem is, that I have to make data available for all nodes, so every pod can access it during is lifecycle. As the VMs should be disposable it is therefore not an option to store the pod data on their local storage. Instead I found a provider which serves huge online-storage for cheap in a similar region. They offer access to their “Storage Box” (Hetzer) using SSH and SMB with unlimited traffic (5TB for 10 bucks a month). My plan is therefore to connect the storage backends as follows:
But, there are some problems… I know that encfs is no fan of its underlying data to change, so at no point in time the same folder should be mounted inside two VMs (so for every Deployment must be an own folder - if it needs to store data). Additionally, as we are running over network I want to add a layer of cachefs to improve latency and save CPU cycles during the encryption. Once again, this FS must be properly unmounted before an other node can use the data.
Do you guys have any idea how I could convince Kubernetes to not only mount one layer of file system, but multiple ones for a pod (or a volume) and to ensure it is (un-)mounted in the correct order? How does it perform with permissions - I know the GitLab image for Docker is pretty picky about its storage backend, but I won’t be able to allow root access to the storage backend - as I don’t have it either. I know that creating an own share for every service I want to run inside the cluster is somewhat tedious, but I think that I have to - so I can ensure proper unmounting and syncing of files between the nodes…
Greetings from Germany! I hope you have some input
Hi! I know minio, but sadly not all applications are compatible with it - especially as I need to run database operations (would be a subset of GitLab) on it, which is explicitly not recommended to do inside the README.md of the necessary compatibility layer miniofs…
Furthermore, how would you propose to use an encrypted block storage here? After all, the block storage (files) must be available to all nodes and I would still like to add some level of local caching here - which would still require Kubernetes to somehow force a flush if the volume is not needed anymore… Adding to that the nodes must be disposable, so they can become unavailable at any point in time - meaning the files must still be synchronized to a common backend. How do I manage the “single point of truth” there?
And yes, the decreased pricing for the storage backends initially triggered my migration plans - as I need a lot of data to be readily available and backed-up at a low cost
I’ve just gotten an idea - of course custom coding is required, but potentially also for interest of you guys…
I could create a single point of truth by just creating a pod which must run on ONE node and which is responsible of transforming the node into a shared storage backend - even with caching and encryption. This pod would then export a NFS/? file server to which the other pods/nodes could connect as a normal volume using the inter-node Flannel-CNI. In case this node would go down, all operations would freeze (in case of a hard-synced NFS) until an other instance would come back up…
This storage-backend-pod could be a simple Python script, which creates (configured using environment variables) a mountpoint on the host system with the different layers active. Only requirement would be that Kubernetes must mount a volume of a cluster-service, which I have never done before…
