I’m experimenting with GlusterFS using some old desktops and hard drives to see how it works and potentially to use it as an NFS share for random other things I’m experimenting with (docker swarm for example).
I figured I’d ask the community to see if anyone with experience using this knows where I can find best practices beyond what is said in the Red Hat and GlusterFS wikis.
Also ideas for cool things to do would be neat. One thing I’m thinking of is to use it as the host for the images I use when I PXE boot systems and install other OSes.
In RAID 1, the whole file is kept on every storage device in the RAID. However, you get increased Read performance for operating systems that are smart enough to utilize it with decreased Writing performance.
The question, does GlusterFS do something similar with replica volumes? I realize there are striped volumes, but I’m asking if replica volumes increase read speeds or no?
It doesn’t increase read speeds, it just decreases your failure domain.
You can distribute the replica volumes to the various clients and try and balance out the network traffic.
GlusterFS has been pretty great to me as long as I’m not serving JIT code (like PHP, or JIT NodeJS). It doesn’t handle small files well to this day, and no amount of fiddling seems to help that. I think it’s just the nature of the beast.
The other issue I’ve run into with GlusterFS is that npm (NodeJS) tends to employ FLOCKs when installing packages. So, I’ve had to abandon all GlusterFS storage for anything running NodeJS. PHP isn’t a fun experience at all unless you’re doing some sort of AOT strategy.
The long and the short of it is: GlusterFS is really good at the same things NFS is good at, plus it lets you have a smaller failure domain if something goes wrong. If NFS isn’t good at it, you need to find yourself a block solution.
Also, if you’re using Openshift, the GlusterFS integration is saWEET.
Things that stand out as “gotchas” are:
Make sure the brick is a directory inside the mount point of your block device, and not the root of the device. So if you mount /dev/sdb1 to /var/brick, make /var/brick/brick1 your brick location. This keeps the node from trying to use the root filesystem as it’s brick location in the case of a failed mount.
options: performance.cache-size “256MB”. I wouldn’t go much beyond that since it seems to have diminishing returns.
Your firewall WILL get in the way. Each volume runs on its own port starting with 49152. Each new volume will increment the port number.
Shows it pointing at a single node in the cluster for the mount. Is it just as simple as setting up a round robin DNS? Or would it be better to set up a single server that monitors the gluster nodes and points the mount to whichever is up at the time?
GlusterFS has a mount option called “backupvolfile-server”, so you can do something like this:
mount -t glusterfs -o backupvolfile-server=gluster2 gluster1:/volname /mnt/gluster
And then you can try and randomize the order in which people mount. So some folks would default to gluster1 and other would default to gluster2. If it were me i’d just have a python script run in the post-install and just rand select from an array.
Additionally, I’d make sure I had 3 nodes. This allows for self healing to work properly. I generally go ahead and create my volumes as replica 3, other folks do replica 3 arbiter 1 to save on disk space. The choice is yours.
BTW, even docker itself has backed kubernetes over swarm, so if you start playing in the container world I’d start there. Redhat has a free release called “origin”. It’s super good, and takes out a lot of the hassle of setting up a vanilla k8s cluster.
EDIT (again I know): Ok, so there was a headline i saw the other day of docker dropping swarm for kubernetes. That seems to have been a bit sensationalist. I still see Kubernetes job postings a lot more than swarm, but it doesn’t look like swarm is going anywhere.
Hmm. Not exactly ideal. I’m actually sitting here wishing GlusterFS had Docker Swarm’s router-mesh functionality.
Oh well. Round Robin DNS it is. Thanks.
Yeah, that’s what I’m working on atm. I’m experimenting with all 4 types of volumes. Distributed, Replicated, Dispersed, and Striped. I realize Striped is being deprecated for Sharding, but I’m using the long term 3.12 GlusterFS release and sharding is still experimental in 3.7 so yeah.
Yeah, we were noticing that when discussing hosting options with vendors that most use Kubernetes. I know Docker Swarm Engine and the Docker Swarm are two different things of all things. So it may be rebranding for them? Not sure.
Gonna be looking into Kubernetes after I’m done using Gluster and setting up an FTP and NFS share for PXE booting and docker container persistent volume storage later.