GlusterFS sharing on CentOS 7: How can I setup NFS sharing when both Gluster focused options don't work?

Operating Systems & Open Source Linux
1

I’ve tried setting up both Gluster-NFS server, and NFS-Ganesha for Gluster. Neither work after following the documentation I’ve found.

For Gluster-NFS, I created a Gluster volume, then gluster volume set <vol-name> nfs.disable off.

I followed this guide for Gluster’s Built-in NFS server: http://serenity-networks.com/how-to-install-glusterfs-nfs-on-centos-7-for-virtual-machine-storage/

When checking the status, it states the NFS server for Localhost is offline/not running. After googling this issue, the normal problem is that NFS is already running for some reason. After disabling the regular NFS services and restarting the cluster of nodes, the NFS server still wasn’t running.

After installing NFS-Ganesha along with the Gluster specific package, I followed the documentation, but even after starting the service without errors and editing the config file, the share is never accessible.

I used this guide for NFS-Ganesha for Gluster: https://docs.gluster.org/en/v3/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

The documentation for NFS-Ganesha seems unkempt even though it’s supposed to be new and replacing the deprecated Gluster built-in NFS server.

I would just use the normal nfs-utils server, but the issue is that then that doesn’t access the gluster volume and instead only accessible the storage point directly, so the files aren’t replicated.

I can use the glusterfs client, but having NFS available matters for varying reasons.

1 Like
2

After some more tinkering, I have still not gotten any progress.

The goal for me is to use Docker Swarm volume mounting with the NFS driver for persistent and reliable storage for containers.

There are gluster drivers for the volume mount that exist but I’m not 100% on trusting that.

So I need to use NFS and Gluster simultaneously. When attempting to setup nfs-ganesha, this is the error I get when starting the service:

Sep 10 10:56:28 node1.domain.com systemd[1]: nfs-ganesha.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 10 10:56:28 node1.domain.com bash[28492]: prlimit: failed to set the NOFILE resource limit: No such process
Sep 10 10:56:30 node1.domain.com systemd[1]: Failed to start NFS-Ganesha file server.
-- Subject: Unit nfs-ganesha.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit nfs-ganesha.service has failed.
--
-- The result is failed.
Sep 10 10:56:30 node1.domain.com systemd[1]: Unit nfs-ganesha.service entered failed state.
Sep 10 10:56:30 node1.domain.com systemd[1]: nfs-ganesha.service failed.
Sep 10 10:56:30 node1.domain.com polkitd[704]: Unregistered Authentication Agent for unix-process:28474:51463736 (system bus name :1.437, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

This only happens after I add this to the /etc/ganesha/ganesha.conf file:

%include "/etc/ganesha/export.conf"

If I remove this line, the service starts up but obviously without any exports.

Looking here: https://github.com/nfs-ganesha/nfs-ganesha/wiki/Configurationfile

I see this:

Another configuration file can be included by:

%include " a/path/in/the/filesystem "'

Which appears to be what I did. My path to the export file is just /etc/ganesha/export.conf, and it is this:

# cat /etc/ganesha/export.conf
EXPORT{
    Export_Id = 1 ;   # Export ID unique to each export
    Path = "/bricks/Brick01/replicated";  # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = replication;
        hostname = "192.168.50.65";  # IP of one of the nodes in the trusted pool
        volume = "replication";  # Volume name. Eg: "test_volume"
    }

    Access_type = RW;    # Access permissions
    Squash = No_root_squash; # To enable/disable root squashing
    Disable_ACL = TRUE;  # To enable/disable ACL
    Pseudo = "/bricks/Brick01/replicated";  # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4" ;    # NFS protocols supported
    Transports = "UDP","TCP" ; # Transport protocols supported
    SecType = "sys";     # Security flavors supported
}

EXPORT{
    Export_Id = 2 ;   # Export ID unique to each export
    Path = "/bricks/Brick01/replicated";  # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = alternate-replication;
        hostname = "192.168.50.55";  # IP of one of the nodes in the trusted pool
        volume = "alternate-replication";  # Volume name. Eg: "test_volume"
    }

    Access_type = RW;    # Access permissions
    Squash = No_root_squash; # To enable/disable root squashing
    Disable_ACL = TRUE;  # To enable/disable ACL
    Pseudo = "/bricks/Brick01/replicated";  # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4" ;    # NFS protocols supported
    Transports = "UDP","TCP" ; # Transport protocols supported
    SecType = "sys";     # Security flavors supported
}
3

So I can’t get NFS to work with Gluster in the two provided ways. My idea is now to mount the Gluster volume using the Gluster Client on a different machine that manages NFS exports.

So while my file access will be going through 2 computers to get to the storage, it’s the only way I can think of to get Gluster Volumes accessible through NFS.

However, I’m running into permission issues. Ye Olde mount.nfs: access denied by server while mounting. My exports file is just /mnt/rep *(rw,no_root_squash). This is on CentOS 7.

Anyone know where I should go from here?

4

Whenever I’m working out a config in CentOS, my first step is to disable firewalld and selinux, get it working and then re-enable.

Is it possible either of those are giving you trouble?

—

edit

selinux autocorrected to demonic

1 Like
5

What is that?

I’m sure the firewall & ports are open and working. What I’m not sure of is SELinux.

Edit:

And it’s live and working. Huzzah. Just did this:

#echo "/mnt *(ro,crossmnt,fsid=root,sync)" > /etc/exports
#echo "/mnt/rep *(rw,crossmnt,fsid=<UUID>,sync)" >> /etc/exports
#exportfs -ra

Also probably helped that I realized I was trying to mount /rep-nfs when the share is /rep

1 Like
6

Is that with Ganesha?

1 Like
7

No, that’s with the normal NFS server.

Ganesha will not work no matter what I try.

So I’m not going to use Ganesha since I can’t get it to work and basically proxy Gluster access through another mount.

1 Like