Proxmox can't connect to my nfs share on freenas anymore

kassim3 · March 17, 2017, 9:27pm

I updated my freenas 9.10 to corral and reverted back to 9.10 because i was having issues. After starting 9.10 all my cifs shares seemed fine except for the nfs one. I only have one nfs share which proxmox uses as storage.

I keep getting the error "storage is not online(500)". I googled around and someone stated to run the following command
nmap -p 111,2049 192.168.1.109 That ip address being my freenas ip, i received the following output:

Starting Nmap 6.47 ( http://nmap.org ) at 2017-03-17 20:31 GMT
Nmap scan report for 192.168.1.109
Host is up (0.00028s latency).
PORT     STATE SERVICE
111/tcp  open  rpcbind
2049/tcp open  nfs
MAC Address: 00:25:90:46:4F:01 (Super Micro Computer)

Then i ran this command:

pvesm status

output:

storage 'freenas' is not online
ext500     dir 1       480588496        35095140       421057748 8.19%
freenas    nfs 0               0               0               0 100.00%
local      dir 1        64935732        59817576         5101772 92.64%

When i run this command for the first time after restarting the nsf share in freenas:
showmount -e 192.168.1.109

output:

Export list for 192.168.1.109:
/mnt/Home/vm-storage/proxmox (everyone)

Then when i run it again i get this:

rpc mount export: RPC: Timed out

Any help would be appreciated

Levitance · March 18, 2017, 9:03am

Maproot User and Maproot Group are set? Or Mapall User and Mapall Group, depending on your setup. But if you do root/wheel for Maproot User and Maproot Group, that will take permissions out of the equation.

Also, if you just do

df -h

on your Proxmox box, what output do you get? I know that if those permissions don't get set, often you can mount the NFS share in Proxmox, but Proxmox can't do anything with the share so it reads it as offline. Here's mine.

Filesystem                    Size  Used Avail Use% Mounted on
udev                          9.8G     0  9.8G   0% /dev
tmpfs                         2.0G  9.1M  2.0G   1% /run
/dev/dm-0                     274G  121G  139G  47% /
tmpfs                         9.9G   28K  9.9G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/sda1                     236M   92M  132M  41% /boot
10.254.0.252:/mnt/CheetahZFS  561G  110G  451G  20% /mnt/storage
cgmfs                         100K     0  100K   0% /run/cgmanager/fs
/dev/sdb1                     394G  158G  216G  43% /mnt/iscsi
tmpfs                         2.0G     0  2.0G   0% /run/user/1000

I mean, technically I'm just running Ubuntu server, but it's all the same really. That 10.254.0.252 is the NFS share. If you don't have an NFS mount showing up in df, then we should try to manually mount it. Make a temp directory in /mnt, and go hog wild.

mount.nfs 192.168.1.109:/mnt/Home/vm-storage/proxmox /mnt/temp

See if it gives us an error. If it just mounts, then try to get a list of everything in /mnt/temp, and maybe even touch blah.txt just to see if you can create the file. This will guarantee that you have read/write access to the NFS share. If everything there works fine, umount /mnt/temp, and maybe try removing and readding the storage via the Proxmox web interface.

kassim3 · March 18, 2017, 11:33am

I have
mapall user = root
mapall group = group

ouput:

udev                   10M     0   10M   0% /dev
tmpfs                 3.2G  9.2M  3.1G   1% /run
/dev/dm-0              28G  3.0G   23G  12% /
tmpfs                 7.8G   43M  7.8G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda2             126M  283K  125M   1% /boot/efi
/dev/mapper/pve-data   62G   58G  4.9G  93% /var/lib/vz
/dev/sdb1             459G   34G  402G   8% /mnt/ext500
tmpfs                 100K     0  100K   0% /run/lxcfs/controllers
cgmfs                 100K     0  100K   0% /run/cgmanager/fs
/dev/fuse              30M   16K   30M   1% /etc/pve

I added verbose and got the following. i manually stopped it since it wouldn't stop trying:

root@proxmox:~# mount.nfs -v 192.168.1.109:/mnt/Home/vm-storage/proxmox /mnt/tmp
mount.nfs: timeout set for Sat Mar 18 11:21:29 2017
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot TCP port 774
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: trying text-based options 'vers=4,addr=192.168.1.109,clientaddr=192.168.1.50'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.1.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.109 prog 100005 vers 3 prot UDP port 774

I logged into my arch linux partition and my cifs mounts weren't mounting as well now. IT does work in windows though. I get the following error when trying to mount it manually:

mount error(115): Operation now in progress

This is how my fstab is et up for the cifs:

#CIFS mounts

//192.168.1.109/Games  		/media/Games cifs credentials=/home/kassim/.smbcredentials,file_mode=0777,dir_mode=0777,uid=kassim,gid=dev 0 0

//192.168.1.109/Media  		/media/Media cifs credentials=/home/kassim/.smbcredentials,file_mode=0777,dir_mode=0777,uid=kassim,gid=dev 0 0

//192.168.1.109/Development	/media/Development cifs credentials=/home/kassim/.smbcredentials,uid=kassim,gid=dev 0 0

//192.168.1.109/Software	/media/Software cifs credentials=/home/kassim/.smbcredentials,file_mode=0777,dir_mode=0777,uid=kassim,gid=dev 0 0

//192.168.1.109/kassim		/media/kassim cifs credentials=/home/kassim/.smbcredentials,file_mode=0777,dir_mode=0777,uid=kassim,gid=dev 0 1

Levitance · March 19, 2017, 8:48pm

So it's having trouble mounting nfs. We've got a protocol not supported message.

I'm stumbling across this Stack answer.

Makes me wonder if your FreeNAS box got bound to presenting NFSv4 only or v2 only or some crazy thing like that. Now that you're back on 9.10, the easiest thing to do may be to nuke your NFS share and recreate it. The CIFS shares can be left alone since they are clearly working properly.

kassim3 · March 20, 2017, 4:30pm

Oh yeh, i didn't even try recreating my NFS share....I'll try that as soon as i get home. However i don't understand why linux can't ping my freenas server and causing issues mounting my cifs share maybe i'll need recreate the cifs share as well?

Levitance · March 20, 2017, 4:55pm

This is very odd. So you were successfully able to nmap your FreeNAS box from your Proxmox box, and you didn't use a -Pn switch, which means that nmap did indeed ping the FreeNAS box. Else it would have errored out, asking if the host was up. You also got an open on rpcbind and nfs, which means you don't have a firewall in the way. That is damn peculiar.

Are these boxes on a dedicated network connection to each other? Is jumbo frames setup on one machine but not the other?

kassim3 · March 21, 2017, 12:23pm

Yeh all the machines are connected via switch. I'll need to check if jumbo frames is set up i remember playing around with that around a year go but i don't think i kept it. I think it is an issue with my linux set up because the linux VMs in proxmox are able to access my cifs share successfully.

I stil haven't recreated the NFS share hopefully i'll have time to today and get back about that

Thanks for helping me btw, i appreciate it :)

kassim3 · March 24, 2017, 10:08am

Jumbo frames is not set up on any machines and recreating the NFS share didn't work. Any other ideas?

I'm guessing the network configs on Arch is messed up or something. I'll have to investigate this weekend if i have time. Any idea where i should start looking first?