I like to start off by saying I’m pretty new to Proxmox and Freenas. I’ve been messing with settings, making vm’s, breaking things and learning lol. The issue I’m having is I have a nfs share on Freenas that Proxmox is using for a Nextcloud VM. I also have another NFS share for backups that Proxmox also uses. It seems that nfs is way slower than iSCSI, which I also messed around with.
I did some reading and at least part of the issue is NFS wants to sync for data integrity, and Freenas which is of course ZFS also wants to do this. I’d rather ZFS do all the syncing since it is the backbone of the storage. Can anyone point me in the right direction for optimizing the Proxmox side of this. Also if I’m all sorts of jacked up please let me know lol.
In the future I’d like to have more ssd storage on the Proxmox host and then just set up backups to Freenas but storage is expensive lol. Thanks in advance for any help!
::edit:: Here is an example of speeds, this is right at an hour of progress. I should add the Freenas has two 10g nics lagg in LACP and proxmox has four 1gig nics also aggregated in LACP. Not that it maters much because 10g switches are also expensive lol
I’m just gonna follow for informational reasons. I too just installed proxmox and freenas. But you got farther than I have. I have yet to get iscsi going successfully. Not enough free time in my life
yeah, what @ulzeraj is talking about is a SLOG (Separate LOG devices). Freenas (really ZFS) writes synchronous files to a ZIL (ZFS Intent Log) before writing to main disk. When all you have is HDDs though, the ZIL is stored on the main pool causing double-writes, which can hurt performance. Adding a SLOG can claw that perfomance back.
If you don’t want to buy a new SSD for this, could turn sync off at the ZFS level or the NFS level. If you’re sharing a whole dataset, you can type zfs set -o sync=off <dataset> (might want to double check the exact syntax) to turn it off at the ZFS level. This way, whenever the NFS server/client calls the sync() function, ZFS will lie and just return true or whatever and write the data once.
I’m not sure if async nfs is supported officially with FreeNas but if you remount the rootfs as rw (sudo mount -o remount,rw / I think), then edit the /etc/exportfs file, you should be able to add async as an option. Then remount as ro and restart the nfs service throught the GUI. I’m not sure if that will survive upgrades/reboots though
That doesn’t add-up. NFS and ZFS sync are two different things. Without NFS sync, the source will think the file was written to the NFS server long before it actually has been. If the NFS server suddenly loses power after soon after, the data is just gone.
Now, if you’re ONLY using the NFS for backups, then it’s understandable you’d want higher performance and not be concerned if the data is lost sometimes. Then you can change sync to async in /etc/exports.
Thanks for the info. I’m not real keen on turning off sync so I went out and picked up a 250gb nvme to add as zil device, better safe than sorry lol. I was also bummed that you cant take snapshots in proxmox if it resides on an nfs share. I’m sure there is more too it than that but no easy button. Although none of this has been “easy”, it wouldn’t be fun if it was.
