I took a Dell PE R730 server with a Xeon E5-2690 v3, 64GB DDR4 and a Mellanox ConnectX 4 NIC. I installed a 500GB WD Blue SSD for boot and installed 4x WD Red PRO 8TB Drives. These are the 128mb cache and 7200RPM drives.
I installed TrueNAS Scale (First time user) and created a RAIDZ1 pool. I created an NFS share and added that to my proxmox.
Now, I have a ProxMox server v7.2 and it’s also a Dell R730 with dual Xeon E5-2690 v3, 256GB RAM and a Mellanox ConnectX 4 NIC. Both servers are connected to a 25Gbps switch.
I started migrated VM images to the NAS and noticed that performance was pretty bad. ±30MB/s. I checked the NIC in TrueNAS and that was showing 54MiB/s. So I did a iperf3 test between the servers and was getting 22Gbps.
I then created a smb share and copied a file from another PC (1Gbps) to the smb share and am only getting 70MB/s (the fastest I have seen yet). In the TrueNAS dashboard, I can see the zfs cache increasing, but even copying that file back, to the same pc, doesn’t give me a faster result.
Yesterday and today, I did some googling, but could not get this sorted, so now I am here asking for assistance?
You should overthink your Pool Layout, for performance it’s usually best to stick to the following formula if you’re going with RAID-Z due to the way ZFS was designed:
But you’d only want to choose RAID-Z2 if you want to maximize Disk Space.
In any case where performance matters - e.g. Shared Storage for Proxmox in your case - Striped Mirrors are the best choice since you can only get so good with RAID-Z Performance unless you have quite a bit more / bigger VDEVs.
So I’d recommend you configure your drives as 2x Mirror VDEVs.
A few more things to consider:
What type of VMDisks are you using? qcow2 allows for thin-provisioning on the NFS Share
Make sure atime and deduplication are off on your dataset
what are your dataset properties like? atime, sync, compression and stuff? Using a Zvol? If yes, low blocksize?
zfs get all poolname/dataset
Pool properties? ashift being important.
zpool get ashift poolname
Any suspiciously high CPU activity during transfers?
zfs cache is always increasing. And even if it at max, the previously written files always stay in the ARC (MRU).
I’d expect Gigabytes per second (which usually means == network speed) on that read, not some 70MB/s.
So that’s odd.
We have no data that indicates that RaidZ performance is causing this. 4-wide-RaidZ isn’t <70MB/s sequential read no matter how old your HDDs are, and we’re talking about cache hit here.
edit: my money is on sync for the slow writes, but don’t know what’s going on with the reads yet
I didn’t mean my reply in the sense “Raid-Z is the fault of all your problems”, but I believe there’s clearly more than one thing wrong here and that’s one of them.
One thing I’d like to add:
Please go to Reporting → ZFS and check out Arc Size and Hit Ratio while writing and reading a large file.
That way we can tell whether the Cache is getting hit as expected.
Maybe check if there are some settings regarding readahead or burst settings for NFS/SMB.
I’m running a small ceph instance at home and at first I got really bad sequential performance with cephfs and iscsi (much less than I single disk would achieve alone). After I increased the readahead size for cephfs client and burst parameters for iscsi client I got much better sequential performance.
Thank you all for the replies. I left the server over night and tried again today and for some reason it is a lot faster. 260MB/s when I rsync data to it and this is the kind of performance I was expecting with spinning disks.
