TrueNas Scale Project

I was going to upgrade my ageing Synology DS1618+ but was struggling to justify the cost (Bang for Buck) without spending silly Money probably the DS1823xs+ or FS2500 and fill these with QVO 4TBs. Both similar Sequential throughputs Read: 4,722 Write: 2,696 (FS2500) Read: 3,199 Write: 2,687 (DS1823XS+). FS2500 is about £2500 ad DS1823XS+ £1700. When I read about TrueNas, I had just built a new Workstation so my old Threadripper 2950x, Matx x399M Taichi, Node 804 build was gathering dust. So I decided to give it a go bought a couple of Mellanox ConnectX 4 100GBE Nics for the insane price of £100 ea. Asus Hyper M.2 NVME enclosure £60. IMO X399m is one of the best boards ever Threadripper on a Matx with no compromise obvs only 4 mem slots(4 way controller) & 3 full x16 slots running at x16(normally Matx boards get cheaped out)Any way this has been a great success as I would be laying out for 4x4tbs qvos which are about the same price as 5 x 990 pro 4tbs the build cost is something like £300 :slight_smile: I have watched loads of vids on 100GBE and it becomes apparent that above 10gbe you enter a world of the Dark Arts. Where benchmarks and reality are in two different parallel universes :slight_smile: . My test file is 6x3GB files(great Album :slight_smile: ) So My Sequential 10GBE Benchmarks v Reality for the Synology are pretty consistent.


Synology Read
Synology Write
The poor write speeds are down to the Synology Processor. Now the TrueNas throws up some strange results achieving probably decent 30-40GBE results.

The strange result is that in reality the Write speed is quicker than the Read speed.
TruNas Read
TruNas Write
Which is bizarre. I only have 1x 990 pro 2Tb in my workstation which could be the issue(waiting for Samsung to eventually release their Gen 5 pro NVME).
I am not too fussed about the overall poor performance of the 100GBEs as I believe that is down to TrueNas not supporting RoCEv2 (RDMA) yet. This defo is a thing, my real world 10GBE transfers on the Synology were 700-800 Mb/s but upgraded to Win for workstations and get a solid 1GB/s now. So I have saved a fortune and got near enough the same throughput as the FS2500. Hopefully that will increase when I go headless and add another Asus Hyper M.2 to bring my pool to 3x RaidZ1 vedevs, a Gen 5 card in my Workstation and the holy grail of RoCEv2 support for TrueNas :slight_smile: Been a journey entering into the world of Linux learned loads, the only Real downer so far is not being able to create multiple gateways(tick box on Synology) but I have my workstation and Synology directly connected (which works for me), Rsync was a mission but works well to back up the TrueNas to the Synology. I feel like a Winner!

2 Likes