Only getting 500Mb/s copying between SAS3 SSDs?

Hardware Hub Storage
1

Hi All,

Building a new Homelab server in a Jonsbo N3

Specs are:

Jonsbo N3 Case
CWWK Q670 Motherboard
Intel i5 12500T
128GB DDR5
LSI 3008-8i SAS3 HBA
2 x NETAPP / SAMSUNG PM163aa SAS3 SSDs (patched so no 32k bug)

Currently doing some tests on an Ubuntu Live image before setting up proxmox fully and running

sudo dd if=/dev/sda of=/dev/sde bs=1G count=10 oflag=direct iflag=direct status=progress

I am only getting 500MB/s

10737418240 bytes (11 GB, 10 GiB) copied, 21 s, 517 MB/s
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 20.7646 s, 517 MB/s

I have also confirmed using the smartctl tool that the drives are connected in 12GB/s mode (single path not multipath) given they are using a breakout cable onto the HBA

sudo smartctl -x /dev/sda |grep "link"
    negotiated logical link rate: phy enabled; 12 Gbps
    negotiated logical link rate: phy enabled; unknown

sudo smartctl -x /dev/sde |grep "link"
    negotiated logical link rate: phy enabled; 12 Gbps
    negotiated logical link rate: phy enabled; unknown

Any ideas what I am doing wrong here?

1 Like
2

Whats the rated speed of those drives?
500x8 = 4000
4Gb/s between only 2 drives is nothing to scoff at. Thats already a third of the way to saturating the channel.
I dont think youll find any single device faster without going to nvme or cxl.

If this is indeed really far below the spec sheet speed of these drives, iā€™d start looking at your expander/jbod for bottlenecks or issues. Not all backplanes do channel aggregation perfectly.

4gb/s might indicate a pcie link issue. Check your bifrication and make sure you actually have at least 96gb/s (8 sas3 channels) pcie link.

3

Those drives arent NVME, so they will have significant performance bottlenecks compared to NVME drives which are simply more efficient at small file transfers and access times compared to older protocols. You are also using a very old HBA, which has significant random IO bottlenecks compared to more modern HBAs. The NetApp controller boards might also be limiting you, depending on what model controllers you are using. Good you already checked single vs dual port mode though, so that rules that out.

So is 500MB/s normal? IDK, but I already see at least two areas that mean you shouldnt be expecting even PCIE gen 3 NVME drive speeds from these.

edit: I see this page finally shows some specs on transfer rates for these drives:

What model size do you have? If you have the 7 or 15TB version then 500MB/s isnt too bad from the maximum rating it has.

2 Likes
4

That sheet says 450-640MB/s write.
500MB/s is within that spec. Id say its definitley within the tolerance given sas3 overhead.

2 Likes