Rsync / scp 150GB files starts fast slows down to crawl after 10 min

Software & Operating Systems Linux
1

I am trying to copy 150GB .tar file from ubuntu 18.04 to my Mac on my local network. I tried rsync and scp. Coping starts really fast but after about 10 minutes it slows down to crawl. Im talking orders of magnitude slower. I did some googling online apparently this is a known issue with linux. Surely this can’t be right. Is there no way copy a file over local network.

I copied from my Mac to Threadripper 2950, 128GB RAM, 2xEvo pro in RAID 0 Ubuntu machine over WAN in like 2 hours via scp.

Coping it back from ubunutu machine to Mac, current estimate over 40 hours.

This is completely insane. Surely there must be a workaround. What am I doing wrong? How do I copy that file over the network efficiently ?

UPDATE: Possible solution found here


So far holding constant speed.

tar zcf - bigfile.m4p | mbuffer -s 1K -m 512M | ssh destination "tar zxf -"

2

I’m gonna need to see some source on that. I’ve done waaaaay more than 150GB at a time and it’s been fine. Largest single file I’ve transfered over SSH was ~8TB. Went at 108MB/s the whole time.

Are you on wired or wireless?

What sorts of speeds do you see?

How’s your CPU/memory usage looking?

Is there other stuff on your network that’s causing it?

1 Like
3

Here are the post:

Mac is on LAN
Ubuntu is on WAN

starts 40Mbs
after 10 min its 900 Kbs

4

Nothing on that page has any bearing over this situation.

That’s about kernel buffers filling.

Are you copying to a USB drive? I thought you said you were copying to SSD…

But yes, the best way to send a file is to compress it.

Absolute fastest would be this:

zstd -T0 -16 $file | ssh $remote_host zstd -d - -o $outfile
2 Likes
5

the fastest way is to use a combination of netcat and gzip with the pigz lib.

The ol’ razzle-dazzle tar pipe.

http://toast.djw.org.uk/tarpipe.html

The neat thing about this is that you can do it on a hard drive that is full as it will compress and stream that on the fly.

1 Like
6

Should have clarified that I felt that encryption was important.

7

It should be possible to tunnel that through ssh?

But it would look like a very fun command to type out.

8

That’s functionally what cat | ssh cat does.