[Solved] Flash drive high IO Wait

Im running Fedora 26 on a LUKS encrypted 128GB Samsung Fit USB 3 flash drive and it seems excessively slow. Ive noticed that on my cpu meter the IO wait is often very high, especially when doing anything that involves a large amount of semi-random IO like updates or when its booting. Watching disk speeds during these periods show only 1-2 MB/s of activity yet tons of IOwait. I get that flash drives are slow and running an OS off them is a terrible idea but ive had better speeds off a class 4 SD card so something is definitely off. I used a SanDisk before and it was quite a bit better so Im starting to wonder if it has something to do with the way the drive handles USB transfers. My other theory as to why its so slow is the system its running on does not have a lot of ram (4GB) and the cache is always full so it pretty much has to read from disk every time, however this still does not explain how slow it is.

If anyone has had a similar experience and has any ideas on how to make it better that would be greatly appreciated. I also dont mind providing any additional info if necessary

Flash drive speed test:
Sequential read - 100MB/s with some random drops
Sequential write - 80MB/s with random drops after 1gb written
Access time - 2.7ms

Well yeah. You’re running it off a flash drive. These situations are only meant for installing via a live environment, not running the system.

The buffer gets full and this is why speed tanks.

The short answer is; you can’t. You need a faster flash drive with more buffer space. It you must run it off of that specific flash drive, then you should run a more minimalist distro without a GUI.

Suggestion

Have you considered just making an external M.2 SSD in an enclosure. These aren’t that much larger and will give drastically better performance, since they were designed to be used this way.

3 Likes

+1 for M.2 SSD in an enclosure.

Even the “fast” Flash Drives don’t have shit on an M.2 soldered to a USB cable. (disclaimer: don’t actually try soldering USB directly to M.2, that’s not how it works)

I use one for when I have to use a shared machine that doesn’t have Linux on it.

1 Like

Wouldn’t they be bottlenecked by the USB interface bandwidth? Roughly 100Mb/s ? Which is not much different from the 80Mb/s the OP is getting…

Uh, depends on the USB protocol. 5gbps is USB 3.0, correct? meaning you should have a theoretical max of 640 MB/s. (or maybe I’m doing my math wrong)

1 Like

I stand corrected. I still had some HDD numbers laying around. Not factoring the SSD potential R/W

Yeah, I’ve got one of those WD easystore 8TB drives and it maxes out over USB3 at around 120 MB/s when using DD.

I haven’t benchmarked my USB M.2 disk, but when I was flashing an image to it, I got 250MB/s write on a shitty, worn out SSD. (It had been previously used as a zpool cache/log disk)

1 Like

The issue with using an external ssd is still size. Due to the way its being used, it has to be tiny or its going to get broken or unplugged. Im using it on a school laptop which gets carried around between classes.
I did at one point have a SATA drive hooked up in a USB enclosure but there were far too many close calls with the connector potentially getting smashed.

Try disabling swapping, which may be painful on a 4GB laptop. If that fixes it, set swappiness so it swaps a lot less. You can also boot with “scsi_mod.use_blk_mq=1”. I would also try booting windows on the same flash drive and see if it’s still slow.

And of course you can always image the disk then install linux to the internal disk, unless you need it for school.

Corsair Flash Voyager GTX 128GB has very nice write speeds

Its not my device so unfortunately I cant overwrite the internal SSD. Ive set the swappiness to 1 already so it only does it if absolutely necessary but ill have to try that kernel argument

Its usb3 so with a throughput of 5Gbps and using UASP the bandwidth there isnt an issue. Im thinking it may be a crappy flash controller inside the drive

1 Like

I have the 64GB version. ill have to see if it has the wierd performance drops that the samsung does. I was originally using a 32GB SanDisk drive but that runs out of space rather quickly. Apparently the larger size ones get just a bit hot so I didnt think that was a good idea going in a cloth case.

Due to how NAND works, basically like a striped RAID of memory chips, larger sizes do tend to be much faster up to a certain point. But I agree your problem isn’t throughput, it’s something with the parallel accesses or locking. That’s why I suggested trying Windows.

I cant really wipe the drive and install windows at the moment. It seems to be small file access that is the problem as it bogs all the way down to well under 1MB/s. You are probably right about it being a linux specific issue so I may have a play around with a few different kernels because I know there has been usb speed issues in the past.

Not all Flash drives hit the theoretical bandwidth of / or even conform to the standards. You might want to check the drive’s manufacturer web page for stats.

It actually had a large enough fat partition to test with windows. Now im fairly sure its the drive and not the os because using crystal disk mark shows 4k random writes of only 198KB/s which about matches up with what im seeing. Its still annoying that it totally locks up the CPU under linux though

Yeah that’s just a piece of crap flash drive.

After testing several different drives under both linux and windows Ive come to the conclusion that flash drives are just really really terrible at random IO – even worse than a mechanical hard drive.
Interestingly, a micro sd card in a USB reader actually fared the best out of all of the drives I tested. its sequential writes were slow at 10MB/s but it was consistent. 4k random writes were around 450KB/s. Weirdly enough the corsair drive I tested actually was by far the worst performer at an amazingly slow 559B/s with 4k random writes using fio, and yes thats Bytes, not a typo. Formatting that drive also took a good 10 minutes so I may have a lemon

To be fair, you’re using a flash drive in a way that is not intended.

Also, if you keep it formatted as FAT32 then you will get heavy fragmentation issues which makes I/O even slower.

1 Like