Fedora 33 slow I/O/kernel comparing to others

Hey there. I’ve been using Fedora 33 for quite some time right now and I since I often switch distributions, I’ve noticed that Fedora’s performance isn’t top notch, heck, it isn’t even close to the “competitors”. Here’s what I am talking about.

I do my daily web development using Docker. Before Fedora 33 got official repository for Docker CE, I’ve used moby-engine, but performance is more or less the same.

MySQL (native) and Docker are having really bad I/O performance. I don’t want this thread to be too long, but here’s what’s the problem:

I am importing this MySQL dump (https://raw.githubusercontent.com/invoiceninja/invoiceninja/v5-develop/database/schema/db-ninja-01-schema.dump).

All these numbers are tested on the same hardware (not virtualisation). Here’s what I was able to get:

Fedora 33 (5,10, ext4, native MariaDB (not Docker)) ~ 2500ms
Fedora 33 (5.10, ext4, Docker MariaDB) ~ 3200+ ms
Arch Linux (5.10, ext4, Docker) ~ 420ms
WSL2 Ubuntu 20.04 (4.2, Docker) ~ 560ms.
Ubuntu 18.04 ~1000ms.

Can someone guide me, what’s the deal here?

Here’s what I’ve tried:

What could be the deal here? I really love Fedora & don’t want to give it up, so… I’d love to find out what’s wrong here.

In case you need to see dockerfiles - invoiceninja5-dockerfiles/docker-compose.yml at master · beganovich/invoiceninja5-dockerfiles · GitHub

For the sake of testing I still have Fedora 33 (btrfs) and WSL2 Ubuntu 20.04 installed, so I can debug stuff.

Thanks in advance!

1 Like

Solved. Install Fedora (Workstation/Server) using XFS.

Ended up with 250ms using Docker :tada:

1 Like

Interesting, do you know if it was an optimization in the Workstation version of the distribution or just the new filesystem?

Not sure to be honest. Maybe it has something to do with the fact RedHat is heavily interested in XFS itself. Who knows.

Could be different tuning decisions with ext4. Could also be different tuning decisions made with regards to disk caching out of the box. It’s been a while since I played with hdparm (I’m sure there’s probably more modern tools) but the difference you can make playing with the switches on it is huge.

I did a quick google and also there’s been recent kernel work on fast commits on ext4 journaling in kernel 5.10.

I very much doubt it is software bloat, more tuning decisions (for speed vs. resiliency perhaps, and also kernel version related).

I think Fedora (from memory) also goes through lvm by default, not sure if there’s any impact there; pretty sure ubuntu for example doesn’t use lvm unless you manually select to use it.

Also - were all of these installed to the same physical location/partition on the physical disk? If it’s rust, different partitions may run at different speeds based on where they are on the physical disk :slight_smile:

hdparm:

hdparam looks interesting, however, I haven’t played with it, since ext4 on Arch was working just fine, I didn’t doubt the file system at all until I have tried xfs.

When it comes to LVM, all tests (including the xfs) are performed without lvm.

Also - were all of these installed to the same physical location/partition on the physical disk?

Yes, same disk. Heck, even same partition. :thinking:

XFS is working super nicely so far, I wonder if there’s any downside using it on desktop, haven’t found any major problems by Googling.

to be clear, hdparm isn’t filesystem tuning, it is device tuning.

Its possible different distros use different defaults in a safety vs. integrity/compatibility trade-off.

one disadvantage to XFS i found with Centos is that i couldn’t resize it (shrink, IIRC) under LVM (only discovered the other day in a play/test VM). YMMV, but something to look into now before you get too attached to the machine, if you’ll ever need to say, shrink one XFS filesystem to make space for another under LVM.

the classic scenario for me was that the entire disk was in a lvm group, i wanted to shrink one XFS filesystem to give space to another. no dice.

1 Like

Could be, unfortunately, my knowledge about Linux stops there to dig deeper. I’ve learnt a lot over these past weeks just searching for the issue.

If that’s the biggest con (comparing to ext4), it’s perfect. I usually do OS per disk. Thanks for the help!

Be aware that can still bite you depending what you’re doing.

This was inside a VM, centos by default used LVM with a layout that didn’t really make sense (IIRC way too small /var). I figured (during the install) “ah its fine, it uses LVM, i’ll just resize that partition later - it will give me an opportunity to mess with LVM as i haven’t really done that” (it was a test vm).

but no…

ZFS and enterprise arrays have ruined me for linux…

1 Like