Here is a late update on this issue. As mentioned above, I’ve kind of pinpointed the problem to be related to Linux native AIO, or in QEMU/libvirt terms, io=native.
Here are some recent representative data (zvol left, qcow2 right, expand to view):
As seen above, performance tanks for qcow2 when io=native (but not when io=threads). Note that the test settings differ slightly from those in my first post, but the pattern is clear either way.
Detailed storage options (hidden for readability)
NTFS allocation unit size: 4k
qcow2 flags: cluster_size=64k,preallocation=metadata,lazy_refcounts=on
virtual disk options (apart from io): device model=virtio-scsi,cache=none, discard=unmap
iothreads: 1 per virtual scsi controller (effectively 1 per virtual disk)
zfs/zvol options: recordsize/volblocksize=64k,compression=on,primarycache=metadata,(xattr=sa,atime=off when applicable)
file/volume size 10G
I tried to match the parameters of the zvol+ntfs and the dataset+qcow2+ntfs complexes as much as possible. Again, raw files on dataset gives similar (bad) performance as qcow2 with io=native, compared to io=threads.
(I’m aware that the 4k/64k mismatch present on both storage models gives rise to some write amplification - both in theory and as verified by zpool iostat - however this does not explain any variance for the issue discussed here)
Explanation?
I’ve seen from various sources that native IO are reported to have problems with sparsely provisioned storage. E.g. in this link:
When space efficient image files are used (QCOW2 without pre-allocation, or sparse raw images) the default of io=‘threads’ may be better suited. This is because writing to not yet allocated sectors may temporarily block the virtual CPU and thus decrease I/O performance.
and in this link (thanks to @Dratatoo and @vitoyxyz in another thread):
Sources of blocking in io_submit(2) depend on the file system and block devices being used. There are many different cases but in general they occur because file I/O code paths contain synchronous I/O (for metadata I/O or page cache write-out) as well as locks/waiting (for serializing operations). This is why the io_submit(2) system call can be held up while submitting a request.
This means io_submit(2) works best on fully-allocated files, volumes, or block devices. Anything else is likely to result in blocking behavior and cause poor performance.
Blocking is a reasonable characterisation of the behaviours I see. When monitoring with zpool iostat, I do see pauses with no disk activity together with the observed performance dips for native io. However, something is still off with this explanation - also my zvols are sparse, they are allocated with the -s -V flags to zfs create. And for them I don’t see any problems.
Moreover, I mentioned above that I did see the same issues with raw files on top of datasets. Though they were created with qemu-img create -f raw ..., which creates sparse raw files by default.
So I tested creating a preallocated raw file, using dd if=/dev/zero of=file.img .... In addition, I put it on a dataset with compression=off, as I know that zfs would otherwise compress the 0s to almost nothing. I wasn’t sure what this compression would imply in terms of sparseness. The result is a file as un-sparse as one could possibly get it on a zfs dataset. And still, the results look like those for qcow2 files, i.e. bad:
Preallocated raw image on top of zfs dataset, io=native
Slightly better than qcow2 file under the same conditions, but nowhere near what we see for threaded IO (regardless of storage model).
One final point of observation, is that when specifying the virtual disks with virt-manager, then io=native is chosen as default for zvols, whereas io=threads is chosen as default for files on datasets. So the issue observed here seems to be known by the libvirt/virt-manager devs.
Consequences?
As I wrote in a previous post, i saw minor to medium increases in CPU usage for io=threads compared to io=native (in situations where performance was comparable). However, I’ve also came across combinations of options where the CPU load difference seemed minor, but I haven’t investigated it systematically. There are a lot of optimization that I haven’t tried, that could mitigate the drawbacks.
Conclusion: threaded IO for files, native IO for zvols
Only io=threads works reasonably well for files on datasets, regardless of whether the files are sparse or not. io=native works slightly better for zvols, sparse or not. Why this is the case is not entirely clear to me.