Background
Intel QAT can be pretty awesome for gzip and zstd compression, but it can be a little tricky to get it enabled on Proxmox. I also don’t really recommend doing this on “production” proxmox boxes unless you really understand the ramifications and potential problems you can create for yourself. You end up using zfs-dkms instead of the proxmox-built version of zfs.
At some point zfs on proxmox may natively be built with QAT accelerator support.
If you aren’t in the know about QAT it is an accelerator that can accelerate data transformation operations; mainly compression and encryption. It shows up like a PCIe device, but is typically built into the CPU (there used to be PCIe add-in cards, but mostly those are slower than what you can do on-cpu today). It is a technology pushing 10+ years old now, was incredibly useful on embedded CPUs like the Cherry-Trail Intel Atom CPUs, but is now on many (but not all) Xeon CPUs.
This has been a long time coming
…and some were excited to talk about QAT+ZFS when we were just talking QAT+nginx (hat tip @Exard3k haha … )
Doing this as a boot-from-zfs is not recommended since QAT likely won’t be detected properly at boot time and then reloading ZFS after boot w/QAT support becomes a problem
This is also good background reading from intel:
https://cdrdv2-public.intel.com/632506/632506-qat-getting-started-guide-v2.0.pdf
Config Notes
Driver for Older/Ancient QAT
There are many versions of QAT hardware. Be aware of this. This one goes all the way back to cherry trail atom and is almost certainly not what you need. I mention this because the other documents I was looking at to figure this out for myself make no mention of this as a possible pitfall at all.
Driver for Modernish QAT
There is firmware, a driver and a userspace library (qatlib) that you will need. Generally the “linux-firmware” package should include it? In my case it was qat_4xx*.bin The repo that contains files normally included in the linux-firmware package come from here in case you need to DIY it. kernel/git/firmware/linux-firmware.git - Repository of firmware blobs for use with the Linux kernel
Ensure the needed firmware is present at /lib/firmware (or where your Linux distribution keeps all its other firmware).
# The kernel module in my case loaded just fine, and I had the firmware
dmesg -i -e | grep qat_4xxx
# ... and the device showed up with these IDs
# lspci -d :4940 -k
# lspci -d :4940 -k
2b:00.0 Co-processor: Intel Corporation Device 4940 (rev 40)
Subsystem: Intel Corporation Device 0000
Kernel driver in use: 4xxx
Kernel modules: qat_4xxx
e0:00.0 Co-processor: Intel Corporation Device 4940 (rev 40)
Subsystem: Intel Corporation Device 0000
Kernel driver in use: 4xxx
Kernel modules: qat_4xxx
e5:00.0 Co-processor: Intel Corporation Device 4940 (rev 40)
Subsystem: Intel Corporation Device 0000
There are sometimes packages in your distro for ```qatlib``; it may just be an apt search away for you. If not, GitHub - intel/qatlib
You should have a ```qat`` service that you must enable and start once you complete this installation.
systemctl status qat
Did you know QAT can do VFIO/SR-IOV as of Sapphire Rapids?
Yup, it’s true. One must explicitly enable IOMMU and SR-IOV in the bios, and configure the kernel boot command line with intel_iommu=on
Then verify:
# For me on Sapphire Rapids, I had these Device IDs. QAT PF id is 0x4940, and the VF id is 0x4941
# lspci -d :4941
# lspci -vn -d :4940|grep -i SR-IOV
# lspci -vn -d :4940|grep -i SR-IOV
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
… but this will have to be left to a future video
Using QAT in user space
This is pretty easy; I’d suggest adding a qat group and
sudo groupadd qat
sudo usermod -a -G qat <YOUR_USER>
Command Notes
lsmod | grep -i zfs
lsmod | grep -i qat
modinfo zfs | grep qat
ls -al /proc/spl/kstat/zfs/
service qat_service status
cat /sys/module/zfs/parameters/zfs_qat_disable
# maybe needed for proxmox
# /etc/modprobe.d/zfs.conf
blacklist spl
blacklist zfs
Further Stuff I read Working On This
Intel Reference PDF
Notes from Phoronix on the November '23 Updates
They like to add support for new algorithms from time to time.