Sound Issues in Linux Guests with passthrough'd USB bluetooth device

Operating Systems & Open Source Virtualization
1

Hello,

I am running a kvm/qemu setup via libvirt where I am experiencing issues with bluetooth sound.

Problem Summary
When sound output from the guest is done via bluetooth, there is very noticable latency (up to a second I would guess), audible distortion, as well as short system freezes every now and then.

My Setup

  • Ryzen 5900x on Gigabyte Aorus Master X570
  • ASMedia ASM2142 PCIe USB 3.1 Controller
  • TPLink UB 400 BT dongle
  • Host: Arch Linux, 5.14.5
  • Guest: Various Linux with same issues

libvirt XML

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>archlinux</name>
  <uuid>bf75fc4c-7888-4d1e-b02e-ead9bd6b3d5b</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://archlinux.org/archlinux/rolling"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">29360128</memory>
  <currentMemory unit="KiB">29360128</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static">12</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="6"/>
    <vcpupin vcpu="1" cpuset="18"/>
    <vcpupin vcpu="2" cpuset="7"/>
    <vcpupin vcpu="3" cpuset="19"/>
    <vcpupin vcpu="4" cpuset="8"/>
    <vcpupin vcpu="5" cpuset="20"/>
    <vcpupin vcpu="6" cpuset="9"/>
    <vcpupin vcpu="7" cpuset="21"/>
    <vcpupin vcpu="8" cpuset="10"/>
    <vcpupin vcpu="9" cpuset="22"/>
    <vcpupin vcpu="10" cpuset="11"/>
    <vcpupin vcpu="11" cpuset="23"/>
    <emulatorpin cpuset="0,12"/>
    <iothreadpin iothread="1" cpuset="1,13"/>
    <iothreadpin iothread="2" cpuset="2,14"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/archlinux_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <reset state="on"/>
      <vendor_id state="on" value="1337"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="6" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="topoext"/>
    <feature policy="disable" name="svm"/>
  </cpu>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="8" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <input type="mouse" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0e" function="0x0"/>
    </input>
    <input type="keyboard" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0f" function="0x0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
      </source>
      <boot order="10"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0b" slot="0x04" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x01" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x10" slot="0x00" function="0x0"/>
      </source>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=mouse1,evdev=/dev/input/by-id/usb-Logitech_MX518_Gaming_Mouse_0379396D3438-event-mouse"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=kbd1,evdev=/dev/input/by-id/usb-Kingston_HyperX_Alloy_FPS_Pro_Mechanical_Gaming_Keyboard-if02-event-kbd,grab_all=on,repeat=on"/>
  </qemu:commandline>
</domain>

In search for a fix for my issue, I have experimented quite a lot with the CPU settings, so the above only reflects my most recent attempt, where I have also isolated the cores pinned to the VM.

In-Depth Problem Description

Sound over bluetooth with pulseaudio has high latency and slows down the system. It will be fine for some seconds, then I get a half-second long freeze, after which sound quality is abysmal with high latency. Running pulseaudio in superverbose mode, I get corresponding messages:

I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 15360 to 14336
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 24
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 25
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 26
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 14336 to 13312
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 27
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 28
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 13312 to 12288
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 29
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 30
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 31
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 12288 to 11264
D: [bluetooth] module-bluez5-device.c: Skipping 521 us (= 92 bytes) in audio stream
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 26
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 11264 to 13312
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 27
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 28
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 13312 to 12288
D: [bluetooth] module-bluez5-device.c: Skipping 1276780 us (= 225224 bytes) in audio stream
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 23
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 12288 to 14336
D: [bluetooth] module-bluez5-device.c: Skipping 1775736 us (= 313240 bytes) in audio stream
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 18
I: [bluetooth] module-bluez5-device.c: Changing bluetooth buffer size: Changed from 14336 to 15360
D: [bluetooth] module-bluez5-device.c: Skipping 96712 us (= 17060 bytes) in audio stream
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 13
D: [bluetooth] module-bluez5-device.c: Skipping 37392 us (= 6596 bytes) in audio stream
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 8
D: [bluetooth] module-bluez5-device.c: Skipping 71451 us (= 12604 bytes) in audio stream
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 3
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 4
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 5
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 6
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 7
D: [bluetooth] a2dp-codec-sbc.c: Bitpool has changed to 8

The skipping messages line up with noticable stutters in audio.

When I skip pulseaudio entirely by playing a .wav directly through ALSA’s aplay (with help of https://github.com/Arkq/bluez-alsa), there seems to be no issue. It’s hard to gauge latency with an audio file alone, but I get no stutters with pure ALSA, while pulseaudio exhibits all those issues on the same audio file.

I have pretty much tried all the fixes w.r.t. pulseaudio found throughout the web:

  • load-module module-udev-detect tsched=0
  • realtime-scheduling = no
  • default-sample-rate = 48000

Because my Arch host has no such problems, this must be tied to the fact of being in a VM, but it is also extremely isolated to the specific issue of pulseaudio + bluetooth. Since sound output through the PCIe sound card works fine, I suspected the CPU configuration and whatever scheduling issues might come of it, but nothing I changed on that front had any effect.

I would appreciate your help!

2

Since you are on Arch it should be pretty trivial for you to switch to Pipewire as it seems to solve many peoples bluetooth issues (which are mostly due to pulse-audio being a raging dumpsterfire).

I’d def do a bit of reading to see all the issues people were having with pulse + bluetooth that Pipewire has already solved.

I’ve been running Pipewire with VM’s with pass-through with no issues for quite a while.
But YMMV I’m not using any bluetooth devices.

3

I am using Ubuntu and had issues too. I fixed my problems by passing through an audio hardware device. My USB headset works fine with Windows 10 although discord sometimes forgets I have a USB headset and reverts to the monitor speakers which is annoying. I gave up on using the bluetooth headset, never was reliable. It seems bluetooth has a problem with a mic and audio on the same channel.

Note, this winter I plan to rebuild this system, cleaning, refreshing thermo paste, etc. and installing Manjaro.

4

Pipewire is a great suggestion, I didn’t know it’s a drop-in replacement for pulseaudio!

The result is still not what I am hoping for, since audio is still skipping here and there but that has brought me to the realization that the cause likely lies in the bluetooth connection itself. The skipping gets worse the further away I get from the dongle (like 2m of unobstructed distance) at which point the headphones even sometimes disconnect entirely.

Pulseaudio just seems to be particularly bad at dealing with a weak and lossy connection. Freezing my entire system for a second had me believe this must had been a more fundamental issue, like something on task scheduling level, instead of just bluetooth.

Now I just need to find out why BT is so bad; the issue must lie somewhere in the chain of passthrough’d PCIe USB controller → USB Hub → BT dongle…

5

Does your sound work normally through bluetooth not in a VM?

6

I had previously only verified that the dongle worked well enough in the host, but with the PCIe USB card I actually have the same crappy connection in the host. Other usb ports work fine. The host does not experience those full-scale freezes with pulseaudio though. Also tried a different dongle with the same result.

Seems like the USB extension card is to blame then and this is not a VFIO issue at all. I assume the only thing I can do at this point is try a different USB card.

7

Can you just pass in a different host usb controller? Or do you have some weird requirement for having it use a usb card?

I always just pass in my host soundcard directly to the VM, and then Linux just snags it again after the VM shuts down, saves me from faffing around with routing audio through different devices etc.

8

To be honest, I never tried - the dedicated PCIe-card seemed like the cleaner solution and appeared to be the general consensus for painless USB controller passthrough.

But it looks like at least one of the USB controllers in my MB is neatly separated IOMMU-wise. And that one works perfectly with the BT dongle in the VM (at least with an extension cable to get the dongle out from behind the case). So that is an option.

Your continuous input steered me into the correct direction @robbbot, thank you very much.

2 Likes
9

:+1: