QEMU + KVM Switch: USB Re-enumeration Issues

I’m using a KVM switch to be able to use the same monitor, keyboard, and mouse for both the host and the VM. The switch handles both HDMI and USB, the HDMI works fine, but the USB pass-through to the VM breaks when switching between host and VM due to the devices getting re-enumerated every time you switch.

Unfortunately it seems I can only specify a bus and device number to assign to the VM and not select specifically for a particular device. Is there a good workaround? It should be noted I’ve been using virt-manager and would like to continue doing so because I actually have things working through there otherwise and do not want to start from scratch using the bash script approach. I tried playing around with udev rules, but wasn’t able to find a solution there. If I could either have a particular device be statically assigned a particular device number, or have the device number to be passed through updated dynamically without the need to shutdown the VM that would be ideal.

I could maybe just have a USB hub between the KVM switch and the guest USB port that will just hold a particular device number on its own, but it would be nice to find a solution with my current configuration.