Libvirt/USB Passthrough Issue

I got an interesting problem with Libvirt/QEMU/KVM here. My PC has 2 mice connected, one of which gets detached from host and passed through to the Windows VM when that gets turned on.

Occasionally, when the VM starts and Windows reaches login screen, the VM abandons access to the mouse and returns it to the host. I am using direct USB passthrough for this, not passing through an entire USB controller.

Worth noting that this happens with either one of my mice (different mice too), but does not happen with any other USB device.

Any ideas?