Evdev and KVM/Guest Sleep

evdev works great, right up to the moment I switch inputs using my USB/HDMI KVM, or the Windows 10 Guest goes to sleep. Then the guest os is forgotten, and I have to manually restart the vm.