[Looking Glass] Mouse stopped working

I have this weird issue with Looking-Glass.
I made sure that everything works before setting Looking-Glass up.
Installed the [spice-guest-tools] on Guest and followed the Installation instructions from the Wiki.

It could move the mouse and use clipboard sharing for a few test rounds after VM setup, but it suddenly stopped working, unless I press the Scroll Key/active capture mode.

How do I go in troubleshooting this?