Remote USB HID

Something I've been pondering for a while is how to control something via hardware USB remotely. I've tried virtualhere and some other usbip software (and hardware) solutions- that's close, but it isn't what I want. I would, however, not be opposed to using one of these software solutions to aid in the project.

The goal is to take, say, a game controller, and output the USB control data over a network or through the internet, to maybe an arduino or rubber ducky-like device that mirrors those commands to another machine. In the case of the gamepad, it could control a console which could have its video captured and sent through the same or a different system over the network.

I've found some arduino-related things that come close, but are mostly local USB hid (plug a keyboard into the arduino pins, then output USB data). There are also Ethernet shields.

I'm thinking that if there is a way to mirror serial input data through a network, perhaps I could go USB device > usb to serial > local computer > serial data through network to something (pi or Ethernet shield) that inputs serial to arduino via Rx/Tx pins > sketch outputs USB data to remote device as a HID.

Perhaps that's the long way around. Looking for tips. I'm sure there is a myriad of methods I'm totally unaware of.

I have some usb over cat5 kits. I have no idea whether they use typical networking protocols or what, but maybe a series of vlans and vpns could connect these over great distances? Or even using the aforementioned software to do most of the work, then somehow mirror the virtual USB activity to a physical serial connection?