tl;dr
What are the practical drawbacks to accessing your primary machine exclusively through VDI that may not be immediately obvious? Does anybody have a setup like this now that they could comment on?
Background
I started spending more and more time away from home lately, which lead me to looking at different ways to keep up with my scheduled monday night gaming sessions while Ooo. My initial test of streaming my desktop to my smartphone with a USB monitor and wireless peripherals was a bit of a flop (poor s9+ couldnāt keep up). Later tests with an x86 laptop were successful enough that I started thinking through what it would look like to move my primary PC into my network closet, set up something like ESXi or Proxmox, and access it through VDI full time.
75% of my weekly computer use could easily be handled by a more energy efficient machine than Iām using now (9980xe, 3080ti). Weekly gaming and daily 3d modeling seems like it could be handled through a remote session easily enough.
Perceived Benefits
It would be cool.
With virtualization, backups would be simple to manage and might actually happen.
There is potential energy savings to having the virtualized PC shut down during lighter tasks (i.e. 75% of the time). EDIT 10/25 - Actual power savings are at best negligible and more likely a net increase in power use.
It would be a great learning experience getting it all set up and working out any bugs.
Get this large and (relatively) loud system out of my office.
Potential Problems
USB Devices - No physical access to the machine means that things like USB to serial programmers would be more cumbersome if developed on the virtualized PC. USB over ethernet is a thing and there are two separate cables going to my office for things like my spacemouse that I use for CAD work.
Single āVirtualā Monitor - Currently I use two (2) 2560x1600 monitors. This could be replaced by one ultrawide, or just use one screen for a virtualized PC and one for the local machine. This might take some getting used to but seems manageable.
More Complexity ā PITA when it breaks - This is adding a lot of complexity to using a computer. I can probably minimize the PITA factor by just doing as much as I can get away with on the local machine.
Just based on the above, it seems viable, and maybe even practical? What am I missing?