Looks like Sabrent implemented exactly this strategy. Apparently they recently came out with a new TB4 KVM, which is more of a physical switch.
I found two critical reviews. One that described the switching mechanism and the lack of EDID, keyboard, or mouse emulation. The other was an unfortunate review that described my fears with the PD feature. Apparently he ended up frying two motherboards when switching between a laptop and a desktop while using PD.
There’s 25 reviews so far, and it goes for about $350, which is about the base I was expecting for something like this lol