Paradoxically, the closest solution I found so far is offloading via NIC SR-IOV. If you plug the VFs to VMs you can have them talking to each other (and other stuff connected to associated PF port) without involving dom0. This has the drawback of using PCIe bandwidth for inter-VM connections, unless the VF driver can handle it better. Did not benchmark it yet, as I’m having problems getting it set up with Xen and HVM OPNsense.
Regarding direct connection, I found one presentation that suggested it’s possible and implemented in Xen, but that’s it so far. I think it was something from these guys (definitely associated with Xilinx), but no concrete solutions yet. dom0less seems to be the keyword.