Hi Looking Glass Team,
First off, I want to extend a massive congratulations on the B7 release. The new D12 capture backend is nothing short of transformative. The engineering effort to create a D3D12 resource backed by IVSHMEM, enabling a direct GPU hardware copy, is incredible. It has fundamentally shifted the performance landscape for this project, and the results speak for themselves.
After digging into how this new optimization works, my understanding of the new high-performance data pipeline is this:
Guest VRAM — (Guest GPU HW Copy) —> System RAM (IVSHMEM) — (Host GPU DMA via DMA-BUF) —> Host VRAM
This effectively achieves “CPU zero-copy” on both ends, which is a monumental achievement and explains the huge performance gains. The primary bottleneck has now logically shifted from CPU overhead to the PCIe bus and system memory controller bandwidth.
This got me thinking about the theoretical “holy grail” of performance for this kind of passthrough: a true Peer-to-Peer (P2P) DMA transfer that bypasses system RAM entirely. The ideal pipeline would look like this:
Guest VRAM — (Host GPU P2P DMA) —> Host VRAM
This would not only further reduce latency by eliminating the RAM “middleman” but also halve the PCIe bandwidth required per frame and completely free up system memory bandwidth for the host.
I’m fully aware that this is likely a monumental task, if not impossible with the current consumer hardware and hypervisor landscape. My assumption is that the primary roadblocks would be:
- Hypervisor Support: QEMU/KVM would need a way to securely re-map a passed-through device’s PCI BAR (especially a ReBAR-enabled one) back into the host’s physical address space for the host GPU driver to see.
- Driver Support: Both guest and host GPU drivers would need to support this kind of cross-virtualization-boundary P2P transaction, which is likely well outside their design scope.
- Security: This seems to fundamentally break the isolation model that virtualization relies on.
My questions for the development team are:
- Is this P2P DMA concept something you have explored or considered for the long-term future of Looking Glass?
- From your expert perspective, what are the biggest technical (or even political, with driver vendors) blockers to achieving something like this?
- Do you believe this is even theoretically possible on current consumer platforms, or is it firmly in the realm of specialized enterprise hardware (like with NVIDIA’s GPUDirect)?
- Are there other, perhaps more feasible, avenues for further optimization being considered, such as using the GPU for on-the-fly texture compression (i.e., BC7) before writing to IVSHMEM to reduce bandwidth?
Again, amazing work on B7. This is purely a “blue-sky” discussion out of technical curiosity. Thanks for creating such a fantastic project!
Cheers