Call me a sceptic, but what is it you’re developing exactly?
The 5ms latency sounds like you’re doing some form of screen recording.
Edit: read on, there’s no 5ms of latency, shuffling graphics around this way should get you mere microseconds of latency at less than or around 1% of CPU core at 4k 60Hz. There might be some overhead somewhere but theoretically this is all coming together as a very efficient mechanism to send graphics from one GPU to another.