Hello All,
Not posted in a long time. but I cant find answers anywhere else.
I have been following the AI clustering topics about USB4.
I need someone to help get me real latency numbers for Mac Thunderbolt RDMA.
Can someone run ib_read_lat, ib_write_lat and ib_send_lat or some equivalent on a 2 node setup.
I need to know if it is real RDMA or just really good software RDMA. My understanding right now is that there is no way that thunderbolt can bypass the CPU/kernel and go to the GPU vRAM with a zero-copy. There is a limit to how low they can get without adding a RDMA chip to the USB port, which apple might do.
If results are >5 than it is just software RDMA.
If results are lower than they must have RDMA hardware.
If it is <2uS then we are really cooking.
Notes:
I have tested both thunderbolt-ibverbs and OdinLink-Five. and they are good but most of the improvements they get in latency is from enabling busy polling. I am running it with Strix Halo and getting 7.8uS over TCP. I want to know if Apple is just doing the same trick.
sudo sysctl -w net.core.busy_poll=100
sudo sysctl -w net.core.busy_read=100
I look forward to getting real numbers for latency.