Aquila: A new Google paper on strange datacenter networking

Disclaimer: I didn’t / don’t work on this - opinions my own - I’m just a networking enthusiast and I like that not everyone is doing the same thing in networking all the time. Cookies are tasty, but it’s good to change the cookie cutter once in a while.

In other words, here’s something totally useless to most people doing anything networking related, because it’s a) probably impractically unobtainable, …and b) probably useful only to people who built it - strange aliens from the future trying to understand cat videos with persistence of the Borg trying to understand the Omega molecule. c) academically interesting.

Aquila: A unified, low-latency fabric for datacenter networks.

TL;DR ;

Conceptually, you take 48 machines, put them in a rack. You hook them up to a switch using PCIe. To each machine the switch pretends it’s a fancy network card. You take 50 of these racks, and all 50 racks/1000 machines look like they’re basically hooked up to one big switch. Aquila is that switch/network cards/topology/interconnect thing.

It’s latency optimized - Ethernet is too slow, when machines send IP packets to each other, each packet is split into cells and multiple cells are sent in parallel using multiple paths, and then reassembled. Overall tail latency for IP <40μs RTT, for remote memory (a-la RDMA) <4μs RTT of 4KiB reads/writes. Majority of that 2.5 - 3 are just because PCIe is slow - (my guess back when this was built this would have been on old Skylake 8280s, I don’t know if Milan/Genoa or Sapphire Rapids are quicker because new PCIe). That remaining 1μs that the switch takes, is also how long light takes to travel around 300m in a vacuum, and there’s about 50 racks in this Clique which take up some physical space ± cable distances. Do your own math how many nano seconds of budget you have left for error correction and routing logic :slight_smile:


Other tech in this space?
Intel OPA (another intentional Sci-Fi reference?.. Omni Path Architecture), and Liqid Composable Infra come to mind.

Hopefully, the pdf is a fun read.

7 Likes

Nice reference, I just saw that episode of Voyager the other day. I actually remembered nothing about that little piece of ST lore.