OK, this post is to document my findings now that I am operating a Rust server on a OVH bare metal dedicated server in Australia.
It has become apparent that it’s commonly thought that OVH’s DDoS protections is filtering Steam based games causing them to not appear in the server browser, or server monitoring websites/services. The primary “proof” people continue to use is that hosting anywhere else doesn’t have this issue, however as everyone should know, correlation does not imply causation.
Unfortunately understanding the issue will require some prior knowledge first, so here we go:
Server discovery
When a dedicated server is started, it registers itself with one of the many steam master servers, publishing it’s details. These master servers are spread out around the globe to not only distribute load, but also for fault tolerance.
When a game or service want’s to get the details of all servers they connect to one of the master servers and ask for information, they can ask either for just a list of valid servers, or detailed information about a specific server.
The steam master servers do not retain all the details of each server, when someone wants these extra details the steam server connects directly to the game server using the steam server query protocol (note I am discussing the source protocol here, not goldsource) where it can request further information, such as players online, title, description, time of day (if applicable), etc.
The server query protocol
Using strace it’s possible to see exactly what is going on under the hood of the dedicated server. You can obviously use tcpdump also to see this.
<... recvfrom resumed>"\377\377\377\377TSource Engine Query\0", 1400, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, [16]) = 25
sendto(44, "\377\377\377\377A\234\317\265\367", 9, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, 16) = 9
recvfrom(44, "\377\377\377\377TSource Engine Query\0\234\317\265\367", 1400, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, [16]) = 29
sendto(44, "\377\377\377\377I\21LRRP Main Au [90% Vanilla|No Pay to Win|QOL Mods Only]\0Procedural Map\0rust\0Rust\0\0\0\1,\0dl\0\0012332\0\261\207i\t\\\24\345KN@\1mp300,cp1,ptrak,qp0,v2332,monthly,h14763fa0,stok,born1649388743,gmrust,oxide,modded\0J\332\3\0\0\0\0\0", 203, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, 16) = 203
<... recvfrom resumed>"\377\377\377\377U\234\317\265\367", 1400, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, [16]) = 9
sendto(44, "\377\377\377\377D\1\0hedwig\0\0\0\0\0.\2708G", 22, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, 16) = 22
recvfrom(44, "\377\377\377\377V\234\317\265\367", 1400, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, [16]) = 9
sendto(44, "\377\377\377\377E!\0build\00070050\0description_00\0AU | Vanilla | 100% Upkeep\\n\\nWelcome to LRRP AU Main. High performance servers running on real hard\0description_01\0ware (not VMs).\\n\\n| Features |\\n- Vanilla like server, no major game changing mods.\\n- Remove tool \0description_02\0(/remove) for everyone.\\n- Furnace splitter for everyone.\\n- No pay-to-win or vip perks, level playi\0description_03\0ng ground for all.\\n\\n| Wipe Cycle |\\n- Map: Monthly, 1st of each month @ 2pm AEST\\n- BPs: Quarterly\0description_04\0, 1st Janurary, 1st April, 1st July, 1st Sept @ 2pm AEST\\n\\nReport cheaters using F7 in-game.\\nFor a\0description_05\0ll else including feedback and support, join our discord!\\n\\nWeb | lrrponline.com\\nDiscord | lrrponl\0description_06\0ine.com/discord\\n\\nServer Sponsored by HostFission\\nhttps://hostfission.com\\nEnterprise Server Manag\0description_07\0ement\0description_08\0\0description_09\0\0description_10\0\0description_11\0\0description_12\0\0description_13\0\0description_14\0\0description_15\0\0ent_cnt\000133365\0fps\000192\0fps_avg\000190.77\0gc_cl\000412\0gc_mb\0003970\0gmn\0rust\0gmt\0Survival\0hash\00014763fa0\0headerimage\0https://lrrponline.com/images/rust_main_au.png\0io\0pl1,sl8,tm15,tr60,cs17\0logoimage\0\0pve\0False\0uptime\000291560\0url\0https://lrrponline.com\0world.seed\00027022022\0world.size\0004500\0", 1258, 0, {sa_family=AF_INET, sin_port=htons(35944), sin_addr=inet_addr("192.99.44.66")}, 16) = 1258
If you are unfamiliar with this output let me explain. Each line represents a single syscall into the kernel, this is when the application makes an API call to perform some system function. This is happening inside the application and is not observable by the outside world.
recvfrom is the API call to ask and wait for UDP network data
sendto is the API call to respond or send UDP network data
The sequence of events here are:
- The master server sends the application on my server an
A2S_INFOrequest, but this request is missing the “challenge” token. - My server responds with an
S2C_CHALLENGEmessage, containing the current challenge token. - The master server again sends the application on my server an
A2S_INFOrequest, but this time it has the challenge token obtained from the last message. - My server responds with the
A2S_INFOresponse - The master server now asks for the list of players with the
A2S_PLAYERrequest, using the same challenge token - My server responds with the
A2S_PLAYERresponse. - The master server now asks for the server rules with the
A2S_RULESrequest, using the same challenge token. - My server responds with the
A2S_RULESresponse.
This completes this session. As you can see there was a total of 4 requests from the steam master server. These four requests are repeated every single time someone asks for the server details, be it steam themselves or a third party with access to this system such as BattleMetrics or ServerRanks or some other third party ranking/metrics website.
Rate Limiting
The game server (in this instance Rust) can have it’s own flood protection built in as these ports/queries could otherwise be abused to DoS a server by flooding it with requests. In the case of Rust, after exactly 30 requests across a period of 5 seconds it stops responding to further requests even if those requests originate from the loopback address (127.0.0.1). Until the flood of requests stop for ~10 seconds, this filtering will persist indefinitely.
Normally this wouldn’t be an issue as there are lots of steam master servers and it’s somewhat random which server is querying your server.
You can verify this yourself by using the following script:
https://github.com/GameServerManagers/LinuxGSM/blob/master/lgsm/functions/query_gsquery.py
And running the following command repeatedly:
python query_gsquery.py -a 127.0.0.1 -p 27015 -e protocol-valve
If you repeat this fast enough, after roughly 30 requests (for rust) the server stops responding to futher requests. Using strace you can see that the application is still getting the requests, but stops responding to them.
Enter OVH (and posslibly other larger providers)
All data centres do what they can to save network bandwidth and reduce latency, for example Microsoft provide the WSUS server which you can host on your network to keep a local copy of all updates for a large corporate network. Often they also have mirrors of popular Linux distributions which not only save them costly bandwidth, but also provide a better user experience for their customers when performing updates or new installs.
The same is true for Steam master servers, OVH have them in house, for Australia at-least they seem to have only one, at the IP 192.99.44.66.
Now for those that have been following along by now might be able to guess what the issue with OVH game hosting actually is now you’re armed with all this information.
That’s right, ALL steam master server requests for servers housed on OVH’s network are coming from the same IP which is triggering the flood protection built into the game itself.
The Solution
Ideally the game developer needs to adjust their flood protection to whitelist these servers, or give the server owner the abillity to adjust these limits, however until that is done the only other solution is something external, and here it is!
Introducing:
This application makes use of NFQUEUE to intercept the steam query requests and serve them directly from cached values it retrieved from the server prior. It’s essentially a caching transparent reverse proxy for the “source protocol”. See the README.md for usage documentation and more details on how it works.
Using this our server now has the fastest response time of any server in Australia holding the top 1-5 position of the Rust server browser, on a OVH dedicated server.
Here we are holding #1 with the lowest ping
Note this is NOT an OVH GAME server, this is a standard OVH dedicated server.
Caveats
OVH do rate limiting, they have DDoS protections and the likes and it could cause issues even still as your player base gets larger and traffic increases. Time will tell though if it really is an issue now that the core problem has been identified and fixed.