Troubleshooting Steam Cache Speed When Downloading non cached Games

Hey Guys. First time posting here, but this seams like a welcoming community. Anyways on to my question. Over the last couple of days I built my self a steam cache server. It’s working, and I am getting great performance when installing games that are cached. The slight problem I am trying to improve on is the downloads form the internet. When downloading non cached games form steam I am seeing fluctuating speeds between 3-9 Megabytes Per Second. If I switch my desktop back to my non steamcache DNS ,and talk directly to steam, I can download the same game at 15-20 Megabytes per second. I imagine this is a limitation of some part of my hardware or software setup. Here are the specs of the machine I am Using,

  • Core i5 3570k
  • 12GB DDR3 memory
  • Running Ubuntu 16.04 off a 2tb Hitachi 7200 rpm sata 2 disk
  • The actual steam files are stored on 2 500gb 7200rpm sata 2 disks in a raid zero array created with mdadm

This machine Is running both the SteamCache Docker Container and the SteamCahce-dns docker container as provided by SteamCache.net . While downloading form steam, I have looked at disk IO, memory usage and network traffic, with nmon, and nothing jumped out at me as a bottle neck. Any recommendations on what might be my bottle neck, what to try, or how best to trouble shoot this further? Also if anyone wants any mroe info that I haven’t provided just let me know. Thanks in Advance for any help :slight_smile:

Update: Decided to update this post with all the info I now have, for anyone currently following this thread and/or anyone who is dealing with the same things who may see this post. I have done a bunch more testing form when I created this post originally. I will now go through every different configuration I have tested to try and get this working. Originally I had this running as listed above. I continued to test on that system for most of process, changing one thing or another. I tried changing the cache location/storage media multiple times, including a single of the two 500gb HDDs I had in raid, cashing directly to the Hitachi 2tb boot disk, cashing to a 120gb Kingston ssd. None of this made a difference. I later tried replacing the boot disk with a faster sata 3 3tb WD HDD, and tested caching directly to it, with no gains to be had. I also have tired several different ram configuration for the host. I tried 8 GB, 12gb and 16gb memory sets, none of which seemed to have a effect on performance. I latter seceded to try the “steamcache:generic” container, which seems to have the exact same performance as the normal steamcache container. I started to think maybe it was indeed a dns issue, even though I could find no indication of this with wireshark. To test this theory I added the dns service to my server 2012 box and created all the necessary steam dns entries as provided by the steam cache github. When using the server 2012 dns server I was able to use the steam cache as intended, but there was no, from internet, download speed benefit. Next I thought it might be an issue with the version of Ubuntu I was running so I upgraded the machine form 16.04 to 18.04, with no effect to performance. My long string of dead ends led me to believe maybe it was more to do with the configuration of the nginx server inside the container, so I went wand edited to the config. After trying every tip/fix I could find on google to get the nginx server downloading faster form the web, I found no configuration change that gave any speed improvements. Next I decided to do my best to make the config of the nginx server inside the container to match the config provided in this arstechnica steam cache guide. This also had no effect on speed. All else failing I tried completely starting from scratch, building a 6 core 4gb memory system on my xenserver box. I after installing Ubuntu 18.04 on the VM I followed the LTT Steam Cache Guide, with the result being a working steamcache that still can only download at form the internet at 5-6 Megabytes per second, the same as all my previous test. My final attempt, as of writing this update, was to build the nginx cache server, as described on arstechnica, atop the Ubuntu VM. Once completed I was able to sue the steamcache-dns container, on the ubuntu VM, to point my steam at the nginx server running on said VM. Once again I had a working cache, with the only issue being the same internet download speed limitation.

1 Like

What does iostat -x report when you’re downloading, how many requests per second are being processed? What’s the average wait time?

It sounds like something is misconfigured, e.g. your partitions aren’t aligned, or as if you haven’t built mdadm with the right flags and you’re getting some write amplification.

There’s another person using a $50 -ish 32G optane drive as an os drive, it might be interesting to couple that drive with your raid0 using bcache in writeback mode.

Edit: what happens when you use one drive, not in raid0

Here is what IOstat looks like while downloading at 8.2Megabytes Per Second. Was about to go to bed when I saw the email for your reply, so I will try moving it over to a single disk tomorrow some time and get back to you on that.

1 Like

Yours is a project I have wanted to do for a handful of years but have never really taking the time and effort to attempt

If a system was always online and you had a shitload of ram could you make a ramdisk and use that to buffer and cache the storage drives?

Most of the data seems to be coming from ram, or from another disk in this case, if you look at iostat there’s very little being read from md0 (sdb+sdc), it just doesn’t add up to 8MB/s.

If the data is coming from the host, then it’s definitely coming from ram and 8MB/s indicates an issue with the client or server configuration. You could try downloading files off of nginx yourself using curl/whether/… maybe that goes faster.

I’ve never looked at nginx or steam client with pprof, or looked at nginx or steam client sources - from what I hear about nginx I don’t think it’s code is stupid to the point of making things very slow.

Sure… mount -t /tmpfs none /mnt/my-big-fat-cache-dir … Point docker container at that, go wild. … It might actually be an interesting test for @TikTok given how easy it is to try.

I’m a bit confused here. Keeping in mind that the game I am downloading is not in the cache, so the docker container has to download it form steam and then forward it on to me, so we expect it to be writing it to disk and then reading it from disk again to send it off to my client? To me it kinda would make sense that in this case most data is coming form ram, as I assume the cache server is just storing the data in ram before both sending it off to my steam client and also writing it to the disk. In any case I did as you recommended earlier and moved the cache over to single disk. I added a Kingston 120gb SSDnow to the machine and am caching to that. still my best download speed on a non cached game was about 7.1 megabytes per second, under half what I would normally get from steam.

You could maybe consider using Wireshark to see how the client achieves these faster connections when downloading directly.

I’m thinking, maybe it’s using more parallelism perhaps if DNS is returning more hosts… but that’s just a guess.

After looking in wireshark, and running some other dns tests it doesn’t seam dns related to me. Everything I have tested, including the dns stuff, is in the update I just added, encase you are curious.