CPU vs caching server throughput, how much oomph do you need?

Hardware Hub Networking
1

I’m building a Steam caching server, and I have a ton of options as far as hardware that I already have. The basic question I have is this: When serving large files to multiple destinations on a local network simultaneously, how much CPU do you really need?

Further details:

I have the fastest internet among all of my friends in my local friend group, so it’s not uncommon for everybody to want to come over and download games, happens pretty much every weekend. At this point I figured I might as well build a steam caching server, both to help avoid constantly hitting my 750 gig download limit, and to (hopefully) make downloading even faster for my friend group while they are here.

Obviously I can’t slap a Pentium 4 in a machine and expect it to feed full gigabit streams to four+ destinations simultaneously… But how much does one actually need? The main reason I’m even asking this question is because I have a ton of spare hardware lying around, and although I could just slap the biggest and best hardware together to do this, I’d rather use some of my less capable hardware and save the good stuff for other builds.

Information on needs and current capabilities:

I have a 300 meg pipe, and a 750 gig data limit. Every time I hit my limit it becomes $10 per 100 gigs after. I usually hit this limit four to five times a month. When my friends are over and downloading simultaneously, It almost never splits evenly. One person always seems to get 120-150 Mbps while everyone else gets like 20-40.

I don’t need to feed a 1 gig stream to everyone, even just averaging 200 each from the caching server would be a massive improvement over the current situation, both in terms of speed and saving my wallet from constantly hitting my data cap. It would be nice to exceed 200 Mbps each, but that is currently my goal.

In ascending order, I have enough hardware to build complete systems with the following:

Core 2 Duo e8400
Core 2 Quad q9550
i3 530
Xeon x5650 (single)
Xeon x5650 (dual)
Xeon e3 1220 (v1, similar to i5 2400)

With all other factors remaining the same, and making the assumption that the CPU is the only bottleneck in the system, what do you think I can get away with? As I stated, I’d rather use some of my less capable hardware for this and save the good stuff for other builds, but if the weaker CPUs simply aren’t up to the task, I’ll begrudgingly used the better stuff.

I guess what I’m saying is I’ve never had to think about the CPU requirement for file serving before, and I’d rather get some community input before I went ahead and put something together only to find out it wasn’t good enough and have to tear it back down again.

Cheers in advance for any help!

PS: just in case anybody brings up the other hardware in the system as potential bottlenecks, this system will have a couple SATA SSDs in raid 0 to help reduce the storage bottleneck, and I have enough hardware to load up even the worst system with at least 16 gigs of RAM if needed. I’ll also be using an Intel 10 gig card as an uplink to my switch, so that won’t be a bottleneck. Finally, any games that will be downloaded, i’ll download myself in advance, to pre-cache the files before my friends arrive

2

I’ve seen this guide used for 400 people LANs before

2 Likes
3

That is actually an awesome resource, for some reason I couldn’t find that when I was googling this question earlier. Honestly, I think that pretty much answers my question, thank you very much for sharing!

1 Like
4

The biggest worry should be disk speed and space
( i only have about 25 games and that takes up 900GB at the moment)

I have a steamcache VM that i gave 2 CPU threads and it averages below 10% cpu all year

max graph

image
image1062×327 35.7 KB

avg graph

image
image1057×332 18.3 KB

And the vm has 4GB of ram and uses about 3.5G of it

1 Like
5

Hi, nginx is fairly efficient, for larger individual files I’d expect it’d be able to saturate a gigabit network on any of the low end boxes you’ve listed, but maybe not if the files are tiny.

Most of it’s CPU would be spent parsing requests, performing TLS crypto and context switching between working on different parts of serving each request.

I’d expect storage / seek times to be your bottleneck.

Out of curiosity, … how do you deal with energy bills for this stuff? (excess solar… or cheap government subsidized electricity?.. or something else).

1 Like
6

That is a good plan. 2-3 SATA SSDs will about saturate a 10 gigabit link fine (1.2GB/s). You may be able to get more throughput from NVME drives because of their better small file performance, but honestly the difference wont be that much to warrant the cost and most motherboards cant even support more than 1-2 NVMEs unless they are the newest around or very expensive territory.
You may want to use caching software to make use of as much RAM as you can to the drive array. Try caching the newest files to memory and play around with the “least recently used” and “least frequently used” algorithms to find which ones keep the most files you want in the cache.

1 Like
7

Energy is really cheap where I live. Running one of the Core 2 Duo machines, for example, is like $8-15 a month, at worst. Also, the caching server itself would only be powered on when we needed it. If my group of friends is not over at my place downloading steam games, then I don’t need it to be powered on.

I’ve got a little baby server rack that is full of random hardware from like 2006-2014. My PFSense box, for example, uses a Core 2 Duo e4600. I’ve got an APC power bar that measures total energy use as well as usage per-socket, and I do keep a careful eye on power use because sure, energy might be cheap and the odd old energy-inefficient box won’t kill the bill… But 7 or 8 can really add up :stuck_out_tongue:

The most expensive system to run is my Proxmox machine, with dual x5650 CPUs, 96 gigs of RAM, a handful of SSDs, and a whack ton of graphics cards. It is my game server for Ark, 7 Days to Die, and a handful of other things. For a while I was also running a Windows 10 VM for my mother, complete with a GTX 970 passed through to the VM, but she hasn’t used it in a while so I turned it off a couple months ago. I’ve seen this one hit $60-70 a month

For reference, my PFSense box costs somewhere in the ballpark of $5 a month to run

8

I have more ECC DDR3 than I know what to do with, so maybe it’s worth it to toss one of my lower power l5520 CPUs in an old supermicro x8 motherboard, then just LOAD UP on RAM… I know for a fact I could do 48 gigs easy, I might even have enough 16 gig sticks to do 96… That’s honestly more than enough to cache an entire game, depending on the game. Don’t know if it’s worth using up all my good sticks of ram though lol.

Might be worth experimenting with just to see what kind of crazy performance I can get. Then again… If the bottleneck then becomes the network interface itself, the performance might not even be that much better than just a handful of good sata SSDs in raid 0

9

If your 10g NIC has multiple ports on it then you can link aggro them together in a load balancing config. One person won’t get more than 10gb connection speed but if you have more friends over downloading then it can balance the connections over the multiple links so the aggregate throughput would be higher. If all data was in RAM and your CPU wasnt a bottleneck, DDR3 memory should be able to push close to 100gb network throughput after file system overhead. DDR3-1333 has about 10GB/s per stick, so dual channel would be 20GB/s. That would be 160 gigabit, but then minus file system overhead and all that.

1 Like
10

It is indeed a dual port intel 10 gig card. With all things going perfect, the bottleneck would be my switch. It’s two 10 gig SFP+ ports, but 48 1 gig RJ45 ports. At absolute most I would have five people over downloading the same thing, and I’ve seen ports on the switch hit 940 Mbps, so I’d be looking at just shy of 5 Gbps total, assuming everything else was running perfect. Honestly I’d be happy with 200+ Mbps each lol, as that would be a tangible improvement over what I’ve currently got. Also, I don’t know what nics are built in to their motherboards, but it’s possible not all of them can hit 940 Mbps, I’ve seen some Broadcom nics struggle to surpass 800.

That would be kind of a cool experiment though… Grabbing a couple of 10 gig cards off eBay to give to my friends and maybe a cheap 10 gig switch, then just seeing exactly how much data I can push… I already have spare 5m DAC cables… No, no, that’s an experiment for another time, right now. I’ll just get it working before I start thinking about doing anything crazy lol

Fun fact, I originally had a “Brocade” 10 gig card, but it couldn’t surpass 7.3 Gbps, so I got rid of it in favor of the Intel card, which seems to be able to hit a rock solid 9.3-9.5 Gbps

1 Like