Solution for solo or small dev teams multi player server

Dev Zone Game Development
, ,
1

So I was looking for a good server solution for a multi-player games as a solo dev, and I don’t like windows but figured I should learn hyper-v and found this. I think this sounds like the set up I’m looking for and thought it might help any solo devs or small teams.

7 hypervisors running Windows Hyper-V Server 2019 in a cluster.

All nodes are running Threadrippers with a minimum of 128 of memory per node, probably half the nodes have 256.

The Threadrippers have so many PCI lanes it’s amazing. We have 3 NVMe drives per node, some nodes have a dual port 10 gig NIC and the ones that don’t have dual cards, and a GPU.

These are used for our build and test farm. All of the nodes have a VM with the GPU passed through to the VM so we can run automated tests of our game.

The rest of the capacity is a mix of Windows and Linux VMs to act as build nodes, test servers (running the server, not game), and a couple random other VMs such as our internal game servers (Valheim, Minecraft, Factorio, etc.).

All the VMs we run can be brought up from nothing to full via Ansible, so we don’t really care about having them on a CSV for high availability. We prefer to have them on local NVMe disks for speed. Our build servers also usually have 750GB or 1.2TB VHDs because our builds are fucking stupid in size. Honestly our biggest limitation is disk space locally… so with that in mind we don’t replicate the VMs. This is why we make sure we can bring new build servers up from scratch quickly if needed.

We do have a couple CSVs though. One for things like ISOs, base VHDs that we can copy from, and other random things. The other CSV runs the VMs that don’t have such critical IO needs, such as our game server. We run those off of a TrueNAS server.

Everything I just described is what we call our “Tier 2” cluster. Basically, if it goes down we fix it, but it isn’t an emergency.

We have a Tier 1 cluster as well. That is 3 Dell R6515 nodes with AMD Epyc CPUs, 256 GB of memory, and runs all the VMs where uptime matters. This is things such as our TeamCity server, Perforce, monitoring infrastructure, Ansible AWX, GitLab, and more. Each node here has 4 SFP+ ports, 2 onboard and 2 via an expansion card.

This cluster is backed by a Dell ME412* and two expansion ME4012*. The head unit is connected to compute nodes via DAC cables directly, no switching layer in between. iSCSI with MPIO access to the LUNs to share the CSVs.

Overall we have been super happy with it. Pushing our build servers onto commodity Threadripper hardware has been great. We get a huge amount more build capacity for the same money, performance has been great for the build machines, especially the newer ones that have PCIe4 NVMe drives.

We could get away with not clustering our Tier 2 nodes. Having the CSV just makes things a smidge easier. We have migrated VMs between nodes and, other than taking a bit of time because 1.2TB aint small yo, never had a problem on the Threadripper nodes.

There is always more then 1 way to weld 2 cats together so if you have other suggestions…

2

After being able to do some tests with a new friend and getting some extra data I needed. I think I have a good hardware setup for my game server that I’m about to order, and figured I should share because I bet everything I order is going on sell after I click order and yall can get a deal.

I am making this to run a mmo style game server that has fps latency, 3000 players per server instance, and creep style AI running server side. I may also make a variant of this with AI GPUs as a local claude bot.

each game server instance will be in an open frame, 3 shelf, 17" deep 15u rack running $250

1 Palo Alto PA-5260 Next Gen Security Firewall Appliance $500

1 867793-B21 HP Synergy 50Gb Interconnect Link Module $400

((this next part will be per server case 3x total per rack))

1 sliger CX4170a case with water cooler $500

1 Supermicro M12SWA-TF sWRX80 Motherboard $600

1 threadripper pro 5995wx $2200

8 sticks 512gb total ddr4-3200 rdimms $890

1 EVGA SuperNOVA 1600 P+, 80+ PLATINUM 1600W power supply $225

1 LSI SAS 9300-16I 12GB/S HBA BUS ADAPTER CARD $60

((1 of the mellanox cards will be run directly between the 3 nodes in each server rack to run inteneral game server instance data. The second is to run external connections and none game server “server environment”))
2 Mellanox ConnectX-4 EN Dual Port 100G QSFP28 PCI Card MCX416A-CCAT $300 total

3 Seagate Exos ST10000NM0096 10TB Enterprise 7.2K 12Gb/s 256MB 3.5" SAS Hard Drive totaling $350

2 Crucial P3 Plus 2TB M.2 2280 SSD totaling $250

this brings the rack all togeather with 192 cores running at 4.5 max clock 1.5 tb of ram 100 tb of storage and great internal and external rack communication for $17,275. that is less then alot of single workstations out there with so much more power to it. I will also be building a second layer system like the one I described in my first post.

clicking order now. price should drop right after. lol

3

@wendell some of my programming is custom code to optimize this system setup but I would love to see what you could/would do with this hardware setup. / I bet you have most of it laying around waiting to be a tightly clustered super rack. I’d watch that video!

4

make it 2022; 2019 is EOL and all indications are pointing to 2025 requiring an active Azure license to boot your on prem server

We’re planning on migrating clients to ProxMox when 2022 EOL’s

5

@TryTwiceMedia I will not be running hyper-v at all. The first post was reporting what a new friends setup was for his game dev team that he let me check out and run some test. I am not running proxmox either but a custom ceph and rancher kubernetes, For out side the game instance server. Inside the game server instance I have a different version of CEPH and rancher with kalfka, mongodb(for now), motion servers, and some unreal engine online services including the packed unreal server wrapped in some custom c++ that will automatically setup including using the appropriate nics. aslong as all rackes are setup the same adding or moving game server instances shoud be easy.

1 Like