Devember - Screep Studio

Shecks · November 27, 2018, 12:23am

Day -5

Hi,

This is my blog for my #devember2k18 project Screep Studio.

The plan is to develop (or at least make a start on developing) a stand-alone client / IDE for the Screeps MMO. For those not familiar with the game it’s a sand-boxed PVP battle MMO aimed at either programmers or people who are interested in learning to program while having fun gaming. The idea is to gain control of rooms within the world by populating empty rooms or taking them form your opponents. However in order to do this you need to program your own little army of A.I “Creeps” using JavaScript (there are “cross compilers” for other languages too)

This is what the website says about it:-

What is Screeps?
“Screeps means “scripting creeps.” It’s an open-source sandbox MMO RTS game for programmers, wherein the core mechanic is programming your units’ AI. You control your colony by writing JavaScript which operate 24/7 in the single persistent real-time world filled by other players on par with you.”

It’s a pretty cool game but I haven’t played it for a while and wanted to get back into it. I also wanted to play around with the private server they released so I could develop my A.I without having to worry about being wiped out by all the top players on the live site.

On the live server you can play using your web browser or a custom client that runs as part of the Steam client. But, if you want to play on your own private server you have to use the Steam client as the web server part of the game was not open sourced with the private server.

And to make a long story even longer … the Steam client is “pants!” (YMMV but at least in my case I just can get it to run well on Linux) … so I’ve decided to have a go at writing a stand alone client that will at least be usable with the private server.

The Current Plan

Find out as much as I can about the Screeps API as it’s not officially documented. As far as I know there are two APIs required to build a client. The first is a REST API that seems to be used to query the state of objects in the game (e.g getting information about the world, the player, the rooms they control and their structure etc) and the second is a Websocket pub/sub API for real time game object updates. I’ve done bit of research and found some open source clients written in various languages with partial information on these API and since the private server is open source I should be able to get any missing info from the source.
In the long term I would like to develop a reasonably usable game client / IDE at least something that can be used as an alternate to the Steam client for private servers but for #devember2k18 I aim to just make a start and build the network and UI components to render a live view of a single game room with all the game components (spawns, creeps, walls, roads, towers, storage etc). If that goes well I would like to add support for a read-only console to view script output. Long term I would like to add a built in JavaScript editor and room object creation UI.
The client will be written in C/C++ using the Qt framework because I’d like to be able to build it for Linux and Windows.

I actually started playing around with the idea on Friday 23rd so I put together a basic proof of concept just to figure out if I could render the terrain using data from the server.

Since it’s not December yet, I’m calling that a prototype hopefully it will look more like the following when it’s done (if I can find some free graphics because I can’t draw)

Contract
I, Shecks, will participate in the next Devember. My Devember will be a game client / IDE for the Screeps MMO. I promise I will program for my Devember for at least an hour, every day of the next December. I will also write a daily public devlog and will make the produced code publicly available on the internet. No matter what, I will keep my promise.

Right, I’m off to do some more “prototyping”

Shecks

FrogE · November 27, 2018, 1:57pm

I always thought this would be a cool idea for a game; glad it exists. Good luck in your Devember!

Dynamic_Gravity · November 27, 2018, 5:21pm

Looking Great!

Shecks · November 27, 2018, 6:53pm

Day -4

Preparation Progress (or not)

Sadly it turns our that my Screeps server running on the beastly “carbug”, a re-purposed Minix Z64 Android TV box running Fedora Server :D, is not up to the job of providing a stable platform of my #devember project.

I think the Screeps sever processes are leaking memory because they are eating through the 2Gb of RAM and then being killed (or partially killed) by the OS. I was really hoping I could use carbug because it’s silent so can run it 24/7 which would give test A.I time to build the different structures I need as test data for the client development.

So today’s “preparation” task will be to setup up a new Screeps server running in a VM on my Dev box.

If that doesn’t work I can always go overkill and set up an entire set of server shards on “blue-midget” … (which also doubles as a space heater). With 144Gb of RAM I should get a couple of days up time

At least I am learning more about Linux and NodeJs configuration.

Shecks

Shecks · November 28, 2018, 6:38pm

Day -3

Planning/Preparation Progress

I have a working test server running in a VM now (thanks to @SgtAwesomesauce for helping me with a Linux config problem I was banging my head against and couldn’t see the forest for the trees)
After I got the VM working I went back and re-built my carbug server, it seems a little more stable (I think there were two copies of the server processes running after my previous setup efforts ). I’m still not sure if the Screeps server processes are leaking memory but it seems to have settled at about 950mb RAM usage of the 2Gb. No idea how people are reporting that the Screeps private server runs on a Raspberry Pi.
I have enough information on the REST and Websockets API to start coding my RestNetworkManager and WebsocketNetworkManager classes. Once I have a basic idea in mind I like to start writing code pretty soon an flesh out the design as I go rather than plan down to pseudo code.

Am I allowed to start writing code yet?

Shecks

SgtAwesomesauce · November 28, 2018, 6:39pm

You can do anything you want. My devember project is one I’ve been working on for about a month already.

Shecks · November 28, 2018, 6:47pm

Excellent … and so it begins …

#ifndef _SCREEPSCLIENT_H
#define _SCREEPSCLIENT_H

#include <QObject>

class ScreepsClient : public QObject {
    Q_OBJECT
public:
    explicit ScreepsClient(QObject* parent = nullptr);

    // TODO: Make it so...
    
signals:

public slots:
};

#endif // _SCREEPSCLIENT_H

Shecks · November 28, 2018, 7:07pm

I think I spoke too soon. It looks like the Screeps processes are munching through memory again and crarbug is about to die.

**Edit: As an aside, does anyone know if those are individual threads being displayed by HTOP because I really don’t think there are supposed to be that many processes running.

***Edit 2:

[shecks@carbug ~]$ ps -U screeps
  PID TTY          TIME CMD
  953 ?        00:00:00 node
  970 ?        01:09:43 node
  981 ?        00:06:41 node
  982 ?        00:58:29 node
  988 ?        16:18:16 node
  989 ?        02:43:55 node
  990 ?        02:44:25 node
[shecks@carbug ~]$

That looks more like it … I guess the entries above 1000 are thread?

SgtAwesomesauce · November 28, 2018, 7:24pm

Did you look at system memory requirements?

I’ve got a feeling, since you were talking about NN earlier, that this server is going to use a lot of memory, as it goes on.

Also, Node is horribly inefficient when it comes to memory usage, but it’s asynchronous, so w/e.

There’s a correlation, but that’s not causation. look at 998 and 999. Those are look like threads as well.

simbyotic · November 28, 2018, 7:25pm

Semi-unrelated, I didn’t know about this game but this thread sold me on it (after some browsing around). Good luck with the project!

Shecks · November 28, 2018, 7:33pm

Check out the live server, the playing experience is a lot better with their large server infrastructure and you can play using the web client in your browser which, IMHO, performs a lot better than the Steam Client they introduced a while back. I think there’s also a free trial and an interactive training mode/tutorial to get you started too.

Shecks · November 28, 2018, 7:54pm

Unfortunately there’s not a lot of information available on the hardware requirements for the private server. It’s something players were asking for for a long time and I get the impression they just hacked together a subset of the live server components.

From a business point of view it doesn’t really make sense for them to release a production quality private server for free and hope that by not open sourcing the client that people would continue to pay for the game when it’s simple enough to build your own client for it. So I’d say there wasn’t a lot of effort made to make the private server performant/resource efficient.

In saying that, I have see some blog/forum posts where players are using Raspberry Pi based private servers for single player use. With the default install there are 4 very basic bots created on the server so I am going to look into killing 3 of those for a start.

I did a little bit of reading an found that they are child processes (I am still in Windows mode and thinking threads rather than processes on Linux). I don’t know much about Node JS but I’m going to have a dig through the code to see if I can tweak the number of processes a bit more (there’s one config file where you can config “runner processes” which I’ve already limited don’t but there’s still a lot more going on there than expected)

More research to be done …

Shecks · November 28, 2018, 9:23pm

Day -3 (Still)

I’m gonna call this my starting point. I have the private server up an running and I have one bot working away in the world which I am going to use to generate my test data. I’ll write my own bot later when I need to build roads and more advances structures.

Plans

I will work on improving the graphics later but a rough rendering of the static room terrain is workable for the moment
The next step will be to work on getting data for the live game entities which can be seen in the Steam Screeps client on the right above. I’m going to work on the websocket API so I can subscribe to the event streap and then render some place holders for the game entities.

If anyone’s interested, here’s an example of the JSON the server spits out for room terrain:-

{
	"ok": 1,
	"terrain": [
		{
			"room": "W9N1",
			"terrain":"
				1111100011111111111111111111111111111111111111111
				1111110000011111111111111111111110000000000000011
				1111110000000011111111111111111100000000000000000
				1111110000011000111000111111100000000000000000000
				0111111000011110000000001111100000000000000000000
				0011111000000111000000000011110000000000000000000
				0001111000000011000000000000110000000000000000000
				0000111100000011100000000000000000000000000000000
				0000011110000111110000000000000000010000000000000
				0000001111000111111100000002000000011100000000000
				0000000111100001111110000022220000011111002000000
				0000000011111000000000000022222001111111100200000
				0000000001111110000000000002322001111111110000100
				0000000000011111100000000000332200011111110000111
				0000111110000111130000000000003220001111110000001
				0000111111100011111100000000000022000111000000000
				<snip..>
				",

			"type": "terrain",
			"_id": "8c670771e7bee62",
			"meta": {
				"revision": 0,
				"created": 1477476556412,
				"version": 0
			},
			"$loki": 29
		}
	]
}

Shecks

AnotherDev · November 28, 2018, 9:49pm

Dynamic_Gravity · November 28, 2018, 10:11pm

I’m stealing that jif

SgtAwesomesauce · November 28, 2018, 10:11pm

jif

REEEE

Dynamic_Gravity · November 28, 2018, 10:12pm

That’s why I do it, to piss the internet off.

FrogE · November 28, 2018, 10:16pm

I’m confused!

Or

?

SgtAwesomesauce · November 28, 2018, 10:17pm

Dynamic_Gravity · November 28, 2018, 10:17pm

if (gif == jif)

debate