Automating and Protecting Minecraft Server w/SystemD and SELinux

So this is here partially for y’all @SgtAwesomesauce @kreestuh @wendell for this years devember and mostly here for peoples reference. A constant struggle was restart resilience of a server crash. Originally we couldnt test this process in prod so we just let it slide. But I want you to have it for future reference. Considering you might go creative this year. You probably want it to restart after a crash right?

Wendell could you potentially tarball up and send to me the worlds? Or at the very least the spawn world that I created for you all? Id like to have it. I created it as a one off with inspiration from various builds. Turns out its damn nice and id like it for my own server/archive purposes.

Lets begin

Im running Fedora in this setup so the directions will be tailored for Fedora. Lets install some prerequisites before we run paperMC

sudo dnf groupinstall -y "development tools"
sudo dnf install -y java-latest-openjdk.x86_64
sudo dnf -y install setroubleshoot

Now we can build MCRCON which is what we will need to access the remote console. However before we do so its up to the server owner whether or not he wants to create a minecraft user. I recommend this step and its pretty easy to do.

sudo useradd -r -m -U -d (path to MC install directory in my case ZFS vols) -s /bin/bash minecraft

My actual command looks like this

sudo useradd -r -m -U -d /mnt/OnePoint21GigaWatts/games/PaperMC -s /bin/bash minecraft

What this is going to do is install the home directory of this user we create on our ZFS pool. I prefer it this way and honestly the load times are just as sweet as my SSD so it has little effect.

Now, switch to Minecraft user by executing the following command:

sudo su - minecraft

Now we are going to make a few directories.

mkdir -p ~/{images,mcrcon,server}

The directories are simple. The images directory is for storing a regular backup image of the server. The mcrcon directory is for our remote console we will build from sauce. The server directory will store the Minecraft server and the related items.

Now we can build MC-RCON. Theres a git for it. Its ultra simple.

cd ~/mcrcon && git clone https://github.com/Tiiffi/mcrcon.git
cd ~/mcrcon/mcrcon 

(this is dirty it could be neater)
Run it through GCC. (One does not need all these flags I just like using the flags for my arch)

gcc -std=gnu12 -pedantic -Wall -Wextra -znver3 -O3 -s -o mcrcon mcrcon.c

You can just as easily use a generic GCC11

gcc -std=gnu11 -pedantic -Wall -Wextra -O2 -s -o mcrcon mcrcon.c

Now make it executable and run it to test

chmod +x mcrcon && ./mcrcon ?

Snag the MC jar file. Grab it from here for paperMC Downloads – PaperMC

wget <URL> -P ~/server 

Lets move to the directory and execute the server first to handle some of the operations that will create server.properties and the acceptance of the EULA before we write the systemD service.

cd ~/server
java -Xms10G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Xlog:gc*:logs/gc.log:time,uptime:filecount=5,filesize=10M -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true -jar paper-1.19-34.jar --nogui

This will fail for needing the sign the EULA and you can do that via

nvim ~/server/eula.txt

and setting eula=true. Now before running again lets setup the RCON in the server.properties file. Find the following and set/edit as follows

nvim ~/server/server.properties

Values to change:

  • rcon.port=25575 (or change it if you like)
  • rcon.password=(I set this to a random long string of alphanumerics+special characters)
  • enable-rcon=true

We need to create a systemD file that integrates nicely, drops privileges and does not make SELinux freak out if enabled.

[Unit]
Description=L1 Minecraft Server Service File
After=network.target # We want it up after network has been handled

[Service]
User=minecraft
Nice=-20 # Set niceness level. For me I like it to be the priority
SuccessExitStatus=0 1 # Define exit statuses
# System Protection Flags and Privilege Dropping
ProtectSystem=full
PrivateDevices=true
ProtectHome=true
NoNewPrivileges=true
WorkingDirectory=<path to SERVER directory>
# Tweak these flags as you see fit
ExecStart=bash -c 'java -Xms10G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Xlog:gc*:logs/gc.log:time,uptime:filecount=5,filesize=10M -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true -jar paper-1.19-34.jar --nogui'
ExecStop=bash -c '<path-to-mcrcon-executable> stop'
Restart=always # Restart if it fails
RestartSec=3 # How long to wait between failures

[Install]
WantedBy=multi-user.target

Enable but do NOT start the service yet.

sudo systemctl enable minecraft.service

We need a few bash aliases to make our life a bit easier when we shell in to access our RCON. Now I have defined them but I leave their declation UP to you. You can either set them globally in /etc/environment or in the .bashrc of the minecraft user (Dont forget to source it)

# MC RCON Variables
export MCRCON_PASS="<password>"
export MCRCON_HOST="127.0.0.1"
export MCRCON_PORT="<port>"
# MC RCON Terminal Access
alias mct='<path to mcrcon executable> -t'
# MC RCON Stop Server
alias mcs= '<path to mcrcon executable> stop'

Pretty simple right? and once you source this its easy to RCON in

Now I am going to leave the firewall configuration, tunnel configuration and backup configuration up to you. Everyone has their own style and it seems futile to recommend a specific way. If you are using systemD then systemD should handle the scheduling of your backups.

Start the minecraft server

sudo systemctl start minecraft

In a TMUX window you can monitor in a divided window the logs of the server by using the journal.

sudo journalctl -feu minecraft

You can access RCON via the alias I set above for terminal model in the other side of the divided window via

mct

IF you have SELinux issues like I did because you dont run permissive mode… Make sure systemd-minecraft has a local policy

ausearch -ts <BEFORE SERVICE RESTART> -te now --raw | audit2allow -M allow-systemd-minecraft; semodule -X 300 -i allow-systemd-minecraft.pp

That is really all there is to it. Good Luck, Have Fun, Configure your stuff and Profit :smiley:

P.S This works flawlessly for my own server. Its really nice to just pull the logs from systemD. Now this could be customized so much more but I kept it simple as a starter place for everyone.

You can extend this concept for:

  • The modern, next-generation Minecraft server proxy forked by PaperMC
  • PaperMCs fork of the BungeeCord software, with improved Forge support and more features.
  • Any additional servers you need to run just tweak the configs and make the new ones for each

:sunglasses:

~PLL Out

7 Likes

(Reserved)

Just a note folks. If you do a multiworld

Augment this and the SElinux policies to all be under the same minecraft user

Make the following based on what was written above

mc_server_1.service
mc_server_2.service
mc_server_3.service
mc_server_spawn.service

A good hierarchial example is like the L1T multiworld setup

mc_devember_2019.service
mc_devember_2020.service
mc_devember_2021.service
mc_devember_spawn.service

Thats how id do it. Let the servers handle the warps and linking. Since they all have shared permissions from a user and drop them appropriately, this setup should not have issues and it keeps the resiliency as the original setup details. If a server stops it will restart

Then make the associated aliases so you can access each mcrcon. What you will have to do is make aliases for each full command. Keep the passwords for each the same and so not set a global port. You probably catch my drift.:

# MC RCON Variables
export MCRCON_PASS="<password>"
export MCRCON_HOST="127.0.0.1"
# export MCRCON_PORT="<port>" COMMENT OUT
# MC RCON Terminal Access
alias mcts='<path to mcrcon executable> -t -P <spawnServer_port>'
alias mct1='<path to mcrcon executable> -t -P <world1Server_port>'
alias mct2='<path to mcrcon executable> -t -P <world2Server_port>'
alias mct3='<path to mcrcon executable> -t -P <world3Server_port>'
# MC RCON Stop Server
alias mcss= '<path to mcrcon executable> stop-P <spawnServer_port>'
alias mcs1= '<path to mcrcon executable> stop -P <world1Server_port>'
alias mcs2= '<path to mcrcon executable> stop -P <world2Server_port>'
alias mcs3= '<path to mcrcon executable> stop -P <world3Server_port>'

These ports will change in your systemD script as well so make sure you do it there too!

Plan it out. Execute. Debug. Profit

1 Like

When I’ve hosted a minecraft server in the past, I’ve done it in a docker container and put the parameters for it in a docker-compose script.

services:
        minecraft-creative:
            container_name: minecraft-creative
            image: itzg/minecraft-server
            ports:
              - 25565:25565
            environment:
              EULA: "TRUE"
              VERSION: 1.19
              TYPE: PAPER
              MAX_MEMORY: "8G"
              TZ: "America/New_York"
            volumes:
              - /mnt/archive/minecraft/creative:/data
            restart: "unless-stopped"

In my script above, /mnt/archive/minecraft/creative is the directory on the main system that I am mounting to /data inside the container. Other environment variables that the container supports can go in the environment section like above. I’ve accepted the EULA, set the server timezone, as well as specified the server version, type, and max heap memory.

I have a systemd task for docker that restarts all the existing containers at startup. This requires each of the containers to have a restart policy of “always” or “unless-stopped” (in case it was manually stopped before shutdown, it won’t automatically be restarted), but this is also useful in ensuring the container automatically restarts in case of a server crash.

The docker image itself is quite good. It contains lots of information about all the different parameters that it supports, it can run any number of server distributions (vanilla, fabric, paper, forge, spigot, etc) and it also by default will run the rcon-cli server console shell inside the container. You can map this to an external port if you want to in the docker-compose script, or not if you want to limit access to yourself via shell access on your server via this command:

docker-compose exec minecraft-creative rcon-cli

From a security point of view, if there’s another minecraft exploit like the log4j exploit late last year, and someone is able to exploit it and gain shell access, here they’re limited to inside the container with the portion of the filesystem mounted for the container, rather than your entire server.

One potential issue here (and also an issue with the systemd implementation of @PhaseLockedLoop above) is if the server gets in a sideways state and becomes unplayable, but the java process stays running and doesn’t actually crash, neither of these implementations will be able to tell that anything is wrong and will still require a human to restart the service or the container.

Its extremely hard to detect this without a bot. Shitty java app is a shitty java app

Though im open to suggestions?

1 Like

:warning: Dumb idea ahead!
Have the server play on the server it is serving!
Setup a bedrock box and have a bot put down a piece of dirt above itself, then destroy it, repeat. A non-contrast change in hotbarslot 1 after time X would mean server soft failure.