[WIP] this guide requires a tarball that isn’t yet public - working on making it a git repo instead, all files are text-files and inspectable.
This guide aims to encompass s6 installation on multiple distros.
What's "s6" you might ask?
It stands for “skarnet.org's small and secure supervision software suite”.
Its init, named s6-linux-init component functions in a similar fashion to daemon-tools and runit, i.e. it starts processes whenever they crash, exit, or are killed, unless you have a “down” file in the service directory (which happens only when you disable the service or it wasn’t enabled to begin with, or at startup when its dependency service is not started yet).
The s6-rc is where magic happens. This is more akin to OpenRC, Upstart and systemd, where a dependency graph is created for all programs, based on user input / configuration. Everything is user configurable, but package maintainers should take care of most of the dependencies for you. User services will typically land in a “multi-user” bundle (basically a runlevel on steroids), where you can define additional dependencies for each service.
You can customize your own service as easily as making a new directory with a custom name (that doesn’t conflict with the original, so package updates from the repo don’t overwrite your changes), copying all the old files in here and modifying them to your needs and finally touch a file with the same name as your custom service to the bundle name. In 4 short commands, you got your custom service definition.
Scope of this wiki.
A separate wiki entry is maintained for how to use s6 without a frontend like 66. This is where most of the explanation on how s6 works and how to use it will go. [WIP needs a lot of updating] This wiki is instead aimed at how to install s6 and the custom “runlevel” layout that I’m hoping will become a standard.
Easy to follow Beginner Guide on s6 Starter Pack
The initial focus for now is on Void, Alpine and Devuan, with the later two coming after a while. Here is where you can help improve this wiki. If you want to test the guide and the services on other distros and you find something’s not working and find a way to fix it it, edit this wiki or add a comment.
Distros might have custom binary names (like cronie, crond, cronie-crond etc.), so services might not easily translate from one distro to another. However, this wiki aims to standardize the “runlevels” (for lack of a better term, as s6 doesn’t work with runlevels, but “bundles,” which are more flexible) and to make it easy for a distro to just copy the service file, slightly modify the run file and be off to the races.
Part of this wiki will end up on github, to make it easy to clone the services and make distro specific branches of services (master branch = generic, void branch, alpine branch, devuan branch, gentoo branch etc).
This method is highly volatile, dealing with “make install” and subject to changes. [WIP] on xbps-src packages, but this method will remain part of the wiki for a generic, distro-agnostic guide (which will contain explanations more than commands).
Artistic considerations. Explaining how the custom startup layout works.
Explanations on s6-linux-init are given in comments in the deploy script, which will turn into the generic script [WIP] at some point. The focus is on s6-rc, where you’ll be spending most of your time (which isn’t a lot, unless you’re going to be one of the maintainers or sysadmins writing the service files).
The /etc/s6-rc folders have 4 top-level folders (aside from compiled stuff you’ll make):
- config
- s6sv
- scripts
- source
config
The “config” folder contains configurations used by the services, mostly environment variables, which counts as log lines for loggers, or custom flags for services (like if you want to enable more verbosity without recompiling the s6-rc-db, but by just restarting the service outside of s6-rc, in s6-svscan - check the s6 starter pack wiki if this sounds confusing).
scripts
This is intentionally addressed early, because it’s not much to mention. The scripts folder is the place where scripts are ran outside the actual service. The s6-rc services, particularly oneshots, should be very small scripts written in execline. If you need to call a huge shell script, it’s best to make a small execline oneshot that calls your huge script from /etc/s6-rc/scripts.
This way, if you modify your script, you don’t have to recompile the s6-rc db, just restart the service and it’ll execute the modified script with the same name and location.
There’s a script [WIP] that reapplies all the dependencies for all the services, to make sure that nothing tries to start early and everything has at least the dependency above them. At some point this will probably be automated with a daemon.
Another [WIP] is automating the enablement of services when new things pop up in the sources folder (so users don’t have to touch).
s6sv
The “s6sv” is more like an “all services” folder, either enabled or disabled. This is where everything packaged by repo maintainers should go. The folders structure underneath s6sv mirrors the one in “source” for the reason that, to enable services a user should only have to add them to “source” and touch a file with the name of the folder in the bundle contents.d folder (coming to this in a bit).
Sysadmins could add service folders in either s6sv or source folders. If it’s in s6sv, make sure they’re not the same name as what comes from the repo, otherwise it’ll get overwritten with updates.
What I recommend is to name your custom service folders with with the suffix “-custom” or “-$(hostname).” If you manage more than one host, -custom would be the generic service that goes everywhere, whilst a host’s name as the suffix means the service would be custom for that host alone. You may create your own naming scheme (like add a suffix based on role, like “sshd-databases” or something, that goes on specific servers you want - the sky’s not the limit).
As far as custom / default services that go in the source directory, it doesn’t matter, because it’s going to be either symbolic links to the s6sv folders, or a custom folder. Bonus points: if you put your -custom service in s6sv, you can do a symlink in “source” with the same name as the repo provider service (/etc/s6-rc/source/20-multi-user/dhcpcd → /etc/s6-rc/s6sv/20-multi-user/dhcpcd-custom), to make a service easier to manage everywhere (you literally invoke “s6-rc -u change dhcpcd” instead of using the “dhcpcd-custom” name).
source
The “source” directory is the most important, as it’s the “source of truth” on the system. While it’s relatively easy to analyze a compiled s6-rc db with the command aptly named s6-rc-db, the command doesn’t show you the bundle names when doing dependencies, making it a bit harder to “decipher” (although not impossible).
Here, I go with the philosophy that source doesn’t need to be tracked. At most, an advanced user would either user git to track the changes in source, or would have a source-initX associated with each s6-rc db compiled-initX database. One could literally copy the “source” folder to the compiled-initX that was compiled from that source folder (s6-rc won’t care about it). That way, you know what services you have defined where.
That’s optional and up to the user. Just keep in mind that if you want to see how your system changed, if you don’t keep a copy of the “source” associated with the s6-rc compiled db, after many changes, you won’t know what went wrong with the current source folder. This is generally fine, because, by this design, users shouldn’t be able to break their system (the core services that makes the OS tick should still come up no matter what).
The folder structure of “source” contains the following:
- 00-default
- 01-ok-all
- 05-ok-init
- 10-ok-local
- 20-ok-multi-user
The folders have names in them just to make it easy for users and admins to understand the dependency tree of the system. Think of these folders like you would of runlevels on steroids.
Services under 20 depend on 10, 10 depends on 5, 5 depends on 1, 1 depends on 00 and 0 has no dependencies. This makes it extremely important that the lower-level stuff works. Ideally, users won’t modify anything bellow 20, but at most, users should only work with 10 and above. For extremely experienced users, the symlinks from 05 and lower can be removed and replaced with their preferred services. The lower-level stuff should ideally be reserved for distro maintainers and should rarely be touched.
Dependencies would look something like this:
00
\-- ok-init
\-- ok-local
\-- ok-multi-user
ok-init
\-- init-dev
\-- init-{btrfs,zfs,dm-crypt,dmraid}
\-- init-proc
\-- init-sys
\-- udevd
\-- init-static-devnodes
\-- init-modules-load
\-- mount-rw
\-- mount-filesystems
ok-local
\-- net-lo
\-- rc-local
\-- multi-user (this is just a welcome to multi-user-land, like 00)
ok-multi-user
\-- acpid-log
\-- acpid
\-- sshd
\-- dhcpcd
\-- iscsid-log
\-- iscsid
\-- iscsi-login
\-- libvirtd
\-- libvirtd-vm001
\-- libvirtd-vm002
\-- libvirtd-vm003
\-- libvirtd-vm004
\-- libvirtd-vm005
\-- cronie
\-- chronyd
You can kinda see how services in ok-multi-user are becoming slightly more parallel, like acpid, sshd (unless you need dhcp, but mine only needs rc.local to run, since that’s where static networking gets enabled), cronie, chronyd and more. Only very few would have crazy dependencies like libvirtd needing iscsi in order to start VMs on system start-up.
Conversely, you can see how lower-level stuff is more sequential. Traditionally, distros would write shell scripts to take care of the early init stuff, before launching PID 1. And some, like void, would have to run shutdown scripts (runit stage3), where after the runit process has killed everything, the system cleanup process starts.
This is not necessary with s6, since everything is handled in reverse-order on shutdown, using either friendly kill signals for longruns, or using the down scripts for oneshots (if applicable). There is however a “rc.shutdown.final” stage, where things can be done after the services are stopped and the file systems have been unmounted. You could literally mkfs your rootfs or zfs-rollback if you’d be using a stateless system (although this particular scenario is generally better performed at boot, since it’s possible you’ll have a forced poweroff that won’t clean your FS). The rc.shutdown.final probably has its own uses, I just haven’t found them yet.
The numeric value won’t be present in s6-rc’s db, this is just for users to make some sense of the dependencies and to make it easy to automate application of dependencies. In s6-rc, you’ll only see the names after the number prefix.
The reason this is not in order, is to give users the ability to add custom “runlevels” in-between what’s currently predefined. For example, Laurent Bercot has 2 more bundles in his examples, called “ok-lan” and “ok-wan.” These aren’t used here, but could be applies by an admin running a router to trigger services.
default
This is the main OS bundle. When the OS starts, unless the bootloader’s cmdline is modified, your bootloader should always be calling on “default” variable (this is true for runit, openrc, s6 and I believe sysvinit and systemd too). Their inits are the ones that call on “default.”
Inside it, you’ll find a service called “00,” which is just an informal service, just outputs a welcome on the screen (notifies you “init stage2” is happening, i.e. you just moved past s6-svscan, a.k.a. PID 1 is active).
Another thing you’ll find in default is the “ok-all” bundle, which is another bundle. This doesn’t need to be modified, assuming users are using the already existing “top-level” bundles on the system (the numeric folders).
ok-all
This contains all the other “ok-” bundles. This is more of a management thing, to make it easy to find what services are going to actually be activated. ok-all is not a runlevel in itself, it’s literally meant to start all the oks in parallel. The ok-all bundle could literally be renamed “enabled” and things would start to make sense.
Under the 01-ok-all, you’ll find the other oks. All oks are bundles. Under each, you’ll find the “contents.d” folder, which shows you what and when things will get activated. Keep in mind that the dependency happens at the service-level, not on a bundle (I discussed this shortly with skarnet, it’s intentional to avoid some weird edge-cases - and it’s also been discussed during the inception of s6).
ok-init
This is where the real OS initialization / userland begins. This is where udev is activated, kernel modules loaded, urandom activated, file systems are mounted, TTYs enabled and so on. You really shouldn’t be touching this.
The idea with the 05-ok-init folder is to basically “for each file in 05-ok-init/*, touch the file in 01-ok-all/ok-init/contents.d/” - it’s all meant to make the logic of activating services easy.
Similarly with the folder structure, all the items under 05-ok-init need to be dependent on 00 (the stage2 startup, that welcome message basically). So, the logic is “for each [ longrun or oneshot ] in 05-ok-init, touch 05-ok-init/service/dependencies.d/00” - this makes all the services in ok-init bundle depend on the 00 service.
Atomic services in ok-init can also have dependencies on one-another (e.g. local disk mount-rw depends on kmods being loaded, which in turn depends on udev being enabled, which in turn depends on dev, proc and sys fs to be mounted).
One important aspect of ok-init and the only time where you can apply a setting on a bundle that gets applied to everything in the bundle, is the “flag-essential” file. ok-init has this flag-essential, which basically means these services won’t go down with just an “s6-rc -d,” but you need to utilize the harder “s6-rc -D” command (this is invoked by the shutdown command, under s6-linux-init/current/scripts.rc.shutdown).
If all services are set to restart, these won’t be impacted. Also, most of ok-init atomic-services are oneshots that don’t have a “down” service (e.g. a remount root rw won’t have a umount root service file - for 1, because s6 handles unmounting of fs by itself and 2, because services with no “down” services are considered they’ll never go down anyway; once activated, the system state is in a good condition to allow other services to start and the state won’t change in the service).
10-ok-local
This is a very short bundle with “pre-mutli-user services.” If you need to mount custom file systems that aren’t handled by fstab (idk, autofs or something?), here’s where you’d be adding things. Currently there’s only a loopback address activation and the rc-local service, which calls on /etc/rc.local file (remnant from void’s runit). The script can be edited to also activate more custom scripts from “/etc/local.d/.start" and made to deactivate by "/etc/local.d/.stop.” Everything in “down” is commented out, thus the “down” script is considered empty.
20-ok-multi-user
This is where most userland begins. DHCP gets activated, SSHD (which depends on DHCPCD or not) starts, so does a cron daemon and an NTP daemon. This is the place most users will be spending time in, by linking services from /etc/s6-rc/s6sv/20-ok-multi-user to /etc/s6-rc/source/20-ok-multi-user.
Just like before, services in the ok-multi-user bundle can have dependencies amongst themselves. For example, sshd
Particularly aesthetic decisions:
The sshd, cronie and chronyd services are not enabled by default. That’s because Void doesn’t enable sshd by default and the NTP and cron daemons are not installed by default, but are left to the user’s choice.
As a homework for you, the first things you’ll want to do once you deploy s6 is to symlink those services to their respective sources locations and touch the file names under the 01-ok-all/ok-multi-user/contents.d folder.
Note: “sshd” is a bundle which includes “openssh-server” and “openssh-server-log.” That means you only need to touch /etc/s6-rc/source/01-ok-all/ok-multi-user/contents.d/sshd to make both of them to get enabled (after an s6-rc db recompilation, of course).
Void install
sudo xbps-install -Su && sudo reboot
sudo xbps-install -S git make gcc
echo ignorepkg=runit | sudo tee -a /etc/xbps.d/99-runit.conf
echo ignorepkg=runit-void | sudo tee -a /etc/xbps.d/99-runit.conf
git clone git://git.skarnet.org/skalibs
cd skalibs
./configure && make && sudo make install
cd ..
git clone git://git.skarnet.org/execline
cd execline
./configure && make && sudo make install
cd ..
git clone git://git.skarnet.org/s6
cd s6
./configure && make && sudo make install
cd ..
git clone git://git.skarnet.org/s6-linux-utils
cd s6-linux-utils
./configure && make && sudo make install
cd ..
git clone git://git.skarnet.org/s6-portable-utils
cd s6-portable-utils
./configure && make && sudo make install
cd ..
git clone git://git.skarnet.org/s6-rc
cd s6-rc
./configure && make && sudo make install
cd ..
[ -e /etc/s6-linux-init ] && sudo rm -rf /etc/s6-linux-init
git clone git://git.skarnet.org/s6-linux-init
cd s6-linux-init
./configure && make && sudo make install
cd ..
sudo mv /sbin/init /sbin/init-void
sudo mv /sbin/shutdown /sbin/shutdown-void
sudo mv /sbin/halt /sbin/halt-void
sudo s6-ln -snf halt-void poweroff
sudo s6-ln -snf halt-void reboot
sudo mv /sbin/poweroff /sbin/poweroff-void
sudo mv /sbin/reboot /sbin/reboot-void
# If your linux kernel's customized, check if devtmpfs is mounted by the kernel, if not, add "-d /dev" to s6-linux-init-maker
# grep devtmpfs /proc/mounts && s6-linux-init-maker -1 -G "/sbin/getty 38400 tty2" /tmp/blah || s6-linux-init-maker -1 -d /dev -G "/sbin/getty 38400 tty2" /tmp/blah
sudo cp -a /etc/s6-linux-init/skel/rc.init /etc/s6-linux-init/skel/rc.init.bkp
sudo cp -a /etc/s6-linux-init/skel/runlevel /etc/s6-linux-init/skel/runlevel.bkp
sudo cp -a /etc/s6-linux-init/skel/rc.shutdown /etc/s6-linux-init/skel/rc.shutdown.bkp
sed 's/# s6-rc-init/s6-rc-init/' /etc/s6-linux-init/skel/rc.init | sed 's/# exec \/etc\/s6-linux-init\/current/exec \/etc\/s6-linux-init\/current/' | tee /tmp/rc.init.tmp
cat /tmp/rc.init.tmp | sudo tee /etc/s6-linux-init/skel/rc.init
sed 's/# exec s6-rc -v2/exec s6-rc -v2/' /etc/s6-linux-init/skel/runlevel | tee /tmp/runlevel.tmp
cat /tmp/runlevel.tmp | sudo tee /etc/s6-linux-init/skel/runlevel
sed 's/# exec s6-rc -v2/exec s6-rc -v2/' /etc/s6-linux-init/skel/rc.shutdown | tee /tmp/rc.shutdown.tmp
cat /tmp/rc.shutdown.tmp | sudo tee /etc/s6-linux-init/skel/rc.shutdown
# [WIP] to publish the source services for s6-rc.
git clone something-at-some-point
cd something-at-some-point
sudo cp -r source /etc/s6-rc/
sudo cp -r config /etc/s6-rc/
sudo cp -r scripts /etc/s6-rc/
sudo find /etc/s6-rc -type d -exec chmod 755 {} \;
sudo find /etc/s6-rc -type f -exec chmod 644 {} \;
chown -R root:root /etc/s6-rc
STAMP=$(date +%s)
sudo s6-linux-init-maker -c /etc/s6-linux-init/init-${STAMP} -G "/usr/bin/agetty tty2 38400 linux" -1 /etc/s6-linux-init/init-${STAMP}
sudo s6-ln -s /etc/s6-linux-init/init-${STAMP} /etc/s6-linux-init/current
sudo s6-rc-compile /etc/s6-rc/compiled-init1 /etc/s6-rc/source/*
sudo s6-ln -snf /etc/s6-rc/compiled-init1 /etc/s6-rc/compiled
sudo groupadd -g 981 s6log
sudo useradd -m -d /var/lib/s6log -s /bin/nologin -g s6log -u 981 s6log
#Maybe remove chrony and cronie services, if they're not installed and the chrony user doesn't exist, as it'll cause activation errors on the tty, although the base services should work. Reminder to remove cronie and chrony from the bundle dir too, as the s6-rc database compilation will fail.
sudo xbps-install -S chrony cronie
sudo cp /etc/s6-linux-init/current/bin/* /sbin/
# if everything works, yolo to:
sudo /sbin/reboot-void