ThatGuyB's rants

[ThatGuyB]

@ThatGuyB

need to move the vm data out of the system.

Just did, runs on my Think Penguin NAS / hypervisor now. But my UPS shows 143W total power consumption (PC, monitors, two switches and the tpnas), which is very spicy. Shutting down the big monitor shows 100W sucked up. Stopping the 2.5G switch, 85W. So the tpnas uses about 70W with proxmox (used to only draw about 40 at idle with void). Stopping the VM shaves off 10W.

Thankfully this is not going to be up 24x7.

@ThatGuyB

I’ll probably burn virt-manager to the ground. It’s been doing weird stuff the past couple of hours (like vm getting stuck at the uefi boot screen after grub, no matter what init I set).

PEBKAC. Well, kinda. Yesterday I updated the VM once I “finished” (there’s a lot more to do on my s6 VM) and got a kernel update. But the initramfs didn’t get regenerated. In my infinite wisdom, I uninstalled bash (why should I keep it if I use ksh anyway?) and dracut failed to run (although the package manager didn’t report dracut / initramfs errors - or maybe I wasn’t paying attention, I was tired).

In a chroot, tried to reinstall the kernel via the package manager, I definitely didn’t see any error there. But the initramfs wasn’t there. Running dracut manually… “file not found.” which dracut showed the file, so that couldn’t have been it. The kernel (for which the version I was trying to make the ramfs) was present (triple checked). I looked at dracut (it’s just a shell script) and I notice the shebang… #!/bin/bash -p.

I had to troubleshoot my tpnas, as I didn’t assign some vlans to it in the switch and the network config on it was also bad (it used to work in proxmox 7, idk what happened in 8?... freaking proxmox… well, now I can just assign a vlan per interface, I don’t need to connect it to a bridge connected to a vlan int to the main bridge - man, am I glad proxmox is only my actual lab and not my homeprod box).

So, that leads me to virt-manager. The VM was kernel panicking because it couldn’t find "unknown-block(0,0). That’s because of the missing initramfs. But proxmox with its novnc console showed me the early error message after grub booted the entry, but virt-manager was getting hung at the TianoCore banner and didn’t show me the error the OS was giving. Proxmox is using the same OVMF / TianoCore UEFI virtual firmware for VMs, but the banner doesn’t show up.

So, idk if this is a bug in the spice server virt-manager is using, or the OVMF configuration shipped by the distro.

And yes, this was my mistake, I own it, but I wish the tools I’m using wouldn’t get in my way and would show me what the issue is. If I was booting bare-metal, I’d likely have seen the kernel panic immediately.

---

I’ve converted chronyd and cronie-crond to execline longrun services. You can still use the runit sv run files written in shell, even without modifying most of them. You’d need to change the ones that contain runit specific code, e.g. things that call upon sv check to verify if another service is up before executing the main daemon. But those are rare. But since I’m converting to s6, might as well stick to what “upstream recommends.”

I was surprised that most tutorials and implementations of s6 I found online combine s6-linux-init and s6-rc folders under /etc/s6. That’s not a hard thing to do, but requires that when you call s6-linux-init-maker to update the init (basically almost never) to specify the parameters to replace the default behavior / location where the program looks.

s6-linux-init: the s6-linux-init-maker program

Default is /etc/s6-linux-init/current. When you generate a new init, you can specify a new location and symlink “current” to that location (I do it in the same folder, with a different name). I think artix-s6 was using /etc/s6/current.

I think so was the old config duncaen wrote for void ages ago (I didn’t realize I was looking at a git commit from 2019!). Well, because of that, a lot of work that I needed to do was avoided, but I still had to troubleshoot execline scripts that were using antiquated utilities (s6 is an evolving project and stuff like s6-linux-utils and s6-portable-utils are always getting updated). The tool s6-test was deprecated in some version and Laurent Bercot recommends you use eltest from execline instead.

Well, I didn’t have the time to check eltest for compatibility and the parameters / flags it accepts, so I just used test (removing the “s6-” prefix basically).

I’m still working on understanding some s6 concepts, in particular the dependencies. IDK what’s going on, but the shutdown sequence looks wrong.

For example, mount-rw is a oneshot that just runs “mount -o rw,remount /” and has no “down” service file, so it doesn’t really matter that it gets stopped early on. But if it was, then services like dhcpcd-eth0-log would fail miserably, because the rootfs wouldn’t be mounted anymore (again, if the script would be doing unmounting, which is not recommended, s6 takes care of unmounting any fs at the shutdown sequence).

That means dependencies might not be properly respected. I have given dependencies on bundles, expecting that services in a bundle would all inherit the dependency, but now that I look at this, I don’t think that’s how it works. That’s what testing’s for! But boy, will it be a ROYAL PITA to configure dependencies manually for all services. I was sold on the whole premise of bundles and was expecting that I can just name a bundle as a dependency and expect everything to wait for it.

Looking through the config, it seems like it’s possible I missed to add some dependencies in the configs.

Ugh, I need to debug all dependency files… that’s what testing’s for, that’s what testing’s for, that’s what testing’s for, that’s what testing’s for…

[ThatGuyB]

I just want to take a moment to appreciate how bad artix-s6 dependency validation is out of the box. This is a system where I didn’t do anything on, I literally installed it according to the artix-s6 wiki and checked how some of their services are written / configured.

OOTB, just the s6 minimal iso… even the networking is not getting up. The DB has the dhcpcd-srv entry, but whatever artix-s6 folks are doing, can’t be anything good.

Note that the folder above is the s6 run directory, where all the services are supervised. If the empty file “down” is present in one of the folders, that means s6-rc decided that the service shouldn’t be up. Either one of the dependencies are not met, or the service is not included in whatever bundle is configured to start the service (to put it in systemd terms, is not “systemctl enabled,” or in runit terms and other rc, the service isn’t enabled to be started).

It looks like dhcpcd-srv is dependent on udev and dhcpcd-log, which makes sense (the logger always needs to be up when a service is started and the service always needs to stop before the logger), but the dhcpcd-log has no dependency.

That means, if it was enabled, dhcpd bundle / pipeline isn’t enabled. Which also makes sense, given the “down” file in the s6 service directory. If started manually, it works without fuss.

---

Idk… just me complaining, that’s what I do best. I still appreciate all the work the artix-s6 folks put into their distro.

Apparently, artix-s6 has a custom helper script, named s6-db-reload.
https://wiki.artixlinux.org/Main/S6

I haven’t checked it in-depth (need to look in the code), but I don’t think it’s really useful (maybe I’m bashing it too early). It says it’s to load a new DB and reload services, but you can do that easily in 2 commands. I guess it helps that it can clean up old DBs for you, which is nice, so if there’s something that makes a routine maintenance task easier, it has my seal of approval, I’m not going to judge it harshly.

In the past week I’ve been working on s6, I had a few s6-rc dbs that I kept around (in case I broke the latest one) and it kinda paid off actually (I had a corrupted db a few times and I fixed that by booting into runit and deleting the broken one and pointing the symlink to the old db and rebooting).

[ThatGuyB]

@ThatGuyB

That means dependencies might not be properly respected. I have given dependencies on bundles, expecting that services in a bundle would all inherit the dependency, but now that I look at this, I don’t think that’s how it works.

I’ve RTFM and bundles don’t have the dependencies directive. Also, the files called “contents” (for bundles) and “dependencies” (for atomic services) have been deprecated. They still work, but the recommended way now is to create the folders contents.d and dependencies.d respective, which in turn contain files with the name of a service (usually empty files, but AFAIK s6-rc-compile doesn’t check the file type or the contents).

An atomic service can have a bundle set as a dependency, but a bundle can’t have dependencies, which is a shame. I got confused, probably because the “flag-essential” directive can be applied to both bundles and atomic services and if it’s on bundles, it recursively applies to all atomic services contained in the bundle. It would make 100% sense to have a dependency on a bundle apply as a dependency for all atomic services in the bundle.

Note: the flag-essential file marks a service as non-stoppable by the s6-rc -d command (because they’re important) and they’ll only be stoppable by the s6-rc -D command. I think this is some internal logic in s6-rc. I don’t know the details yet, I need to return to writing the s6 tips wiki and expand the explanations (once I understand myself how it works).

I’m randomly guessing that when you update an s6-rc db with s6-rc-update (moving from one compiled-db to another), if a service is found in both versions of the db and has the “flag-essential” file, when inevitably s6-rc -d is ran on it, nothing will happen and the service will stay up among desired states transitions. But if the service (no matter if it’s essential or not), if it’s absent in the new compiled db, it’ll probably have s6-rc -D ran on it. At least that makes the most amount of sense from what I currently know.

---

So, in its current form, as of 2024-06-09, to get a working system using s6-rc, you must:

• make a bundle called “default” (it’s what s6-rc is invoked to start, by s6-linux-init*) which contains all other bundles, which in turn contain the services that you need activated
• split your services in stages (you could call them runlevels ), represented by bundles
• for each service add the lower stage bundle** as a dependency under dependencies.d folder***

* You can change the “default” value of $rl by using -D argument in s6-linux-init-maker, or by appending a numeric value to your kernel command line, which can be viewed in your bootloader conf (i.e. grub) and when booted by cat /proc/cmdline.

** The documentation of s6-rc-compile doesn’t mention it explicitly, but you can add a bundle under dependencies.d and s6-rc’s db will compile / translate the whole bundle as a dependency for the service (i.e. all dependencies must be met before launching the service).

*** You can’t put a dependency on the bundle, you must mention the dependency in the atomic service.

---

Other intricacies that I need to take note of for myself:

• the up and down scripts of a oneshot service are interpreted by execlineb and compiled and stored in the s6-rc database; if they’re an execline script, they don’t need a shebang
• seems to be recommended that if one needs a script, particularly shell, but can by python, perl or whatever suits one’s fancy, that the up or down script contains only a call to the script (i.e. for oneshot serviceX, up = /bin/sh -c /etc/rc.local.d/service.start and down = /bin/sh -c /etc/rc.local.d/service.stop)

---

In my chase for knowledge, I also learned runit has a “down” option for enabled services - instead of removing the symlink from /etc/runit/runsvdir/default, you just add an empty file named “down” to /etc/runit/runsvdir/default/service or to its pointer folder /etc/sv/service.

Having the symlink in the runsvdir folder makes runit “aware” that a service exists and tries to start it. Adding the “down” file makes runit aware of the service, but it never tries to start it automatically, kinda like systemctl disable service. That means you can always manually start a service on-demand (like maybe start it after another service is started), similar to systemctl start service even if the service is disabled.

Runit also has a “finish” script for services, where some “cleanup” can be made after a service / daemon dies / gets stopped, e.g. say you have service1 that has “run” file that does stuff while running and outputs it to /run/tmp/service1/output.txt and you need the output processed every time service1 is stopped or dies. Using “finish” you can, say, rsync the output.txt to somewhere, or process it further. After a while, runit will try starting the service1 again and the cycle repeats.

I don’t think it’s as powerful as s6, but it does have its uses. But with s6, you can make a transitional oneshot that has a dependency on service1 and if service1 is stopped, the down script of oneshot automatically runs to do whatever cleanup or post-processing after service1.

But idk how that’d work when service1 dies unexpectedly. Actually, I need to read on this more (or test it). If service2 has a dependency on service1 and service1 dies and quickly gets restarted by s6-rc, will service2 get restarted too, or will it keep running? That’ll make a huge difference depending on the scenario and requirements of a program one might run.

Tangential systemd ramble - it's not all doom and gloom, you can read it.

To make the systemd analogy with the above, in systemd there’s the “After,” “Requires” and “Wants” directives. The 1st is just ordering to start and not get killed, the 2nd is a requirement, but without a Before / After also mentioned, with just “Requires” systemd will start the services at the same time (which makes requires basically useless - well, not really, because systemd has weird states, like “starting” and a servie will basically be in “starting” until its Requires is “active”). And the 3rd is…?!

Well, it’s more like:

• After = do Y after X

• Requires =
  - don’t do Y if you can’t do X
  - if you did X and later X fails, don’t stop Y
  - if you stop X, also stop Y
  - recommended to be combined with After

• Wants =
  - weaker Requires;
  - I think maybe something like: do Y no matter if you can or can’t do X

• BindsTo =
  - stronger Requires
  - don’t do Y if you can’t do X
  - if you did X and later X gets stopped or killed, stop Y
  - even stronger with After

The weird part about Requires is that, both X and Y are started together, but if Y gets active and X never managed to gets into “active,” then Y gets stopped. It’s basically a race condition if you don’t also use After.

Man, reading systemd documentation gives me a headache. Why is it so hard to comprehend? Can’t someone dumb it down, or are these people just so elitists that they can’t write documentation for the masses?

Also, what’s with all those directives? There’s even more that I haven’t read, like PartOf and Requisite.

The “finish” script in runit might not be a 1:1 equivalent feature actually, as the runit finish will always save the exit code of service1 (be it 0, 1, 2, 3, 100, 111 or whatnot) and can be automated to do stuff based on that, but a dependent s6 service won’t know that information.

Well, that’s the point where you can literally run an s6-rc supervised runsvdir process, if you really need such a functionality. Best of both worlds! (but then, idk how runsvdir will react when it gets a kill command, if it’ll try to stop services and then run “finish” as usual, or if everything just dies - with the existence of “timeout-down” file in s6-rc db, I’d assume that s6 just waits for things to stop gracefully, absent that file and when the file exists, it just kill -9 after the timeout - at least that makes the most sense to this file).

That’s what I get for writing before finishing reading. Longruns have a “finish” file in s6-rc too.

And s6 also has a “max-dealth-tally” file (I guess it’s the number of times a service can die before it stops getting restarted, which might make some sense to the assumptions I had above about services getting restarted).

---

I just remembered I read that runit has custom control files on how to stop a service.
https://wiki.gentoo.org/wiki/Runit#Custom_process_control

All the s6-rc journey started when I had the need to not only start services one after another when starting a server (networking → iscsid → iscsi-login oneshot), but also safely shutdown a server (iscsi-logout → iscsid → networking).

I think it might’ve been possible to do something with /etc/sv/service1/control/t. But it’s not an explicit dependency, it’d be more of a hack-job. And at this point, I’m so far into s6 that I doubt I can go back to runit, oof.

[ThatGuyB]

@ThatGuyB

Apparently, artix-s6 has a custom helper script, named s6-db-reload.

I’ve found yet another artix-s6 custom thing. I wish they’d respect Laurent’s guidelines and not name their scripts with the s6* prefix. It gets confusing knowing if commands I find are distro-specific, part of another s6 package, or a deprecated command.

s6 - skarnet's small supervision suite

Why “s6” ?

skarnet.org’s small and secure supervision software suite.

Also, s6 is a nice command name prefix to have: it identifies the origin of the software, and it’s short. Expect more use of s6- in future skarnet.org software releases. And please avoid using that prefix for your own projects.

The thing I’ve found now is s6-service. I was trying to convert openssh run file from sh to execline and wanted to see if any work was already done on that.

IMO a distro shouldn’t just add every service you install to auto-start (that’s what ubuntu’s doing). I think it’s for usability reasons (making it easy for a user), but I think there’s certain things that would be better off left to the user’s control (I think fedora wasn’t enabling services on startup and left that to the user to enable, which is a good practice).

Also, artix-s6 folks just launch everything at startup (add to default bundle), with no regards to dependency.

Why would you start sshd before you even have an ip address? There’s a race condition waiting to happen, since the loopback interface gets started in parallel. It’s not a “big deal,” because if sshd starts before, it’ll just crash, then net-lo gets up, then s6-supervise restarts sshd and you’re off to the races.

But that’s no better than just infinitely restarting services until they manage to get started (a la plain supervision suites, i.e. daemontools or runit style), so why use a service-manager at all if you’re not going to bother to have dependency resolution?

s6-rc: why?

[ThatGuyB]

In my goal to understand s6 more, I found and I’m currently reading its inception stages, from almost 10 years ago!
https://www.mail-archive.com/[email protected]/msg00325.html

[ThatGuyB]

I just finished replicating the setup on a new, fresh VM. I also removed a bunch of the needless steps and went full retard s6 and don’t use any of the packaged services. If I need a service, I’m writing it in execline.

Speaking of which, everything in the s6 sources has been written in execline. There’s some oneshots that invoke “sh,” like rc.local (which you kinda need), but the idea is to not modify the rc-local oneshot service, but just to modify the /etc/rc.local script externally.

This was tested on Void, but I’ll probably launch an Alpine instance quickly to test if the setup is portable. From what I checked in execline documentation, it seems eltest might be good enough replacement for test command as it is (at least for how I’ve defined the service checks in the services). Which means I need to change the services to use eltest instead. I don’t know if ash test is as feature complete as the standard POSIX test (which eltest supports + more).

If it’s too different, I’ll write an Alpine separate guide to s6. I remember that Alpine was working towards s6 adoption, but was still using openrc and I don’t remember what else. I’ll check separately if Alpine has all the needed packages in the repos (at least world, if not main / base).

Unlike Void, where you just run make and make install outside of the package manager, Alpine might have everything managed in the repo. Not planning to use Alpine, but I’m keeping an open mind about it (I might use it in containers if some packages are more supported there).

I can’t wait to finish documenting everything and get back in business in my advancement in my homeprod. I’ve spent too much time on this side-projects (but I’d like to have a fallback in case I forget what I’m doing and maybe help others that are looking for a similar setup - there are at least a few people on the forum that might actually use this).

The next step after the documentation of the bare basic make stuff, I’ll look into making custom xbps-pkg for all the needed dependencies (and document it). The xbps repo versions are old (and s6-linux-init is completely missing) - not sure if they’re unmaintained, or if the folks at void are keeping the old versions for compatibility reasons, most of the people like the 66 stuff, but me and I think some main Void maintainers prefer to not touch 66. I’m not a void maintainer, so I can’t touch the void-packages repo, but maybe I’ll try my luck doing a pull request after upgrading the templates. If not, at least people can just clone my repo and xbps-srv the packages themselves.

[ThatGuyB]

@ThatGuyB

probably launch an Alpine instance quickly to test if the setup is portable

I ended up using alpine’s own repo versions (which are almost on-par, with just very few, like 2 or 3 packages, being just one minor behind the one in skarnet’s git), but I couldn’t make alpine easily bootable. I even had to install agetty (because busybox likely wouldn’t have worked).

I’ll have to add some debugging to the boot and see what’s breaking where. What I’m sure of is that it’s in the init stage 1, because I’m not even getting an early tty. If it was during s6-rc launch, my 00 oneshot would output that stage2 is beginning.

Anyway, alpine was more of a curiosity. I’ll get back to setting up void, release a bare basic script to migrate, a few services, update the forum’s wiki entry on s6 (I got a lot of improvements) and get on with my life. It’d be cool to use alpine too (because it’s a very minimal distro), but I’m not willing to put too much effort in understanding the inner workings of alpine (and I’ve used normal sys install, I don’t want to imagine what headaches a diskless install would’ve caused me).

[ThatGuyB]

Just finished installing Void with root-on-zfs and bootstrapped s6 on it. I also tested that my guide is all working. I’m surprised this thing worked out of the box, no modifications needed to my s6-rc services.

I need to test it with encrypted ZFS next and if that works, I am willing to slap it on my daily driver (after a backup). To be fair, if you have a copy of your original runit files (which my guide includes), you can always just add a kernel boot parameter (init=/sbin/init-void) and it’ll just work with runit, like before.

I’m surprised how little you actually need out of s6 to get it running, after all the services have been properly created. It’s wild. Aside from the comments in my doc, it’s just 54 commands to run (maybe a few more on another PC, like scp a tar file or likely in the future git cloning my service repo).

I’ll publish the initial release, which makes heavy use of git clone, make and make install from Skarnet, but later (or sooner rather than later), I’ll make xbps-src templates for all the latest Skarnet packages and test my luck to pull request to void-packages. I’ll probably get rejected, but my own repo will still be up and anyone can inspect the files (no binary blobs needed and I definitely don’t want a tar file option, you should be able to compile your own stuff - and compiling this takes probably less than 2 minutes combined for everything).

[ThatGuyB]

Have I complained before how much I hate proxmox? I probably did, people on the forum know I’m a hater and only begrudgingly recommend it to others (because there’s no real good, easy and libre alternatives and virt-manager seems to be a dying product too, just like a lot of red hat’s legacy).

You try to clone a VM from an existing ZFS snapshot, it tells you that a full clone is not supported and doesn’t give you the ability to make a linked clone (I don’t think that’s possible with the zfs backend anyway).

Here’s the snapshot. What’s so wrong about this?

So I took it in my own hands, literally zfs-send from vm-disk-164-0@ready and vm-disk-164-1@ready respectively, to new datasets, vm-disk-165-0 and vm-disk-165-1. Deleted the vm-disk-165-{0,1}@ready snapshots and was left with perfectly usable datasets.

Created an empty template, similar to 164, looked at /etc/pve/qemu-server/164.conf and added the efidisk0 and virtio0 to 165.conf, changed the boot order and was left with a perfectly working VM 165 in the screenshot (formatting and installing root on encrypted zfs).

Proxmox, how hard can this be? I literally did it in 7 commands, maybe 8 or 9 if you count clicking the snapshot button in the GUI.

---

Anyway, finished testing with encrypted zfs-on-root and s6 still works without any modification. Kinda expected, given what I’ve seen yesterday (the /boot/efi takes care of loading zfsbootmenu and ask you for the password, then everything is the same as usual).

Time to re-sharpen my xbps-src skills…

[ThatGuyB]

I’m satisfied with the current state of my s6-rc services source folder. Tomorrow I’ll probably publish it on github. I haven’t figured out (yet) how to make the fsck service not mistakenly interfere with other kinds of fs, like btrfs, zfs or xfs (among others).

Currently it’s doing the normal check, if /forcefsck exists, run fsck. I should check if root = ext4 or something, but that would imply a little more complicated script, which I don’t think it’s that much worth it. The way I have it currently set up, which might not be a good idea, is for mount-rw (the rootfs oneshot) to have a dependency on fsck, but if fsck fails to run, because it’s zfs or btrfs (if you mistakenly have that /forcefsck file), then your system might fail to boot until you remove the file.

I think I’ll drop the dependency in the final source. Technically, the fsck service has as its only dependency 00, meaning that fsck will be started way before things like udev even start up, let alone the mount-rw service. It’s technically a race-condition, but in reality mounting root should happen way later. But the thing is, if mount-rw doesn’t have a dependency on the fsck service, then it’s not going to wait until fsck is finished and attempt to remount root rw and fail. If set indefinitely, s6-rc will attempt to remount root until it’s blue in the face, which ain’t really a good idea. So I should probably handle some exceptions.

I’ll probably mention this is a readme or something. Looking at the runit core-services in void (with everything serialized, even when things make sense to be started in parallel, like loading file systems), it seems like if fsck exits with a code greater than 1, then the system goes into emergency_shell. IDK how to emulate that in a service just yet (I think I can use a finish script, depending on the exit code of the service, launch emergency_shell or continue to mounting the fs).

I haven’t thought of a git layout yet. My initial idea was to separate each distro’s services into a different branch, but I might end up making void the master branch for now, to make initial deployment easier (and mention it in a readme). I’m looking for suggestions on git branches layouts and help would also be appreciated on how to launch a tty from busybox, without busybox being the system’s init.

[ThatGuyB]

I had to do some final touches on the s6 services. They’re public on github under s6_services.git. I forgot git didn’t track empty folders, so I had s6-rc compile errors for ok-wan and ok-lan (because they had empty “contents.d” folders, because these bundles had no services and weren’t used by me).

In retrospect, I might end up combining all services under a single folder “s6sv” and have my opinionated way in symlinks.

Anyway, it’s late, probably tomorrow I’ll end up fixing the [WIP] wiki for the s6 deployment and add a small git clone command and then resume the script.

It’s kinda sad this forum has put in the effort to make wiki entries, but wiki discoverability is kinda bad (unless you click the wiki tag for more of the same kind).

[wendell]

@ThatGuyB

sad this forum has put in the effort to make wiki entries, but wiki discoverability is kinda bad

Working on a new website.

Been pantsed by broadcom and doing a lot of the same kinda scripts for vmware as clients up pm renwewal and mobing begrudgingly to proxmox

[ThatGuyB]

I’ve just finished doing the updates to s6 xbps-src templates and I’ll need to look into the rules of pushing them upstream. If not, you can build s6 yourself, but having to git clone 2.7 GB of packages and binaries just to build 26 packages amounting for about 1.3MB in size (in total!)… just saying it gives me goosebumps.

I’d appreciate if someone told me how to git clone and exclude certain paths, with the exception of a few others, I could automate the build of the xbps packages.

Next in line would be alpine services and the script to switch alpine to s6-linux-init and s6-rc, followed by devuan. Those might still take a bit, because I plan on working on my own projects that require s6, in my own home-prod.

[ThatGuyB]

Using my odroid n2+ (aarch64), I ran through the same conversion schema and now I’m running s6, without a fuss. The chronyd server took a while to sync, but it did work (I wish I had a logger enabled for it).

I’m now updating the s6_services with more services (iscsi and nfs), which I’ll need to push to git.

[ThatGuyB]

Today I’m ranting about “free-loading.”

The term is very loaded and I hate the way people use it to trigger an emotional response from others. The term “free-loading” is literally an argumentum ad passiones fallacy (appeal to emotion).

It’s used in so many different contexts and does have some validity in certain cases. The best example of it not being a fallacy is adult offspring still freeloading off of their parents, without contributing anything to the household. There is clearly a victim involved (the parents) that, absent their consent to allow the “freeloader(s)” to stay, are literally bleeding money / other resources, that they could otherwise use for other purposes.

I think there was a L1 Show non-sense section, where 2 Italian dudes in their 40s or 50s got evicted by their mom, who was something like 60 to 80 y.o. So that’s a totally valid use of the term.

In the context of software, especially when it comes to free software, the term is very loaded. When we think of something like Redis changing from a BSD license to a source-available one, they claimed the reason for that was “oh, those giant corporate freeloaders that didn’t give us a dime, yet made millions off of us.”

While I agree corporations should at least consider sponsoring a project, particularly if they are making money from something, I don’t think the terms of your own license allow you to complain about not receiving something in return. One successful story (although so many wrong things done on the receiving end) was NixOS with the Logicblox sponsorship, which ended when the latter got acquired by Infor (NixOS was paying tens of thousands of $ monthly in aws s3 costs!). There’s no reason other projects can’t have the same success as NixOS had.

So, back to freeloading. When you release your software as GPL, ISC, Apache and so on, you are giving away your software. It doesn’t have to be for free, you can ask money for your work. But what someone does with your work later, like redistributing for free or charging money for it is up to the new copy’s owner. He can decide to pay you if he’s making money, or he can just keep all the earnings to himself.

Don’t like it? Don’t use a libre license. There’s stuff like Aladdin Free Public License, which restricts the resell of the software, there’s probably some form of license that is only libre for non-commercial use, or even a license that asks that you’re entitled to a cut if the software’s being utilized in a commercial environment. There isn’t one? Make one. And you can even go as far as allowing it for commercial activities that are adjacent (like a steel industry using your software for accounting or something), but require payment if your software is used directly to make money (like hosting redis instances for users).

This gets into things like Redis calling Amazon a freeloader and Réd Haté calling Enterprise Linux forks freeloaders. By your own terms, you agreed that they aren’t required to pay you anything, or after you got your initial payment, they can do whatever.

To go even beyond just licensing (and common courtesy) arguments, there’s a very strong point to be made about whether an entity would’ve been using your software from the get-go, were it not libre. If redis or rhel was proprietary from their inceptions, they probably would’ve made tons of money anyway (there was a demand for them - and keep in mind rhel always was a paid-for free software anyway).

But if these were proprietary, would amazon have been using redis, or forks of EL using that as a base? The answer here is a very resounding no. Amazon would’ve probably invented some proprietary version of redis and charged money (like they do now) for it. If EL was proprietary, then these projects either wouldn’t have existed, or they would’ve been either independent distros, or based on other distros (idk, given the age of centos, probably on Novell Linux or something).

The same arguments that people have been making for píracy, can be used to to defend freeloading basically. Had there been no way to acquire the software for free, nor allowed paid-for redistribution (or workarounds, like offering you the software, but selling you the hardware it’s running on, like elastic compute), people wouldn’t have been using said software.

So you wouldn’t have made money on it otherwise anyway. Well, Pink Hat’s arguments are literally “we’re bleeding money because someone’s offering a gratis version of our paid product” - again, if people had to pay for EL and there was no EL-based alternative, they’d be using Debian, Ubuntu or OpenSUSE (not even Ubuntu Pro or SLES, because they wouldn’t be paying anyway).

It’s a shame free software makers (usually of large proportions, like redis or RH) make these emotional arguments, but it’s even worse when even free software advocates are spewing this non-sense (which is what triggered the writing of this rant).

The freeloader argument has been used so many times as excuses to do rug-pulls of previously free licensed software and people have been buying it. This is now a basics 101 trick in the corporations books. Make your software free, have a bunch of people contribute improvements, do a rug-pull and close your source or do an “open-core” model and make loads of money from the free labor of others. Or sometimes they use another excuse, “developers need to eat too.” If they want to be compensated for their work, they should charge for their work. They can charge money for free software too, but they don’t like the idea that someone can just release a no-charge version to the world. In that case, don’t write free software, go write SaaS platforms.

This is part of the reason why I’ve become very against corporate free software. There are many practical reasons for being against corporate software in general: they make up for lack of sane design and quality code, with massive amounts of code and resources (coders and hardware), they sometimes not even use their own creation (gitea doesn’t use gitea to develop it, they use github) and on the philosophical aspect, they don’t care about using people’s work as free labor and taking all the credits (if you care about that).

For me, the biggest reason I’m against corporate software, is because of their borderline sociopathic behaviors. They don’t care about you (sometimes rightfully so), but they claim some kind of moral high-ground against other corporations, because they’re providing libre software (not that rainbow hat can even be classified as that, their contracts make them de-facto proprietary). The second, less important one, would be in the quality of software provided (they suck).

[Marandil]

@ThatGuyB

But the thing is, if mount-rw doesn’t have a dependency on the fsck service, then it’s not going to wait until fsck is finished and attempt to remount root rw and fail.

This is precisely what After= directives are for in systemd. Is there no sane equivalent in s6-rc? Can’t you order them up with a bundle of something? Or does that incur hard dependency as well?

[ThatGuyB]

I had the init-fsck service figured out. It’s a one-time service anyway and it’s not supposed to be demonized. It’s doing a bunch of if-then-else’s, where it checks if /fastboot exists (and exits, or if not), then if /forcefsck exists (if not, it exits, otherwise it), then runs the fsck on most fs. If it couldn’t, it reports an error and prevents the next service from launching, otherwise everything moves on. It’s nothing fancy actually.

I didn’t know that all that fsck needs to detect it needs to run is the /forcefsck file. That’s how it’s done in different places too, that’s how it was done in s6-rc. The mount-rw service is dependent on fsck and that’s it. The oneshot service will run, do its thing and because of how s6 handles oneshots (they must exit 0 for dependent service to start), then it’s really easy to handle. Just make mount-rw dependent on init-fsck and you’re done. It will almost always exit 0 (except cases where something’s wrong with the file system, or if /forcefsck existed but something wrong happened that prevented the execution).

---

There’s no “After=” in s6-rc, precisely because ordering without dependency make no real sense. I believe I mentioned somewhere the example of web servers. If your site can work without a db service, just won’t have all features enabled, then it’s up to the admin to decide if the db service is important enough that the website shouldn’t start if the DB can’t start, or if the web server part shouldn’t have a hard dependency on the db service at all.

From a design perspective, after= is kind of a hack, which is why it isn’t implemented in s6-rc. If someone wants to implement such a function, it should be fairly easy in the service itself (check if the other service was started, i.e. if there’s no down file in the other service’s service directory, wait a little, then start your service). I don’t see how after= can be of use to anything. You either have a hard dependency on a service, or you don’t.

[Marandil]

@ThatGuyB

From a design perspective, after= is kind of a hack, which is why it isn’t implemented in s6-rc.

I don’t agree it’s a hack. It’s a way to model these kinds of non-dependency ordering requirements. You don’t need to run fsck before mounting rw, but if you want to run fsck it has to be done before mount.
The web server analogy is a poor one IMO, because it’s too simple and it doesn’t have these kinds of ordering requirements.

An example of when “After=” makes sense is a service that has both local and network-facing interface (say listens on Unix and TCP sockets). This service may use “After=network” to ensure it’s started after all interfaces are up and configured. However it doesn’t Require network to be running properly and so it should be able to run even if you run without the network target/service/whatever, or if the network target failed starting.

Does that make sense to you?

[ThatGuyB]

@Marandil

An example of when “After=” makes sense is a service that has both local and network-facing interface (say listens on Unix and TCP sockets).

In technical terms, I see that example as a viable option. However, in practical terms, I don’t see any reason why you’d want to use a program with networking capabilities under normal circumstances, without having network enabled.

Postgres comes to mind immediately. You can connect on either unix socket or tcp socket. If your software stack only uses pg locally and nothing accesses the DB from the network, just don’t do a dependency on networking, only connect on local socket. If you do need PG to be accessed on the LAN for normal operations, you do a hard dependency on the networking service.

Your software stack can either be local / confined, or be a network resource, but I don’t see any scenario where you’d want to define a “nice to have, but not mandatory” dependency. And again, this can be handled by the service itself, if really needed and doesn’t need to be implemented in the service manager.

This is the execline version I just came up with: if -n { eltest -f /run/service/networking/down } { pg_ctl -D /path/to/pg/db start }. It means “if the down file doesn’t exist for networking (denoting it was started, but doesn’t check if the networking service is up or not), start postgres.”

The “-n” is negating the test and implicitly in execline if the “if” fails, it’ll exit with code 128 + the exit code of the command inside the if block (eltest’s exit code). So if eltest exists 1, the if will exit 129 and the pg service won’t be started.

s6-svscan then reattempts to start the service, once it failed. So the admin must ensure networking is enabled (in systemd terms), not that it’s running / up, to start your service (I believe this is true for systemd too, if a service is not enabled, then you have to start it manually for others to kick off after them, right? please correct me if I’m wrong, I didn’t read into this behavior). In the s6-rc case, this will just go on for the defined number of retries (which can be infinite) until the networking service is started (again, the networking itself can be in a failed state, but the service “networking” is started).

s6-rc works this way: if a service has a dependency (which in s6 is always a hard dependency) and it’s set to start up (either at boot or manually), it will always first start its parent dependencies and if these cannot start, the service will never start (if you have a service that can only be launched if you have networking running, then you shouldn’t try to start it without networking).

---

Back to my question on systemd, where you can correct me, to add a bit more detail. If you have a custom service “rsync” that is enabled and has a dependency (either After= or Requires= or both) on sshd, but sshd is disabled, will systemd try to start “sshd” service because the “rsync” service is enabled, or will the service stay down indefinitely because sshd is not enabled?

[ThatGuyB]

For the past week I had some really weird startup race conditions in my wireguard service in runit. I haven’t edited the service at all, it’s a default configuration.

So, about 3 or 4 times, after booting up, I had to stop the wg service and troubleshoot what’s happening. Simply restarting the service didn’t cut it. Eventually, I found out that wireguard was starting so fast and trying to prevent all connections outside the tunnel, that the date wasn’t properly set up by the chronyd service, by the time wg was already up.

The reason stopping the service, poking things around and it suddenly starting to work, was because chronyd had the time to check the date and update it. Sometimes chronyd started and updated fast enough that allowed wireguard to connect to its peer. Most of my SBCs don’t have an RTC battery (although I think all except the odroid hc4 have support for RTC), so a NTPD needs to run on boot.

This further accentuates the need for proper service dependency in my lab. For now, I’ve edited the wg runit service to exit if dhcpcd and ntpd aren’t running. This doesn’t necessarily mean things will work out. Just because the chronyd service is up doesn’t mean the date is properly set up.

---

As a side note, I didn’t think I’d be looking at s6-networking, but I just found another one of the skarnet unix gems: the TAICLOCK protocol (written by DJB) and the s6-taiclock program.

I used to have my own NTP server on pfsense long time ago, but that wasn’t really that great (it took quite some time for time to sync). I’m hoping taiclock will work better.

I think I can have a single device on the network (it’ll most likely end up being another odroid n2+, these things use peanuts for power) that’ll be the dedicated host of some important services, like taiclock for time syncing on the LAN and probably the main DNS (although I still want to implement a recursive and an authoritative dns, a la PLL’s guides and I’d rather do that in containers). It will probably not be powered on 24/7, but as long as it has a RTC battery, it’ll be the first thing that gets powered, followed by my router and switch, followed by everything else.