I know I’m jumping the gun here a bit, but things take a while to compile so I want to get a head start.
Goals
- A fully automated way to generate custom stage4 tarballs based on Gentoo
- A fully automated way to generate custom liveCDs based on the custom stage4 tarballs
- A simple website to host tarballs and ISOs
Stretch Goals
- Figure out some sort of logrotate-esque way to keep past snapshots
- Keep past 3 snapshots? Past 10? minutia…
- Custom installer that installs Gentoo using custom stage4 tarball to a hard drive when ran from liveCD
- Preferably using golang and GTK since I’m interested in the technologies
Plan:
Phase 1
- Get Gentoo’s Catayst working in a VM, in a non-automated way
- Get Catalyst configured the way I want my custom distro to be (OpenRC, SELinux, Hardened profile, etc.)
- Upload “finished” spec files and tarballs to a public repo and link them <here>
Phase 2
- Convert the VM setup into a series of container images
- Create simple webserver to host files from my personal domain
- Create series of Openshift Cron Jobs that rebuild the entire chain
- Portage snapshots → stage1 → stage2 → stage3 → stage4 → liveCDStage1 → liveCDStage2
- Cronjob will need to dump all the tarballs/isos into shared storage for the web server
Notes
Catalyst and Portage seem like a really flexible way to create a custom distro, I mean it’s what Gentoo’s Release Engineering team use for Gentoo after all. The problem, as is always the case with opensource, is that the RelEng team documented somewhat how they use Catalyst, but didn’t seem to cover all the bases. There seems to be more information when running man catalyst, man catalyst-config, and man catalyst-spec, but there are still some pitfalls that aren’t called out (like how specifying a kernel config in your stage1.spec will fail at the very last step, without any indication as to why, AFTER rebuilding GCC 3 times… in a VM… hence why I’m getting cracking on this early).
I am kinda going overboard with LTO, SELinux, and other fun features, but that’s fine. I don’t expect anybody to use my exact configuration, rather I am trying to explore Catalyst by doing, rather than by reading. Hopefully my explorations will help other people learn about this tool.
I may or may not post my personal website on this when I’m done, but even if I don’t I will definitely post all my build scripts and try to post my build artifacts to a public git repo.
Also, you definitely don’t have to use OpenShift/k8s cronjobs for this. You could very easily adapt this to use gitlab’s CI or GitHub Actions, but I like tinkering with OpenShift so that’s the road I’m taking.
I will keep updating this as I make notable headway.
3 Likes
Tree
One of the first things I had to wrap my head around was the various files and directory structure needed for this. I’ll try and provide my thoughts on that here:
/
├── etc
| ├── catalyst
| | ├── catalyst.conf // Generic catalyst config.
| | ├── catalystrc // Config to inject into `make.conf` while building
├── var
| ├── tmp
| | ├── catalyst
| | | ├── builds // Holds final tarballs/ISOs
| | | ├── kerncache // Mine is empty so???
| | | ├── packages // Holds packages build from one stage to be used on the next stage
| | | ├── portage // Normal portage layout including `package.use` and `package.env`
| | | ├── snapshots // holds portage snapshots
| | | ├── specs // holds specfiles which specify how to build each stage
| | | ├── tmp // Where building is done. This can be a tmpfs to speed things up
Again, this could be wrong, and some of these depend on what’s in your spec file, so take these as a general guide, not a hard and fast rule.
Specs
I guess I should cover spec files as they seem to be the real meat and potatoes of this
Stage1
/var/tmp/catalyst/specs/stage1.spec
subarch: amd64
target: stage1
version_stamp: hardened-selinux-openrc-latest # the final tarball will be named "stage1-<version_stamp>.xz. It's useful to add dates instead of using 'latest' here
rel_type: hardened # I think the only release types are hardened and default. Hardened is more secure but takes longer to compile and may be a slower resulting binary
snapshot: latest # Before you begin building, you need a snapshot of the current portage tree by running `catalyst -s <snapshot>`. This tells catalyst what snapshot you are using. Again, it's useful to use dates instead of latest
source_subpath: hardened/stage3-amd64-hardened-selinux-openrc-latest-upstream # name of tarball you are starting from. This should be a regular Gentoo tarball. Name is relative to `build` directory
compression_mode: pixz_x # How should compress the tarballs? pixz_x is a parallel xz
portage_confdir: /var/tmp/catalyst/portage/stage1 # location of `package.use`, etc. files. Optional
update_seed: yes # Rebuild the source tarball before building stage1?
update_seed_command: --emptytree --ask n @world # what command to run to update_seed. Defaults to --update --changed-used --deep @world
portage_prefix: reavessm # IDK what this does but I put my name here?
common_flags: "-O3 -pipe -flto=4 -fipa-pta ..." # Common CFLAGS, CXXFLAGS, etc. Optional
cflag: # Optional
cxxflag: # Optional
Stage 2
Very similar to stage 1, minus update_seed
/var/tmp/catalyst/specs/stage2.spec
subarch: amd64
target: stage2
version_stamp: hardened-selinux-openrc-latest
rel_type: hardened
snapshot: latest
source_subpath: hardened/stage1-amd64-hardened-selinux-openrc-latest
compression_mode: pixz_x
portage_confdir: /var/tmp/catalyst/portage/stage2
# NO UPDATE SEED FROM HERE ON OUT
portage_prefix: reavessm
portage_overlay: /var/db/repos/lto-overlay # Optional additional overlays to add to build
portage_overlay: /var/db/repos/mv # Extra optional additional overlays to add to build
common_flags: "-O3 -pipe -flto=4 -fipa-pta ..."
Stage 3
Basically the identical config to stage2, but incrementing the necessary fields.
I did have to change a fair bit of package.env files, which I don’t understand, but meh
/var/tmp/catalyst/portage/stage3/package.env
# NOTE: I split these up per package, but I'm joining them for this blog
sys-apps/checkpolicy no-funny-business.conf
sys-libs/glibc no-funny-business.conf
sys-devel/gcc no-lto.conf
dev-util/gperf no-lto.conf
app-arch/gzip no-lto.conf
dev-util/pkgconf no-tlo.conf
sys-apps/which no-lto.conf
sys-libs/zlib no-lto.conf
/var/tmp/catalyst/portage/stage3/env/no-funny-business.conf
# Completely overwrite CFLAGS
CFLAGS="-O3 -pipe"
CXXFLAGS="${CFLAGS}"
FFLAGS="${CFLAGS}"
FCFLAGS="${CFLAGS}"
/var/tmp/catalyst/portage/stage3/env/no-lto.conf
# Only overwrite LTO flags
CFLAGS="${CFLAGS} -fno-lto"
CXXFLAGS="${CXXFLAGS} -fno-lto"
I will update the later specs when I have the building
2 Likes
Can you necro your own post? Well, idk. But I finally got this at least building. Now I just need to think about how to finish the installer and how to host the artifacts.
If you’re interested, you can follow the progress here
1 Like