Faun makes a build and repo server

Void Linux is soon dropping support for 32 and 64bit BE PowerPC Architectures. I want to take that over completely and have either a G5 and a G4 run a few hours a day building packages, or have a build server set up with a bigger machine (R620 needs something to do for example lol). I want the machines to have runit jobs that have them look at xbps-src, see what packages aren’t built that are buildable, build them, then check for any updates every day at a certain time. I then want the output to go to one specific machine that will act as a repo machine and will have access to the outside world. This would also act as the repo for my OS I want to make.

I have most of the pieces here, but I am a little stuck on a few parts:

Would this be better to have crosscompiled or is native better? Thinking about power consumption*
Should I use HTTPS or FTP? Does it matter?
What would I use to bit-by-bit copy built packages to said repo server? RSYNC? Should I use something like Git?
If I used Gitlab on a native install would that make management easier or would that be needless hassle?
Could I run PPC32 and PPC64 Void in a VM on an X86 machine?
Are there web tools I could run to see like HDD smart status and things like that?
If the repo server is on an old processor arch, will that affect download speeds? (478 presott P4 2.4ghz 800FSB)

I am working with these machines:
Mac G5 - Dual 970FX @ 2.0ghz, 2GB Ram, 320GB HDD - build
Mac G4 - Dual 7445 @ 1.2GHzw 2GB Ram, 80GB and 120GB HDD - build
Dell Optiplex SX 270 - Single Pentium 4 @ 2.4ghz, 2mb cache, 800mhz fsb, 2GB Ram - Repo
All running Void Linux

I want packages to be built specifically for macs. Later when I have a Talos 2 I can worry about bigger fish, but for now I want to take up the supporting role for the machines I want to keep alive. I have genuinely seen excited users on Facebook talk about having their favorite laptops back thanks to void, and some people legit only have their G5 they bought a billion years ago and are looking for the best options. If I can have kernels up that help those machines too, thatd be amazing. Then I don’t even have to fuck with all the dumb stuff online its all local.

3 Likes

Power Consumption note: I have servers that I want to set up that I haven’t figured out. If it would be better to cross compile and I could have one machine going instead of a whole bunch, cool.

The only reason I am using the machines listed is because I have been having troubles getting my servers working

Ok I have something of a framework for right now. I won’t do cross compile immediately as I will need to read a lot of stuff to make that happen, but when I switch to that it’ll be done on my R620.

For now, I will gun a G5 dualie and a G4 dualie for building their respective packages. I’ll run a base install so it’ll be easy to run on the G4, and so I can basically just copy paste. I will have them dump to the pentium 4 machine for now, later to the R510 as the main repo and the SX270 as the alt repo for when the main one is down. I don’t have to worry about DL speeds that way, its just a fallback.

Then I can SSH into them when I need to.

Later on I will build a strictly packaging distro that will help just set up all the shit I will be setting up, then anyone can do this easily. I already have a name for it, VXB.

just gotta do the math on power consumption, but I’m gunna start on this tomorrow. I’m beat.

1 Like

I have a workspace set up to work on this today. I will probably have questions

1 Like

Just dropping in to offer this small tidbit for consideration - cross compiling is the way to go:

… yes, I know MIPS aren’t everything, and G5 isn’t a G4, and packages aren’t going to somehow magically compile almost a 1000x faster, … but, this contrast is something to consider.

… folks I know who maintain these kinds of “tinderboxes” run either headless machines with local caches in ram and all storage remote, or just keep all packages locally on raid0 nvme, and use a lot of bind mounts.

3 Likes

If you look at my post I basically just want to set two machines loose and have them dump packages into a nas and ftp server and update them. I’ll need to see about file compression for the ftp server though as it can only take 500gb max. Or rolling availability.

But yeah thats basically what I’m doing. I just need ssh and processors. I want to know if nativecompile would have a byte compatability advantage. I want these old machines to boom.

1 Like

Technically native compilation should be better, unless your hardware is so old that it takes days to compile a bigger software like libreoffice and firefox. You will have to test both cross compiling from a modern x86 and native compilation preferably on the G5.

Just use https, that’s what alpha.repo.voidlinux.org uses. Also, ftp is not secure.

Technically you should be using git pull --rebase upstream master once in a while on the void-packages github repo, to grab the latest xbps-src templates.

There’s nothing wrong with gitlab or gitea, but IMO there’s not much need for that, git by itself should suffice.

I don’t know about that, but you don’t even need to run a VM, just have qemu installed and use the cross-compile flags for xbps-src (xbps-src -a $target-architecture). There are the cross-compile options

I didn’t even know we have support to cross-compile mips, that’s pretty nice.

Not sure about that, other than alerting software. Take a peak at voidlinux grafana dashboard
https://grafana.s.voidlinux.org/d/xfpJB9FGz/node-exporter?orgId=1&from=now-7d&to=now
They are using node_exporter, so behind grafana there’s a prometheus server, like I am running.

So long as it is running ppc and ppc64 (not LE, as I understand you want big endian only), it should run on them. I personally prefer musl, but whatever you choose to support, you can’t run too many compilations on those old machines.

For the repo, I suggest you mount the hostdir/binpkg to a NFS location to another box, so that you can send the packages on your network to another box, so you don’t waste the local space of the build server.

I highly encourage you to reach to q66 on Libera chat and maybe take over the maintenance of ppc, ppc64, ppc-musl and ppc64-musl builds (those are all BE) in his infrastructure, if he allows you to. You could benefit from running a better server (using his), the voidlinux-ppc project doesn’t get split and you don’t need to host a repo for yourself, you just have to maintain the packages. Just my $0.002.

2 Likes

In regards to cross/native compilation, there a a bunch of packes that simply cannot be cross compiled.
If you have a decently beefy machine, I would recommend using a cross-architecture chroot. Less overhead than a full VM. Worked quite well for me when I was doing that stuff. The 32bit stuff is kind of my baby :slightly_smiling_face: Compared to native, single threaded stuff like configure scripts are dog slow but multi-thread compiles are nice and fast.

Binaries shouldn’t really be any different, same compiler, same target arch, same code.

If you need access to a fast G5, I could give you ssh access to my G5 quad 2.5GHz.

@q66 are you stll here somewhere?

2 Likes

I mean if you’re gunna nit pick

Yeah see the convenient part is its physically in front of me and if something pisses me off I can just get a cross over cable and a laptop. Can’t really do that if the machine ain’t here lol

Well see I plan in building my own distro for fun at some point. I see your point, however I think itd be better to just make repos available and more focus on the original goal of combining projects

1 Like

That’s a good point, I’m hoping most are niche things not many people would care about much, is there any distro out there that keeps track of this cross-compilability(?) in the package sources?

What would be the notable examples that pop to mind?

Yeah Void tracks it, there’s a variable nocross in the build scripts that marks whether that package can be cross compiled.

Quckly looking at the void ports tree with a grep -R nocross, some packages that stick out include emacs, libreoffice, webkit2gtk, mplayer, texlive, handbrake and a bunch of programming language compilers (presumably due to compilers compiling other compilers all the way down. Standard bootstrapping issues).

Nothing absolutely crucial but a few nice-to-haves.

1 Like

God dang it, that’s why LO isn’t available in the aarch64 repo and I always have to compile the latest version myself (main PC = RPi 4). Now this tempts me to buy an ARM server or something and donate some of it to native compilation. I wonder how easy would it be to make a distributed computing compilation tool, like say, have a master node that checks on the workers to see which one is free to compile stuff, and have some dumb preference scales, like say, prefer a certain server over another.

I believe his last comment was almost 2 years ago. He also didn’t respond to a mail or 2 I sent him. I don’t want to harass the guy though, lol. But I haven’t checked Libera chat, I’m pretty sure he should be over on the Void Linux IRC and if not, someone there, like Piraty or Duncaen can contact of him.

Meh hes kind of a jerk. Also why I don’t want to work with him lol

He probably has a lot of shit going on where I don’t lol

Hmmmm, I wonder if it isn’t perhaps a better use of personal time to setup GitHub Actions somehow to trigger package building in the correct order (linearize reverse deps of a package that changed recently into a commit), thus you end up using their compute to maintain a distro.

I don’t have much experience with GitHub Actions but there’s these “strategy.matrix” thing which theoretically let’s you build using 256VMs in parallel on every commit.

Apparently you get 2 fairly modern (<5yo xeon) cores/7G ram/14G ssd per VM. It’s not much, I imagine it’d be good enough for majority of packages.

I mean it’s free, could it hurt to try?

Maybe even run qemu inside of an action in some cases?

(I assume if GitHub ends up not liking this approach for whatever reason, not all work would be wasted ; and some might be interesting for your own build farm).

Dude I’m not a developer half of that is moonbat gibberish to me

Sure as long as it isn’t a massive pain in my dick

I’m sorry, I was being overly terse, here’s the LCA video where the guy uses words like “unlimited free compute” as long your repo is public.
Basically, you can have GitHub (Microsoft), start some VMs to do whatever you want them to on every commit.

He also mentions qemu.


So, what you’d do, is make a repo, and make script that looks at changes to package tree, git log basically. And for every change to any one of the packages, it computes what needs to rebuilt and in what order by following the dependencies.

Looking at xbps repos, tmux depends on libevent-devel ncurses-devel. That means you should probably rebuild tmux every time those two change. libevent-devel depends on openssl-devel.

So, when you get a new version of openssl you need 3 commits. First, build openssl. Second build anything that depends on openssl incl. libevent, third you build anything that depends on anything that depends on openssl incl. tmux

Now, how do you know what is built and what needs building, for starters, you make an http server at home to receive built packages.

Then you invert the dependency tree from a version of xbps, start with all the packages that you have installed on your systems.

For each package, look at makedepends and for each line of make depends write a line in a file <dep> <pkg> and just sort that file in the end. Eventually, you’ll see something like libevent in xbps git log, and you can reference this kind of inverted tree (more like a list of edges in a tree) to know what needs rebuilding at first.

Take note of those packages, and check them for interdependencies. If you have something that depends on something else, you can’t rebuild it now, have to do it later.

So, you make a git commit with stuff you can rebuild now, and wait for binary archives to come back.

Once that’s back, mark them somewhere so that you know archives you have received are current for the xbps commit you’re working on. You’ll be able to send the remainder of the list in a next commit to a GitHub action repo (just remove any cross dependencies), and so on.

It does imply that when GCC changes, you’ll need to rebuild everything,… this is what distros do pretty much, it’s a huge deal takes a couple of days usually, and they always look at every commit skeptically.

Good news is, as long as your doing it for some packages like only those you use on your system or maybe some common set of popular stuff, it’s no different than e.g. running Gentoo - as in it’s probably doable.

Also, if GitHub says “this naughty project has 256 VMs running all the time and a weird set of commits only for triggering actions, let’s shut it down”, much of this is not GitHub specific. You can have your own git pull builders do the work at home or wherever.

The curse is in that when things break in build for whatever reason, you need to figure out how to deal with stuff because nobody else will.

2 Likes

I’m now curious of renting POWER VM’s at OVH or something… But only if I did patreon or something.

One reason I am doing it the way I am doing it is because I am A) not like a great dev or anything, B) can only afford so much, and C) will eventually have a Talos system. So one way or another I’d like to make it component-level simple and even just spin up an ISO specifically for making a build server.

Interesting about only needing so much space tho. Especially if I can do it over SSD and stuff that SATA 1.5 bus like crazy, I could get some oomph if I am careful.

As well I am still working on firmware things. So I would like to measure performance in between shit when I start messing with clocks and memory blob sizes in OpenBoot.

I will look into using it for kernel builds as I really want to tighten down what a machine needs to be capable of doing and strip everything else out. Builds only, no other things running at all. Is it possible to build stuff with parallel cores? Run multiple machines in beowulf?

Would there be benefit to a local appliance for this? I only have so many gigabit connections and I want to use them for wifi points and shit. Servers etc. A lot of my network is 10/100 because flat ass its what I can afford, though I have 2 routers I use as 4 port switches and both my rack switches have sfp connectors. I’d be worried about uplink throughput to an outside server. While a G5 isn’t exactly fast, as few of them together or a quad is.

Right. So what I want to happen is every time a package gets a change from main I want to have the system build it, and anything it attaches to. What I’m worried about is rebuilding a package that would be pointless to update at the moment and just wait for more packages to pop up and actually have a feature added, rather than just something no one will notice or know was changed. Save some power. I guess… ok theres one way to use actions then… I’ll think about that more. I need some cash to get some NIC’s if I am doing that.

I’m writing a set of scripts for this shit its too tedious for no reason.

Ok but would I have issues with wasted compute time? I don’t want to rebuild something pointlessly.

This is where I will have my R620 step in with what it can. If theres anything in a slump I can have my R510 dump to a web builder somewhere and come back. Ok. Seems simple…

Yeah that kinda knocks out the possibility of using git. Not that I want other people to do it for me neccessarily, I’m just not up for spaghetti.

Some things sound useful, other things sound like a linux user did some aspie shit. I’ll look into this today and see if theres anything on coder radio about GHA’s.

Actually this might be too deep for me to dive i to right now but I will keep a botepad handy today.

1 Like

Linux packages in general, especially libraries, don’t really follow any sane semantic versioning rules ; many do, but more don’t.

For example, if you knew that e.g. openssl or java bumping the version number by one doesn’t require all dependants to be rebuilt, that’d be great… but you can’t know this for sure in general.

… this whole computer science idea of reproducible and cacheable builds and action graphs is relatively new to Linux packaging people.


BTW - succeeding in maintaining this would be epic.

(Oh and GitHub actions can run qemu, I thought I mentioned it).

2 Likes