Ok. No real documentation, messy wiki’s (spread across 20 repos). 20 machines with different but same configuration.
Different but same = same motherboard, different versions, different hdd, different kernel, ubuntu 20.04, 5.x nice!
Not a lick of updates. Kernel 7? Forget it, we’re not even on 6.1. Updates? Security? Ha!
Pure bash for provisioning machines.. basically a clusterfuck in progress.
I’ll post my adventures in standardizing this fleet. And you haven’t seen anything yet..
Perhaps copy Jeff Geerling, he puts his exploits in Ansible on Github. Added bonus: rollback feature on various scripts 
This was us 5 years ago 
. Running RHEL 5, 6, 7, 8, and 9. Using rear recovery as an alt-PXE solution. It is a painful path, hopefully you have a team that will adopt some standardization and data hygiene. Git was the game changer for revisioning and preventing legacy rollback from a random script or playbook saved on a share in the wild. That being said, it is incredible when it starts coming together.
Already on it. I have a pretty extensive playbook setup. In the process of migrating from the old bash scripts to a more organized library.
(Actually used his stuff as a reference.)
I used ansible to manage almost all production systems and related stuff at my old job.
We had a DTAP street for pushing stuff to release, any changes got logged in git before rollout.
This was just before Redhat bought the project.
Only annoying part was you had to keep your playbooks updated as Redhat was on a programming crunch adding new stuff and deprecating every few weeks.
It’s been a few years I last used It, I hope the burden of maintaining reduced a bit.
I’m looking in to getting my RHCE updated so I guess I’ll figure out fist hand soon enough.
I haven’t hit any snags with deprecated features recently.
I am actually stuck on an older version but for our use-case it’s much much better.
Recently I have attempted to update a single node. By update I mean anything that doesn’t touch the kernel or usb, we are 6 years out of date. Our camera & focusers drivers are touchy (no DKMS - perhaps the next step is to introduce this ourselves..).
So far so good, I only caused 1 machine to go down for 24 hours - due to a self introduced bug. Basically I shipped /etc/systemd/network/50-wol.link matching Driver=igb with only WakeOnLan=magic and no NamePolicy.
This whole setup is relatively far away (5ish hours), but was not designed with complete remote control in mind (no vPro/BMC). So if the bios reset for some reason (ongoing issue), I have to ask someone onsite, or drive there to fix it.
A KVM would probably solve this issue, but I am still looking into cheaper solutions, perhaps one of those remote power on/off solutions.
I see you have your hands quite full working towards a propper solution.
Still nicely done if only one system up until now had a oopsy.
If your current playbooks are functioning I think it’s a good choice to prioritize getting the servers compliant first.
You can always modernize them later when more pressing matters are resolved.
Not to venture to far off topic but I have used multiple ipkvm solutions.
After some headaches with vPro I got some to test as I was a way more reliable solution.
I had a PI-kvm v3 pcie sized card (Raspberry PI 3 and 4 based), sipeed pico kvm and multiple gl Inet kvm models (Both have cheaper RISC/arm cpu’s).
They are all priced around 100-150 ish euro’s (Except the PI-kvm with current price hikes) and have full power control support build in or a add-on board. (Usually by pigtailing the case front panel headers)
Also vpn support is excelent on all of them.
I do suggest you setup some kind of overlay vpn like tail/headscale, or zerotier as those are better supported on most ipkvm’s I know of and way more flexible in setup.
Only downside on the GL Inet’s no opensource firmware and they can call home to the manufacture (It’s a legitimate feature to help with remote access and help with webRTC support) , so you might want to be a bit careful with those in a corporate setup.
He literally wrote the book on ansible!
Right, we have some corporate VPN, do you reckon tailscale/headscale is still required? Perhaps as a second layer against a bad actor within the org vpn.
We have a budget issue so currently I’m experimenting with some “remote power boards” that are much cheaper ~10$-5$/each. Currently with no success, if I can’t get the provisioning to work on them I’m probably going to have to flash them with OpenBK7231T or something similar. Seems these boards don’t play well with local wifi networks.
I will try to connect them to an AP that’s connected to a hotspot most likely. I have a feeling the our org wifi network blocks certain packets from being transferred (just a feeling).
Otherwise I will definitely check those pice cards you recommended.
If you have a site-to-site (router to router) vpn it’s redundant to run tail/headscale.
If it’s a server-client (sslvpn, client must be running on the device) you can probably get it to work on the PI and general ARM based kvm’s but no guarantees on the other boards as some run a custom RISC CPU and might just not have a native client available.
As for the powerboards, a lot of corporate wifi ap’s can block or limit broadcast and multicast packets. Some off those boards ( especially Tuya ) use multicast for sending commands.
OpenBK support mqtt which can use udp, but probably no auto discovery as it’s multicast based again.
It’s probably smart to have a talk (if you haven’t already) with the guy who is responsible for security policy in the org as those boards are a security officers worst nightmare.
Also if your org uses 802.11x (username and password login for wifi) it might be the power boards just have no support for the wifi security or you org might require a signed certificate file only available for devices who have joined the domain.
But a network admin should be able to tell you more.
Also it’s probably best to have a separate IoT network for those devices just to be safe.