Marelooke's mess

Discovered another drawback of custom modbus sensors in Home Assistant: they don’t retain state through reboots.

Which is a bit annoying since some sensors only provide an absolute value. For example the inverter provides the total amount of energy produced over its lifetime (or at least until someone manually resets it), so to get the daily value Home Assistant needs to keep track of what the value was at the end of the previous day. This cut-off point does not survive reboots, leading to … interesting consumption/production graphs… But at least the derived numbers are still correct.

So looked into pushing that data into Prometheus instead, so I can plot things using Grafana. There’s a few modbus projects out there for Prometheus, but most suffer from the inability to deal with multiple slaves. Thankfully Telegraf appears to have modbus support, unfortunately the (imho anyway) nicer request-based configuration syntax hasn’t made it into any release yet.

Debating whether I should just run a dev version, wait for the new syntax to make it into a release, or suck it up and use the old register-based format. Rather leaning towards just using the dev version right now since Telegraf doesn’t actually store any data, so switching back to a stable train later shouldn’t be breaking.

“Solved” it by just pulling the data straight from the SMA panel inverters instead of from the Victron that handles the batteries.

Now that we have solar, and days are longer, I’ve also picked up messing with some of the more power hungry systems again. Picked up the backup project with the old dual Xeon system that was sorta on the backburner waiting for TrueNAS Scale to mature. Unfortunately they locked down apt, which makes it kinda unusable for my purposes since I run my storage network on Infiniband.

Back to the drawing board. Since Unbuntu Server supports ZFS out of the box I started looking in that direction, unfortunatley it doesn’t support installing Root on ZFS yet without jumping through hoops. I’ll see how much the bloat annoys me (will have to nuke Snaps off of it again, for starters) but for now I’ll just let it do its lvm/ext4 thing on the root partition.

It’s not unlikely I’ll nuke the Ubuntu Server install and replace it with a more sensible OS, probably either Debian or FreeBSD.

1 Like

I never tried Proxmox Enterprise Backup myself, but may be something to look into specifically for backups. It has integration with Proxmox VE, but even without it, it’s still a Debian base with root on ZFS support and you can likely customize it a bit.

I think it may be easier to strip the Proxmox stuff from it and just use the base OS, instead of trying to get Debian installed on ZFS. Maybe. I don’t really like recommending things I never tried myself though, so I can’t say how good or bad it might be.

My main issue with Proxmox is the rate of updates is pretty high on the CE, and the licensing is prohibitively expensive for homelab use as it is per socket. Don’t think PBS has a license, but still, former point still applies :slight_smile:

I had a go at installing Debian following the ZOL guide to get root-on-ZFS, unfortunately this requires one to use the livecd and that one didn’t want to boot into a GUI. Which wouldn’t have been a problem (don’t need the GUI…) but I also couldn’t get into a console. Not sure what is causing that last issue, if they’re using systemd on the live images that would explain it because they disable that functionality for “security” reasons iirc.

So just did a normal Debian install, installation took a fraction of the time Ubuntu’s did (if that’s any indication of bloat then Ubuntu server is bloated as).

Getting back to my desire to Ansible-ify things I’ve started setting up stuff through Ansible immediately, should be making some more progress when using an OS I’m more familiar with rather than my OpenBSD attempt that kinda ended up on the backburner.

The first hurdle is configuring the Infiniband interface, as I don’t want to install NetworkManager to just make that slightly easier. Not sure if there’s a way to identify an interface of a specific type that has a “link up” state (definitely can’t rely on MAC for IB interfaces), or whether I should just hardcode the name (which should be fairly consistent nowadays). Maybe I’m just overthinking it though…

2 Likes

Are you using networkd or the old ifupdown to control network?

Probably still ifupdown, looks like that is still the default, at least all my other Debian VMs are still using /etc/network/interfaces.

2 Likes

It is. networkd is there in the base install though (I assume because systemd all or nothing anti-modularity). I disable ifupdown and use networkd on Debian when using Ansible. The config is pretty easy to manage with templates and I can reuse networkd Ansible tasks across Debian, Ubuntu and Arch. I suppose I could get rid of Network Manager on RHEL hosts too but a lot of things in that ecosystem expect Network Manager so that would be problematic.

1 Like

What I meant was to install PBS and then remove all the Proxmox packages from it, including the Proxmox repos, so you would only get Debian updates. It should be doable and it would land you with a bare basic Debian install.

Besides the updates from Proxmox for their own utils, all the other updates are coming from Debian repos as far as I can remember.

Well, it is. Ubuntu is running a bunch of stuff in the background and if we compare their server image to Debian, it is a lot bigger. It is weird though how Ubuntu has everything it needs in a 1.2 GB iso, while Debian DVD has 3.6 GB, but I believe the DVD has all the desktop environments and other stuff included. The netinstall is only 400 MB in size, but obviously it still downloads stuff from the repo. I can’t do a fresh install size comparison right now, I don’t have any working x86 PCs I could test on, lmao.

For Proxmox the kernel is custom, not sure about PBS, if it just uses Debian’s then this might actually be a viable route…

Just did a quick test install in a VM, all defaults except I enabled OpenSSH in both cases (and disabled GUI in Debian’s).

Debian 11 Bullseye: 1.1GiB (everything in one partition)
Ubuntu Server 22.04 LTS: 4.6GiB (with an additional 125MiB in /boot)

I now noticed Ubuntu 22.04 has a “minimised” installation option (I was using 20.04 before). Quick Google wasn’t able to figure out what it’s supposed to contain, only that it’s supposed to bring down installation to less than 100MiB, which seems suspiciously small for a modern, functional system but I might have a poke at it and see what it actually contains.

1 Like

The root partition of a “Minimised” install was still 4.2GiB, so…errrr, really not sure what it’s supposed to contain now, as it only shaved of 400MiB…

1 Like

HomeAssistant decided to go absolutely berserk tonight with their fallback DNS spam. In less than an hour it managed to make ~10m (yes million) requests to Cloudflare IPs (1.1.1.1 etc.) that got blocked by pfBlockerNG.

Not sure what all happened that it suddenly stepped up its game like this but the consequences were fun. This spam filled up /var with logs despite there being log size limits set for both the firewall as well as pfBlockerNG, presumably the system wasn’t able to roll the logs over fast enough to cope. Once /var ran full Unbound just fell over…yay…

Thankfully the HA team finally provided an option to disable their beyond broken fallback DNS implementation: ha dns options --fallback=false so I can at least keep using their container instead of having to rebuild the setup in docker.

A reboot to clear /var (which is a tempfs partition) later and we were (mostly) back in business. “Mostly” because one of the solar inverters hadn’t taken the whole thing well and was no longer reachable on its IP (it might have just fallen back to a default IP due to the router being unavailable). After forcing it to get an IP things went finally back to normal.

1 Like

Didn’t really find how to do it with networkd, so I ended up with this hacky solution:

    - name: ensure Infiniband packages are installed
      apt:
        pkg:
         - rdma-core
         - infiniband-diags
        state: latest
    - name: Create Infiniband interface configuration
      copy:
        dest: "/etc/network/interfaces.d/ibp8s0.cfg"
        content: |
          auto ibp8s0
          iface ibp8s0 inet static
                  address 192.168.1.153/24
                  bridge_stp off
                  mtu 65520
                  pre-up echo connected > /sys/class/infiniband/ibp8s0/device/net/ibp8s0/mode
    - name: Bring up the Infiniband interface
      ansible.builtin.shell: ifup ibp8s0

It’s kinda ugly, especially, since it requires knowing the interface name…

There is a package to manage Debian interface files but it apparently can’t create new interfaces, only manage existing ones.

1 Like

Not sure about infiniband specifically, but in general networkd interface config is split between these 3 files:

https://freedesktop.org/software/systemd/man/systemd.netdev.html
https://freedesktop.org/software/systemd/man/systemd.link.html
https://freedesktop.org/software/systemd/man/systemd.network.html

You probably only need link and network files though, not netdev unless you’re using vlans.

Here’s an example of my .network template:

[Match]

{% if iface['hw_addr'] is defined %}
PermanentMACAddress={{ iface['hw_addr'] }}
{% else %}
Name={{ iface['dev'] }}
{% endif %}

[Link]

{% if iface['mtu'] is defined %}
MTUBytes={{ iface['mtu'] }}
{% endif %}
ActivationPolicy=up

[Network]

{% if iface['description'] is defined %}
Description={{ iface['description'] }}
{% endif %}
LinkLocalAddressing=no
{% if iface['dhcp4'] %}
DHCP=ipv4
{% elif iface['ip4'] is defined and iface['prefix4'] is defined %}
Address={{ iface['ip4'] }}/{{ iface['prefix4'] }}
{% else %}
DHCP=false
{% endif %}
{% for vlan in iface['vlans'] | default([]) %}
VLAN=vlan{{ site_subnets[vlan]['subnet_id'] }}
{% endfor %}
{% if iface['dhcp4'] %}

[DHCPv4]

SendHostname=false
{% if iface['rand_mac'] %}
ClientIdentifier=duid
IAID={{ iaid_int_var }}
{% endif %}

{% endif %}
{% if iface['routes'] is defined %}
{% for dest in iface['routes'].keys() | default([]) %}
[Route]
Destination={{ dest }}
Gateway={{ iface['routes'][dest] }}

{% endfor %}
{% endif %}

Note that some of the config didn’t exist yet in the version shipped with Debian so you might see some warnings about unknown settings (ActivationPolicy if I remember correctly).