Monitoring literally everything with Prometheus & Grafana (and Other Misc. Things)

AI Disclosure:
I used Claude to walk me through some troubleshooting steps I otherwise didn’t expect. For instance, Tailscale does not initialize normally for Prometheus and Grafana in the Unraid UI, so Tailscale had to be created and linked as a separate Docker container. Or, when the Miktotik switch would not work I asked Claude to help me with a custom user for scraping. I also periodically sanity-checked with Claude to make sure I was on the right track to avoid hours of frustration. Otherwise, I wanted to understand well how everything links together.


I’ve wanted better all-in-one monitoring for my Unraid server, plex server, pihole, OPNsense box, dad’s pc, mikrotik switch, etc. for a while now since it’s just cool to have, but never got around to it since it would be a sizable chunk of work to figure out something I’ve never used before… Then I realized what better time than Devember?? :partying_face: naturally, all of these platofrms have great monitoring systems I can individually go to, but we’re making the one-stop-shop for whatever I want to know.

Not done. WIP. Will update my list periodically. Once said and done with things talking correctly, I will customize my dashboards and probably upload the custom board for people to download. I will also be adding more and more pictures over time for other visual learners like me.

Legend

:check_box_with_check: done and working in grafana.
:white_question_mark: made some progress, but not done/scraping properly yet.
:cross_mark: not started.

Progress

Status Machine OS Exporter Port
:check_box_with_check: big-boss (NAS/*arr Suite) Unraid node_exporter OR alloy?, smartctl_exporter, cAdvisor 9100/514, 8088, 9633
:check_box_with_check: little-boss (plex server/live transcoding, tdarr/cpu archive transcoding) Ubuntu node_exporter OR alloy? 9100/514
:cross_mark: little-boss nvidia dash Ubuntu node_exporter? ?
:white_question_mark: plex Ubuntu (Docker) plex-exporter 32400
:white_question_mark: tautilli Ubuntu (Docker) tautulli-exporter 9595
:check_box_with_check: tdarr Ubtuntu (Docker) tdarr-exporter 9596
:white_question_mark: amber-pc-win11 Windows 11 windows_exporter OR alloy? 9182/514
:cross_mark: dads-pc-win11 Windows 11 windows_exporter 9182
:cross_mark: raspberrypi-system Linux node_exporter OR alloy? 9100/514
:check_box_with_check: raspberrypi-pihole Linux pihole-exporter 9617
:cross_mark: opnsense FreeBSD node_exporter plugin 9100
:check_box_with_check: mikrotik 2.5G switch RouterOS mktxp 9115
:cross_mark: xbox series x (1) n/a ping 9115
:cross_mark: xbox series x (2) n/a ping 9115
:check_box_with_check: cyberpower ups n/a apcupsd 9162

Things to look into

  • alloy – I want to get prometheus, exporters, and grfana all working and then I’ll go back and check out alloy in place of node_exporter and windows_exporter.
  • loki – want to do, will do after exporters. Logging would be very helpful.
  • uptime kuma
  • power cycle persistence
  • gcx tool

prometheus setup

Unraid UI → APPS

ts-prometheus container

You need a 1-time use tailscale auth key from your admin console:

link prometheus to container

permissions issue

Unraid assigns Prometheus a dummy user ID of 99:100 by default, but it doesn’t have proper permissions to access the appdata folder. This has to be fixed so it can initialize.

docker stop prometheus
chown -R 65534:65534 /mnt/user/appdata/prometheus/data

Then we just delete --user from Extra Parameters of the container.

basic prometheus.yml for us to configure slowly

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets: ['LOCAL-IP:9100']   # Unraid
        labels:
          host: unraid

  - job_name: cadvisor
    static_configs:
      - targets: ['LOCAL-IP:8080']   # per-container stats

grafana setup

Unraid UI → APPS

default permissions:
admin | admin

add data source (prometheus)

Load 1860 (Node Exporter)

#$ empty grafana, let’s configure!

prometheus is itself a tailnet node, so it can scrape the 100.x addresses directly. They don’t change, which suits Windows PCs on DHCP. Then, I used the LAN address for little-boss because it’s on the same network.

scraping

scrape localhost data

Unraid → APPs → Node Exporter Plugin Installs

  - job_name: node
    static_configs:
      - targets: ['TAILSCALE-IP:9100']
        labels: {host: big-boss}
      - targets: ['TAILSCALE-IP:9100']
        labels: {host: little-boss}
      - targets: ['TAILSCALE-IP:9100']
        labels: {host: raspberrypi}
      - targets: ['TAILSCALE-IP:9100']
        labels: {host: opnsense}

  - job_name: windows
    static_configs:
      - targets: ['TAILSCALE-IP-IP:9182']
        labels: {host: amber-pc-win11}

  - job_name: cadvisor
    static_configs:
      - targets: ['TAILSCALE-IP:8088']
        labels: {host: big-boss}

that’s great and all, but the devices are not scraping and sending data yet

UPS

Thankfully, my UPS makes this very easy with a plug-and-play USB-A (Unraid) → USB-B 2.0 (UPS) cable!

  - job_name: ups
    static_configs:
      - targets: ['TAILSCALE-IP:9162']
        labels: {host: big-boss-ups}

Settings → UPS Settings
Start APC UPS Deamon: Yes
Apply.

However, to see it in Prometheus, we need another container that’s not in Uraid’s community apps:

docker run -d --name=apcupsd-exporter --network=host --restart unless-stopped okaeri/apcupsd_exporter

ID: 21119

smartctl

id: 22604

This has already been helpful as I apparently have an m.2 dying… lol

plex server pc (ubuntu server) plex, tdarr, tautilli

sudo apt install prometheus-node-exporter

Should pickup auto-magically in the prometheus health overview.

sudo nano docker-compose-exporters.yml

Install exporters

name: exporters

services:
  plex-exporter:
    image: cplieger/plex-exporter:latest
    container_name: plex-exporter
    restart: unless-stopped
    network_mode: host
    environment:
      PLEX_URL: "http://LAN-IP:32400"
      PLEX_TOKEN: "YOUR_PLEX_TOKEN"

  tautulli-exporter:
    image: mm404/tautulli-exporter:latest
    container_name: tautulli-exporter
    restart: unless-stopped
    network_mode: host
    environment:
      TAUTULLI_URL: "http://localhost:8181"
      TAUTULLI_API_KEY: "YOUR_TAUTULLI_API_KEY"
      METRICS_PORT: "9595"

  tdarr-exporter:
    image: homeylab/tdarr-exporter:latest
    container_name: tdarr-exporter
    restart: unless-stopped
    network_mode: host
    environment:
      TDARR_URL: "http://localhost:8265"
      PROMETHEUS_PORT: "9596"

start containers

$ sudo docker compose -f docker-compose-exporters.yml up -d
curl -s http://localhost:9594/metrics | head -5
curl -s http://localhost:9595/metrics | head -5
curl -s http://localhost:9596/metrics | head -5
[+] up 33/33
 ✔ Image mm404/tautulli-exporter:latest Pulled                                                                      1.6s
 ✔ Image homeylab/tdarr-exporter:latest Pulled                                                                      2.7s
 ✔ Image cplieger/plex-exporter:latest  Pulled                                                                      3.9s
 ✔ Container tdarr-exporter             Started                                                                     0.3s
 ✔ Container plex-exporter              Started                                                                     0.4s
 ✔ Container tautulli-exporter          Started                                                                     0.5s
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0

add scrap jobs

  - job_name: plex
    static_configs:
      - targets: ['TAILSCALE-IP:9594']
        labels: {host: little-boss}

  - job_name: tautulli
    scrape_interval: 30s
    static_configs:
      - targets: ['TAILSCALE-IP:9595']
        labels: {host: little-boss}

  - job_name: tdarr
    scrape_interval: 30s
    static_configs:
      - targets: ['TAILSCALE-IP:9596']
        labels: {host: little-boss}

add windows 11 pcs

Had to get Claude help because I had no clue what to do lol

PS C:\Users\Admin> # 1. Must print True. If it prints False, reopen PowerShell as administrator.

PS C:\Users\Admin> ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
True

download the latest release

PS C:\Users\Admin> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
PS C:\Users\Admin> $rel = Invoke-RestMethod https://api.github.com/repos/prometheus-community/windows_exporter/releases/latest
PS C:\Users\Admin> $asset = $rel.assets | Where-Object { $_.name -like "*amd64.msi" } | Select-Object -First 1
PS C:\Users\Admin> $msi = Join-Path $env:TEMP $asset.name
PS C:\Users\Admin> Invoke-WebRequest $asset.browser_download_url -OutFile $msi
PS C:\Users\Admin> Get-Item $msi


    Directory: C:\Users\Admin\AppData\Local\Temp


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         11/8/2026   3:32 PM       14643200 windows_exporter-0.32.1-amd64.msi


install and check

PS C:\Users\Admin> Start-Process msiexec -Wait -ArgumentList "/i `"$msi`" LISTEN_PORT=9182 REMOTE_ADDR=100.100.217.91 /qn /l*v `"$env:TEMP\we-install.log`""
PS C:\Users\Admin> Get-Service windows_exporter

Status   Name               DisplayName
------   ----               -----------
Running  windows_exporter   windows_exporter


PS C:\Users\Admin> (Invoke-WebRequest -UseBasicParsing http://localhost:9182/metrics).StatusCode
200

creates a PC selector in node exporter

mikrotik switch

https://forum.mikrotik.com/t/guide-grafana-

I needed to configure the Mikrotik for LAN access. I have been using the switch for a while, but realized it was still thinking I was using the default “192.168.X.X” default IP config. My home LAN in “10.1.1.1”, so although it was functioning for its purpose, I actually had no LAN access.

IP → DHCP Server → Networks

  • Address: 10.1.1.0/24
  • Gateway: 10.1.1.1
  • DNS Servers: 10.1.1.155

IP → Routes

  • Dst. Address: 0.0.0.0/24
  • Gateway: 10.1.1.1
  • Immediate Gateway: 10.1.1.1%bridge

new read-only user to scrape data

[admin@MikroTik] > /user group add name=mktxp_group policy=api,read
[admin@MikroTik] > /user add name=mktxp_user group=mktxp_group password=CHOOSE-A-STRONG-PASSWORD
[admin@MikroTik] > /ip service print
Flags: X - DISABLED, I - INVALID
Columns: NAME, PORT, CERTIFICATE, VRF
#   NAME     PORT  CERTIFICATE  VRF 
0   telnet     23               main
1   ftp        21                   
2   www        80               main
3   ssh        22               main
4 X www-ssl   443  none         main
5   api      8728               main
6   winbox   8291               main
7   api-ssl  8729  none         main
[admin@MikroTik] > /ip service set api disabled=no address=10.1.1.183/32

add to docker-compose-exporters.yml

name: exporters

services:
  plex-exporter:
    ...
  tautulli-exporter:
    ...
  tdarr-exporter:
    ...
  mktxp:
    image: ghcr.io/akpw/mktxp:latest
    container_name: mktxp
    restart: unless-stopped
    network_mode: host
    security_opt:
      - apparmor=unconfined
    configs:
      - source: mktxp_conf
        target: /home/mktxp/mktxp/mktxp.conf

configs:
  mktxp_conf:
    content: |
      [CRS310]
          enabled = True
          hostname = 10.1.1.2
          port = 8728
          username = mktxp_user
          password = ${MKTXP_PASSWORD}
          use_ssl = False

          interface = True
          monitor = True
          dhcp = False
          dhcp_lease = False
          pool = False
          firewall = False
          route = False
          wireless = False
          wireless_clients = False
          capsman = False
          capsman_clients = False

ID: 13679

pihole

Make sure docker is installed and working

sudo apt update && sudo apt upgrade -y
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
docker --version
docker run hello-world
sudo systemctl enable docker

set up docker compose file and pihole exporter

sudo nano docker-compose.yml

services:
  pihole-exporter:
    image: ekofr/pihole-exporter:latest
    environment:
      PIHOLE_HOSTNAME: HOSTNAME
      PIHOLE_PASSWORD: "$APP_PASS"
    volumes:
      - ./secrets/pihole_password.txt:/run/secrets/pihole_password:ro
    ports:
      - "9617:9617"
    healthcheck:
      test: ["CMD", "./pihole-exporter", "-healthcheck"]

find and add your pihole web app password

make the .env file

sudo nano .env
APP_PASS=***********

add scrape job:

    - job_name: "pihole"
      static_configs:
          - targets: ["TAILSCALE-IP:9617"]

ID:10176

alloy setup

grafana app

I like it. Can easily connect and see from my phone with tailscale. Necessary? No lol

From the three dots in the corner choose to add shortcut. Then it will let you install grafana PWS-style.



nice

FileBrowser Quantum

I wanted to experiment with this so I could basically google drive-style share links to video or photos safely from my server.

Featuring: Tailscale Funnel!

Highlights

  • No Port Forwarding: You do not need to open inbound ports on your home router or firewall.

  • Double Encryption: Traffic entering the public Funnel is encrypted via HTTPS/TLS, and the data is tunneled over directly to the local node.

  • Automatic Certificates: Tailscale automatically manages trusted Let’s Encrypt TLS certificates for the ts.net subdomain.

  • Restricted Ports: Funnel only listens on specific, hardcoded public ports (such as 443, 8443, or 10000), refusing exposure on other ports.

setup

todo – add pics

Using filebrowser to share links

funnel enabled

go to the tailscale address and page settings

install pws app

on Unraid, upload files to the public folder

files immediately appear on filebrowser that was uploaded to unraid

right click file you want to share

configure share settings - password, duration, etc.

copy clipboard link

link opened in phone with tailscale app disconnected

video preview

3 Likes

I might be stealing parts of this one for my own setup.

:yay:

1 Like

Feel free ofc, and lmk if I can clarify anything lol

I was word-vomiting info in here last night as I was coming to it. Needs reorganized and to have proper chapter markers.

I’ve worked 6 years in cloud observability so I know a few tricks.

The first one is to use alloy instead of prometheus node exporters. It is also from Grafana but its more modern and configuration is much more sane than node exporter. It also features many things out of the box such as smart monitoring.

Also I recommend using Loki for log collection. Collecting logs is useful because you can render based on queries and also set alarms for error messages. Alloy can send logs to Loki.

Don’t sleep on alerts. Grafana’s alertmanager is quite good.

Alternatively victoriametrics and victorialogs are lightweight alternatives to Prometheus and Loki respectively. These are also compatible with alloy.

1 Like

Ditto.

I like the To Do documentation within a table.

2 Likes

NOOOOOOOOO

lol I love this setup and project, watching your progress!

1 Like

Thank you for this info! I’ll check it out for sure

1 Like

YESSSSS :sob:

Slowly chugging along

If you run into broken panels or the dashboards don’t graph what you want the gcx tool and your favorite LLM really help take the tedium out of the maintenance and updating.

1 Like

This is a great Devember project. One thing I’d add to the plan is a check that lives somewhere other than the box doing the monitoring. I run Uptime Kuma on a separate Google Cloud free-tier VM so I can see whether things are up from inside the tailnet and from outside it, because when the main server goes down it takes its own dashboards with it. Since you’re running Tailscale as its own container for Prometheus and Grafana, I’d also test what scraping looks like after a reboot, since containers don’t always rejoin the tailnet cleanly.

1 Like

Thank you!

I’ve added this info to my to-do list. i.e. persistence testing and also to look into uptime kuma

I will give this a look! I haven’t heard of gcx, but that seems helpful.

Correct me if I’m wrong, but this would only replace the node_exporter and windows_exporter? ← would be helpful, I want to replace them.

From my understanding when looking at alloy, the mikrotik switch, ups, etc. would all need their dedicated exporters for prometheus instead?

It does. You can install via winget on Windows. For replacing node_exporter on Linux look for the Unix component in Components | Grafana Alloy documentation

Instructions for Windows: Install Grafana Alloy on Windows | Grafana Alloy documentation

If you can’t run alloy directly on the device you can have it scrape the data and send to prometheus or victoriametrics.

1 Like

Great thank you

I’ve updated my reply about the Mikrotik monitoring.

1 Like

played around with FileBrowser Quantum today. Very cool container! Added a section to the post. Basically, share media easily, safely, and without port forwarding thanks to tailscale funnel.

1 Like