OBSERVABILITY / BY BREE
Monitoring a homelab without alerting on everything
Dedicated monitoring, inventory-driven targets, and knowing when an offline machine is perfectly normal.
A workstation being switched off should not mean the homelab is having an incident. That sounds obvious, but it is a useful starting point for deciding what monitoring should actually tell me.
Pathfinder includes machines with different jobs and different expectations. Some support services that should stay available. Others are used for work, games and experiments, then turned off. Treating every missing scrape as equally urgent would hide that distinction.
This note describes the Monitoring v1 design recorded in my September 2026 platform notes.
Give monitoring its own home
Monitoring runs on a dedicated container on the always-on virtualisation host. It does not live on my workstation, which I expect to reboot, switch off and experiment with.
The flow is straightforward: node exporters expose host metrics; Prometheus collects them; Grafana displays them; alert rules feed Alertmanager, which sends notifications to Discord.
Moving monitoring off the workstation removes one unnecessary dependency. It does not make monitoring independent of the virtualisation host beneath it. That remains a failure boundary to understand, rather than a reason to describe the setup as highly available.
Keep the target list with the inventory
The Ansible inventory supplies the monitoring targets. Prometheus uses generated file-based discovery, so there is no separate list of machines to maintain by hand.
Host-specific exceptions stay with the host in inventory. That gives me one place to explain both what a machine is and how it should be treated.
Metrics travel over the local network. Remote management and offsite connectivity have a different role; there was no need to make them a dependency of the normal metrics path.
Offline does not always mean broken
The inventory includes an expected_up distinction between infrastructure that should remain available and machines that can legitimately be powered off.
The point is to make alerts reflect how the platform is used. A missing workstation and a missing infrastructure node can look similar to a metrics collector while meaning very different things to me.
I still want visibility into both. I want the interruption to match the situation.
Test the notification, not just the configuration
The Monitoring v1 notes record an end-to-end test of both firing and recovery notifications. That matters: a valid configuration alone does not show that the message reaches its destination, or that I hear when the condition clears.
Prometheus history also has its own persistent volume rather than existing only inside the monitoring container’s root filesystem. Keeping the monitoring service and its history distinct makes that dependency easier to reason about.
Stop at a useful first version
The first version covers host metrics, fleet dashboards, persistent history and basic alert delivery. Container metrics, Proxmox API metrics, centralised logs and deeper application checks are separate extensions.
There is plenty more I could add. For now, the useful lesson is to give monitoring a clear job, test that it does that job, and make its alerts match the platform I actually run.