Project Pathfinder: Making the Platform Observable — and Giving It a New Home

The last few weeks have been a fairly big turning point for Project Pathfinder.

A lot has changed underneath the surface: the infrastructure has been reorganised, responsibilities have moved between systems, the documentation has had a fairly brutal reality check, and Pathfinder now has a proper monitoring platform watching over it.

At the same time, LetsGeek itself was increasingly showing an older version of what I was working on.

So it was time to fix both.

Monitoring Pathfinder properly

One of the things I wanted from Pathfinder from the beginning was proper observability.

Not just “can I SSH into the server?” or “the application seems to be working”, but somewhere I can actually see the state of the platform and be told when something goes wrong.

The first version of that monitoring platform is now operational.

Monitoring lives on a dedicated Debian 13 LXC called Collective, hosted on my always-on Proxmox server, Empoknor.

Collective currently runs:

Prometheus for metrics collection, Grafana for visualisation, and Alertmanager for handling alerts and notifications.

Each managed Pathfinder node runs node_exporter, installed and configured through Ansible.

Prometheus discovers those nodes from the Pathfinder Ansible inventory rather than maintaining another manually edited list of machines.

That distinction matters to me.

One of the recurring ideas behind Pathfinder is that information should have an obvious owner. If Ansible already knows which systems exist, I don’t want to quietly build another configuration database inside Prometheus.

At the Monitoring v1 checkpoint recorded on 28 September 2026, Pathfinder had 13 monitored nodes and 14 healthy Prometheus targets — the 13 node exporters plus Prometheus itself.

All fourteen were up.

Monitoring things that actually matter

The first alert rules deliberately aren’t especially clever.

They cover the things I would genuinely want to know about:

• a node expected to be online disappearing;
• filesystems getting low on space;
• critically low filesystem space;
• sustained high CPU usage;
• sustained memory pressure.

There’s also an important little distinction in the configuration between a machine that is down and a machine that is unexpectedly down.

USS-Enterprise, my main CachyOS workstation, is monitored when it’s running, but it isn’t an always-on infrastructure dependency. I should be able to shut it down without Pathfinder deciding the Federation has collapsed.

So hosts can carry an expected_up property, and the NodeDown alert only fires for systems that are actually expected to remain available.

That is a tiny piece of configuration, but it represents one of the architecture lessons Pathfinder keeps reinforcing:

monitor the intended behaviour of a system, not merely the fact that it exists.

Proving the alert path

I didn’t want “Alertmanager is running” to count as finished.

The whole path needed testing.

So I deliberately stopped node_exporter on one of the lab systems.

Prometheus detected the failure, the NodeDown rule entered its firing state, Alertmanager processed it, and a notification appeared in Discord.

Then I restored the exporter.

The alert resolved and Discord received the recovery notification.

That gave me a proper end-to-end test:

Node
  ↓
node_exporter
  ↓
Prometheus
  ↓
Alert Rule
  ↓
Alertmanager
  ↓
Discord

Only after that did I consider Monitoring v1 operational.

There’s plenty more I want to add — container metrics, Proxmox visibility, PBS health, dashboard-as-code and possibly a self-hosted ntfy endpoint for important alerts — but none of those are required for the first version to be genuinely useful.

Pathfinder now watches itself.

That’s a pretty satisfying milestone.

Meanwhile, LetsGeek had become historical documentation

While all of that was happening, there was another problem.

This website was badly out of date.

The previous version of LetsGeek was built around projects I was concentrating on at the time, particularly Hobby Match and Aligned.

Those projects are still part of LetsGeek, but opening the site no longer gave a particularly accurate answer to:

What is Bree actually building?

The answer now is very heavily Project Pathfinder.

Linux, automation, infrastructure, monitoring, backup and recovery, programming, embedded systems, electronics and whatever interesting rabbit hole I find next.

The old site wasn’t bad.

It just described a different point in time.

And after spending a lot of time cleaning up Pathfinder’s documentation with the principle of document reality, not aspiration, leaving my public site describing old reality felt slightly ridiculous.

LetsGeek v3

Rather than continuing to modify the old site, I decided to start again with a clean Astro project.

The new LetsGeek uses static generation, Markdown-friendly content and a lightweight deployment model, with the structure and design rebuilt around what the site needs to be now.

Visually, I wanted something that felt unmistakably mine.

That means a dark base, lots of purple, magenta and pink, and just enough LCARS influence to make my Star Trek problem obvious without turning every page into the bridge of the Enterprise.

The site is now intended to be the public companion to Pathfinder.

Not a raw copy of my private infrastructure documentation and definitely not a live inventory of my network.

Instead, I want to use it to talk about the interesting parts:

why I made particular architecture decisions, what went wrong, what I changed, what I’ve automated, how I tested it and what I learned.

That’s a much better representation of the project than a collection of hardware specifications.

The private platform and the public story

One useful boundary has emerged from all of this.

My private Pathfinder documentation is the canonical engineering record.

It contains the real infrastructure, operational notes, recovery information, tasks and architecture.

LetsGeek is the curated public version of that story.

That means I can talk about things like monitoring architecture, Ansible roles, backup design or a future Kubernetes cluster without publishing information that has no reason to be public.

It also means the website can be written for humans rather than behaving like exported internal documentation.

That distinction feels important.

What comes next

Monitoring v1 is complete, but observability isn’t finished.

The next monitoring work will probably involve tightening exporter firewall access, putting more dashboards into version control, monitoring Docker workloads, and eventually bringing Proxmox and PBS health into the same picture.

There is also an offsite PBS server waiting for its first proper synchronisation, more recovery testing to do, a three-node Raspberry Pi Kubernetes lab waiting patiently for attention, and an extremely hackable Spaceagon badge currently making dangerous suggestions from the electronics pile.

And now I finally have somewhere current to write about all of it.

For the first time in a while, the platform, the documentation and the website are all telling roughly the same story.

Systems nominal.

Mostly. 🖖

Further reading

• Monitoring a homelab without alerting on everything
• One Ansible baseline across VMs, containers and Raspberry Pi
• Why I started Project Pathfinder