Building a homelab from scratch: observability - Homelab (05/06)
In the first four posts we built the infrastructure (overview), the network (WireGuard, VyOS, BGP and DNS), the deployment (Kubernetes and GitOps) and the security and operations. The piece that makes all of that truly operable is still missing: seeing what's happening.
A system you can't see can't be operated. Without metrics you don't know if it's tight on resources. Without logs you don't know why it failed. Without traces you don't know where the time went in a slow request. And when part of the system is AI agents that spend money per token, observability stops being a luxury and becomes an economic necessity.
This post covers how the homelab's observability is set up: a central hub on escipion and three signals —metrics, logs, and traces— plus a fourth, AI-specific one.
The hub: all telemetry to escipion
The house rule is simple: all telemetry converges on escipion, the platform cluster. The full stack runs there:
- Prometheus — metrics (kube-prometheus-stack)
- Grafana — dashboards and exploration
- Loki — logs, with 30 days of retention
- Tempo — distributed traces, with 15 days
- Alertmanager — the brain of the alerts
Clusters and hosts don't keep their telemetry: they forward it. Alejandro's local Prometheus retains only 7 days and does remote-write to escipion. Logs travel with Promtail and an OTel collector. And the VyOS routers send their syslog via Vector. The admin interfaces are only accessible from the tailnet — observability defends itself too.
Metrics and probes
Prometheus scrapes the exporters across the whole stack: nodes, pods, PostgreSQL, Redis, Traefik, the AI gateway, and the routers —the VyOS routers export metrics from FRR, their BGP daemon, so a downed BGP session shows up immediately—.
On top of the metrics, the blackbox probes check what the outside world sees: every 30 seconds the public endpoints are probed and Alertmanager fires if an endpoint goes down, if it responds with 5xx errors, or if a certificate is about to expire. It's the difference between finding out from Grafana or from a user.
Logs: they arrive on their own
No service sends its logs by hand. Pods write to stdout, Promtail picks them up and forwards them to Loki; the routers do the same over syslog. Against Loki I can ask “all gateway errors in the last hour” or “the logs of this specific pod before it died” without logging into any server.
Traces and the AI part
Traces travel via OTel (OpenTelemetry) to Tempo and, for the AI part, to Langfuse self-hosted. Each agent run leaves a trace with tenant and agent tags — a single shared Langfuse project, isolated by tags—. The agent runtime and the LiteLLM gateway nest their parts in the same trace, so at a glance you can see how long retrieval took, how long the model took, and what it cost.
And for model spend there's a dedicated Grafana dashboard fed by the gateway's metrics: spend per model and per agent, latency, time to first token, and errors. Watching cost with the same seriousness as availability.
Alerts that don't wake anyone up with noise
Alertmanager on escipion centralizes the rules: down public endpoints, 5xx, certificates, BGP, resources. The philosophy is few, actionable alerts — an alert that requires no action becomes noise, and noise kills the channel.
Closing
Observability is what turns a homelab into something you can operate with confidence 365 days a year. With everything converging on escipion there's a single place to look, and with AI measured in tokens and cents there are no surprises on the bill.
The next post covers the other side of operating: backups and restoration — what gets backed up, how, and what happens on the day it's actually needed.
Next: Post 6 - Backups and restoration | Previous: Post 4 - Security and operations | Back to Post 1
The complete series
Setting up a homelab from scratch: backups and restore - Homelab (06/06)
LLMOps on your own cluster with LiteLLM, Langfuse, and ArgoCD
Building a homelab from scratch: security and operations - Homelab (04/06)
Postiz: scheduling social media posts from your own server
Building a homelab from scratch: Kubernetes and GitOps - Homelab (03/06)
