Building a homelab from scratch: security and operations - Homelab (04/06)

In the Post 3 we saw how applications go from Git to running containers. But we skipped a critical question: how do secrets get into those containers?

A database password, an API token, a TLS certificate — none of this can live in Git in plain text. And once deployed, they need to be backed up. And if everything fails, you need a way to rebuild from scratch.

This post covers the three layers of secrets management, the backup strategy, home automation, and what you would need to recover from a total disaster. It is the boring but essential part that makes the difference between a weekend project and infrastructure you can rely on.

The secrets problem

The entire homelab is managed from a single Git repository. Great for reproducibility and auditing, but it creates a tension: configurations have to be in Git for GitOps deployment to work, secrets cannot be in Git for obvious security reasons, and yet secrets have to reach the services automatically at deploy time.

There is no single tool that solves this. There are three, each covering one part of the problem.

Layer 1: Infisical (the vault)

Infisical is a self-hosted secrets manager. Think of it as a password vault for infrastructure. All the homelab secrets — database passwords, API tokens, OAuth credentials, SMTP passwords — live in Infisical as the single source of truth.

Infisical runs on the escipion cluster (Roma network) and is reachable at secrets.zetesis.localhost. Because it is self-hosted, secrets never leave the homelab's control.

Two ways out

For Docker Compose services — ptolomeo (the CD agent) fetches the secrets from Infisical at deploy time and injects them as environment variables:

# .doco-cd.yaml — ptolomeo lee esto
external_secrets:
  DB_PASSWORD: <project-id>:prod:/my-app/DB_PASSWORD
  SMTP_HOST:   <project-id>:prod:/my-app/SMTP_HOST

When ptolomeo deploys a stack, it calls the Infisical API, fetches the values, writes them to .env and runs docker compose up. The container sees DB_PASSWORD=xyz123 as a normal environment variable.

For Kubernetes — the External Secrets Operator (ESO) runs inside the cluster and periodically syncs Infisical secrets as native Kubernetes Secrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: web-secrets
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: infisical-production
    kind: ClusterSecretStore
  data:
    - secretKey: AUTH_SECRET
      remoteRef:
        key: AUTH_SECRET
    - secretKey: AUTH_KEYCLOAK_SECRET
      remoteRef:
        key: AUTH_KEYCLOAK_SECRET

ESO creates a Kubernetes Secret called web-secrets with the values from Infisical. Pods mount it as environment variables or files — without knowing where the secrets come from.

ESO refreshes every minute. If you rotate a password in Infisical, the Kubernetes Secret updates automatically. No redeploy needed.

The ClusterSecretStore pattern

In alejandro, secrets are organized with Kustomize components following the same base/overlay pattern as the infrastructure (from Post 3):

The base defines a ClusterSecretStore template. Each overlay patches it with the correct Infisical project and environment. That way, production reads from the Infisical prod environment and staging reads from staging — same structure, different credentials.

Layer 2: SOPS/age (the file lock)

There are files with secrets that have to exist in the Git repository. For example, Talos Linux needs the cluster certificates and bootstrap tokens to generate the machine configurations. These live in talsecret.sops.yaml next to talconfig.yaml.

SOPS encrypts the values of a YAML file while leaving the keys readable. Combined with age (an encryption tool), it produces files that can be committed safely:

# Lo que vive en Git (cifrado)
cluster:
    id: ENC[AES256_GCM,data:QqUK9Xr6cWv0ed9F...,tag:tnMuAX...,type:str]
    secret: ENC[AES256_GCM,data:ncoTvU2yA5He7n2r...,tag:o5SrlE...,type:str]

You can see the structure (cluster.id, cluster.secret) but not the values. Decrypting requires the age private key — which is stored in Infisical, never in Git.

The .sops.yaml file at the repo root tells SOPS which public key to use:

creation_rules:
  - path_regex: \.sops\.(yaml|yml)$
    age: age1ejemplo0000000000000000000000000000000000000000000000000
  - path_regex: secrets\.env$
    age: age1ejemplo0000000000000000000000000000000000000000000000000

To decrypt (when applying Talos configurations):

Infisical provides the age private key as an environment variable. SOPS uses it to decrypt the files. The decrypted output goes to clusterconfig/ (gitignored, never committed).

What is encrypted in Git

Everything else stays in Infisical and never touches Git.

Layer 3: the chicken-and-egg bootstrap

The puzzle: ESO needs credentials to connect to Infisical. But those credentials are a Kubernetes Secret. And ESO is precisely what creates Kubernetes Secrets from Infisical. So where do ESO's credentials come from?

The answer: Talos inline manifests. The inline-secrets.sops.yaml file contains SOPS-encrypted Kubernetes Secret manifests. When Talos brings up the control plane, it applies these manifests directly — before ArgoCD or ESO even start. That breaks the circular dependency:

These bootstrap secrets are the only ones managed via SOPS. From there on, everything is automatic through ESO.

How the three layers fit together

Infisical is the single source of truth. ptolomeo bridges Infisical and the Docker containers (environment variables). ESO bridges Infisical and the Kubernetes pods (native Secrets). And SOPS/age covers the special case of files that have to exist in Git.

Backup strategy

All homelab data is backed up to a single place: MinIO en spinoza (10.1.0.11). The backup tool is Restic — an encrypted, deduplicating backup program that supports S3.

Docker hosts: tolstoi

All VPSs and VMs run a service called tolstoi — a Restic wrapper that runs daily at 3:00 AM. It backs up Docker volumes to MinIO through the Tailscale gateway (cervantes):

# Configuración base de backup (services/tolstoi)
services:
  resticker-base:
    image: mazzolino/restic:1.8.2
    environment:
      BACKUP_CRON: "0 3 * * *"
      RUN_ON_STARTUP: "true"
      RESTIC_FORGET_ARGS: >-
        --keep-daily 7
        --keep-weekly 4
        --keep-monthly 12
        --keep-yearly 7

Each deployment extends this base with its own volumes and S3 bucket:

Kubernetes: K8up

On the Kubernetes clusters, K8up replaces tolstoi. It is a Kubernetes-native backup operator that discovers and backs up PVCs (Persistent Volume Claims) automatically. The schedule is defined as a Kubernetes resource:

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: backup-schedule
  namespace: turing
spec:
  backend:
    s3:
      endpoint: http://minio-atenas.data.svc.cluster.local:9000
      bucket: escipion-restic
  backup:
    schedule: '30 2 * * *'    # 2:30 AM a diario
  prune:
    schedule: '0 4 * * 0'     # Domingos a las 4 AM
    retention:
      keepDaily: 7
      keepWeekly: 4
      keepMonthly: 6
  check:
    schedule: '0 5 * * 0'     # Domingos a las 5 AM

Three automated operations: backup daily at 2:30, which snapshots all PVCs in the namespace; prune weekly on Sundays, which deletes old snapshots according to the retention policy; and check weekly after the prune, which verifies backup integrity.

K8up is configured per namespace. On escipion there are separate schedules for turing (Infisical), gauss (Harbor) and traefik. Each one backs up to its own prefix in the bucket.

Everything converges on spinoza

The Docker hosts reach MinIO through the cervantes Tailscale gateway. escipion and alejandro reach it over the Tailscale mesh (Roma to Atenas).

Home automation

The homelab is not just servers — it also controls the house. Two services run on the aristoteles VM on the Atenas network (marco-aurelio).

rothbard: Zigbee2MQTT

Zigbee2MQTT bridges home Zigbee devices (lights, sensors, switches) and MQTT, a lightweight messaging protocol. It runs alongside a Mosquitto MQTT broker:

services:
  mqtt:
    image: eclipse-mosquitto:2.0
    ports:
      - "1883:1883"    # MQTT
      - "9001:9001"    # WebSocket
  zigbee2mqtt:
    image: koenkk/zigbee2mqtt:1.42.0
    depends_on: [mqtt]
    ports:
      - 8082:8080      # Dashboard web

The Zigbee coordinator (a USB dongle) talks to the devices directly — with no cloud service in between. Zigbee2MQTT translates device messages into MQTT topics, making them available to any automation system.

mises: Homebridge

Homebridge exposes non-HomeKit devices to Apple Home. It runs in host network mode to handle mDNS discovery:

services:
  homebridge:
    image: homebridge/homebridge:2026-02-25
    network_mode: host
    environment:
      - HOMEBRIDGE_INSECURE=1

Homebridge picks up the devices published to MQTT by Zigbee2MQTT and exposes them as HomeKit accessories. A "Hey Siri, turn off the living room lights" ends up triggering an MQTT message that goes from Homebridge to Zigbee2MQTT, from there to the Zigbee radio, and from the radio to the light bulb.

Both services are reachable through the local Caddy reverse proxy (trajano) at rothbard.zetesis.localhost and mises.zetesis.localhost.

Disaster recovery

If everything goes wrong, you need exactly six values stored outside the infrastructure to rebuild:

With these six values and a clone of the Git repository, the rebuild path is:

1. Decrypt the SOPS files with the age key
2. Boot the first Talos node with talhelper genconfig + talosctl apply-config
3. Bootstrap ArgoCD with kubectl apply -f bootstrap.yaml
4. Restore Infisical from the MinIO backup using the Restic and encryption keys
5. Wait — ESO connects to Infisical, secrets propagate, services start

Once Infisical is up, the rest is automatic. ESO syncs secrets, ArgoCD deploys applications, ptolomeo pulls on the Docker hosts. The full rebuild is documented in the FIRST_STEPS.md file in the repo.

Loss scenarios

The essential part: the age key and the Infisical backup credentials have to exist somewhere outside the homelab — a password manager, a piece of paper in a safe, an offline USB drive. Without them, you start from zero.

GitGuardian: the safety net

Since all the infrastructure is in Git, there is one more layer: GitGuardian scans every push and every pull request for accidentally committed secrets:

# .github/workflows/main.yml
name: GitGuardian scan
on: [push, pull_request]
jobs:
  scanning:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: GitGuardian/ggshield/actions/secret@v1.47.0
        env:
          GITGUARDIAN_API_KEY: ${{ secrets.GITGUARDIAN_API_KEY }}

If someone accidentally commits a plaintext password, GitGuardian catches it before it reaches the main branch.

What we have built

Let's zoom out. Over the course of this series we have covered:

1. The big picture — Hardware, naming, what runs where
2. Networking — Site-to-site WireGuard, Tailscale mesh, Cloudflare at the edge, Caddy reverse proxy
3. Kubernetes and GitOps — Talos Linux, ArgoCD App-of-Apps, ApplicationSets, ptolomeo
4. Security and operations — Three layers of secrets (Infisical, SOPS, ESO), backups with Restic, home automation

Everything runs from a single Git repository. Two CD systems (ArgoCD + ptolomeo) deploy to Kubernetes and Docker hosts respectively. Secrets flow from Infisical through three different mechanisms depending on the destination. Everything is backed up to MinIO on spinoza.

Is it overkill for a homelab? Probably. But every piece exists because I ran into a real problem — and the solution taught me something I now use professionally. That is the real value of a homelab: a playground where the stakes are low, but the lessons are real.

With secrets sorted and a written answer to "what if everything burns down?", what remains is watching the system run and knowing how to recover it. That is what the last two posts cover.

Next: Post 5 - Observability | Previous: Post 3 - Kubernetes and GitOps | Back to Post 1

The full series

Setting up a homelab from scratch: backups and restore - Homelab (06/06)

Jun 18, 2026Case studyHomelab

Building a homelab from scratch: Kubernetes and GitOps - Homelab (03/06)

May 9, 2026Case studyHomelab

LLMOps on your own cluster with LiteLLM, Langfuse, and ArgoCD

Jul 10, 2026LabLLMOps

Postiz: scheduling social media posts from your own server

Jul 28, 2026LabHomelab

Building a homelab from scratch: observability - Homelab (05/06)

Jun 16, 2026Case studyHomelab

Building a homelab from scratch: networking - Homelab (02/06)

May 8, 2026Case studyHomelab