Week 44: the lab learned to run itself
The Proxmox node became the center of everything — registry, password vault, DNS, media. Then came the part that matters: a lab that updates itself every night and rolls back when something breaks.
This week I rethought the lab from the ground up. The new rule is that every machine has exactly one job:
- The Proxmox node is the always-on server.
- The NAS is storage, nothing more.
- The Raspberry Pi builds arm64.
- The desktop is just a desktop again.
What moved
The container registry (Harbor), the password vault (Vaultwarden), DNS (AdGuard Home, in two copies), the reverse proxy (Caddy, with a wildcard certificate), the media server (Jellyfin, transcoding on Intel QSV) and the start-page dashboard each got their own LXC. Every service now answers to the same name at home and over Tailscale.
A lab that runs itself
- Pinned, then bumped. Every version is pinned in the repo. Renovate notices new releases, bumps them and merges on green CI — major versions included.
- Applied at night, safely. At 04:30 the node pulls the repo and updates whatever changed. Before each change it snapshots that LXC; if the health check fails, it rolls back — data included — and pings my phone.
- Two directions, two copies. Service LXCs are backed up to the NAS every night, and the NAS’s irreplaceable data is pulled to the node.
- Watched from outside. Every service is probed. Three minutes without an answer, or a certificate close to expiry, and an alert goes out.
What I learned
- Backing up unprivileged LXCs to NFS needs a local temp directory; otherwise the user namespace runs into “permission denied”.
- LXCs with bind mounts can’t be snapshotted, so they need a file-level backup instead.
- A single apostrophe in a comment is enough to break a single-quoted bash block. With shellcheck in CI, it was caught on day one.