I used to write periodic homelab update posts but they’d go stale almost immediately. So instead, I’m just going to keep this page current. It’s the state of things as of the last updated date at the bottom.

Everything is managed via a private Git repo with FluxCD handling the GitOps side of things.

The Hardware

It’s a 5-node K3s cluster, all sitting under my desk. A mix of whatever I’ve picked up cheaply over the years:

  • Control plane (chilli): An old laptop with an Intel i5 (4 threads), 16GB RAM, running Ubuntu 24.04. It does its job.
  • Two x86 workers (bluey, muffin): Both around 8GB RAM, 4 cores each. One’s running Debian 12, the other Ubuntu 24.04.
  • A Raspberry Pi 4 (raspberrypi): 4GB RAM. Just fully rewiped and rejoined the cluster on a fresh Raspberry Pi OS (Debian 13 trixie) install. Needed cgroup_memory=1 cgroup_enable=memory added to /boot/firmware/cmdline.txt before k3s would run at all - undocumented anywhere obvious, but it turns out to be a pretty common requirement for k3s on newer Pi OS. It’s tainted arm=true:NoSchedule, so only apps I’ve explicitly confirmed have arm64 images get scheduled there - a handful of lightweight utility apps like Filebrowser and Memos.
  • The NAS itself (lucky): the bigger change. This used to just be an NFS target sitting outside the cluster; it’s now also a full k3s node, joined 9 days ago. It still runs the same ZFS array (two 12TB Seagate Ironwolf drives, mirrored), but it’s tainted dedicated=jellyfin:NoSchedule so only Jellyfin gets scheduled there. The idea is transcoding happens right next to the media it’s transcoding instead of over a network hop to the NAS.

More than enough compute for what I’m running.

A recent power outage exposed a boot-order bug worth noting: on one node, containerd’s data directory lives on a drive mounted via /etc/fstab with nofail. Nothing told k3s-agent to actually wait for that mount before starting, so a reboot could race containerd against the drive and corrupt its local image cache - which is exactly what happened. Fixed with a systemd drop-in adding RequiresMountsFor= for the mount path.

Storage

ZFS on the NAS

The NAS runs ZFS with the two 12TB drives mirrored, so I get about 12TB usable with redundancy. I’ve split it into datasets so I can apply different policies:

  • nas/media - all the media, lz4 compressed
  • nas/important - the stuff I really can’t lose, with snapshots enabled
  • nas/k8s - where Kubernetes PVCs end up
  • nas/backups - what it says on the tin
  • nas/tmp - scratch space

Snapshots on the important stuff are handled by zfs-auto-snapshot. It’s not a proper backup strategy on its own but it’s saved me more than once.

How the cluster uses it

On the Kubernetes side, I use the nfs-subdir-external-provisioner to create a nfs-client StorageClass. Most apps just request a PVC and get a directory on the NAS automatically. There’s a second provisioner just for music storage (Navidrome, Picard, etc need access to the same music library).

Some apps that need better I/O use local-path storage directly on whichever node they’re scheduled on.

The cluster has 80+ PVs at this point. Media-heavy apps like Jellyfin and Immich mount the big ZFS datasets directly. The monitoring stack (Prometheus, Tempo, Alertmanager) takes up less storage than it used to now that Loki and Grafana are gone. Each database gets 8Gi via CloudNativePG.

How it all fits together

FluxCD

The whole thing is GitOps. I have a repo with a pretty simple structure:

acplpuss/ters/k3s/##OCnleusdtierre-clteovreylpceornfaipgp,liFclautxiobnootstrap,registrysecrets

Each app directory has a kustomization.yaml pointing at its namespace, Helm release, ingress, certs, and storage. Push to the repo and Flux picks it up. There are currently 38 HelmReleases running across the cluster.

Sealed Secrets

Since the repo contains secrets (database passwords, API keys, etc), everything is encrypted with Bitnami Sealed Secrets. I create a normal Kubernetes Secret locally, seal it with kubeseal, and commit the encrypted version. Only the controller running in the cluster can decrypt them. It works really well and means I don’t have to worry about what I commit.

Renovate

Renovate runs as a CronJob inside the cluster itself. It watches Helm chart versions and container image tags, then opens PRs when updates are available. Patch and minor bumps on low-risk stuff auto-merge - through Renovate’s own token once every check goes green, since GitHub’s native auto-merge needs branch protection that isn’t available on a private repo under GitHub’s free plan. Everything else I review manually, but it means I’m never more than a PR away from being up to date.

For a while the only check gating that automerge was pre-commit - yamllint, checkov, kube-linter, that kind of static linting. Nothing actually rendered a chart or built an overlay, so a Helm bump that broke its values schema would sail straight through. I added three more CI jobs: kustomize-build renders every changed Kustomize overlay client-side, helm-render resolves each changed HelmRelease’s real chart/version/repo and runs helm template against it (the highest-value check, since that’s the actual failure mode for chart bumps), and cluster-dry-run spins up a throwaway kind cluster on the runner and does a server-side dry-run apply of the whole Flux tree against it, with pinned CRDs installed for every operator the repo uses.

cluster-dry-run looked fine locally but turned up two bugs once it ran for real. CloudNativePG’s CRDs are vendored in the repo, which I’d assumed meant they came along for free, but a CRD and a resource depending on it applied in the same batch doesn’t work reliably since kubectl doesn’t wait for the CRD to register first - the fix was installing it as an earlier step. And kube-prometheus-stack’s Prometheus and Alertmanager CRDs hit an actual API server limit: client-side apply stores the whole submitted manifest in an annotation for future diffing, and those CRDs’ schemas are big enough to blow past the 256KiB annotation cap. Server-side apply doesn’t create that annotation, so switching to it fixed it.

Ingress and TLS

Traefik comes bundled with K3s so I use that for all the routing. Each app gets its own subdomain and cert-manager handles the TLS certificates. For local dev access I use mkcert.

Databases

Postgres is handled by CloudNativePG, which lets me run proper Postgres clusters as Kubernetes workloads. Immich, Gitea, Fittrackee, Wallabag, Paperless-ngx, Honcho, and Coder all get their own dedicated Postgres instance. Immich’s uses pgvecto.rs for its ML-powered photo search which is pretty neat.

What’s running

The dashboard

I use Homepage as a landing page. It shows all the services with live health checks and a cluster resource overview. It’s the first thing I check in the morning.

Media

Productivity

  • Miniflux for RSS - it’s fast and stays out of the way
  • Paperless-ngx for scanning and archiving documents. The OCR is brilliant, I just scan everything and search for it later
  • CryptPad for documents when I want end-to-end encryption
  • Wallabag for saving articles to read later
  • Memos for quick notes and scratchpad thoughts

Self-hosted infrastructure

  • Gitea with Valkey for caching. It’s my own Git server but doesn’t serve as a source for Flux, as I worried about circular dependencies.
  • Immich with Redis. This is probably my favourite thing running on the cluster - it’s a full Google Photos replacement and it’s genuinely good
  • Coder for self-hosted cloud dev environments, with its own Postgres
  • Perkeep for personal content-addressable storage
  • Filebrowser as a simple web file manager, backed by NFS

I also decommissioned Matrix (Synapse + Element, plus bridges to WhatsApp, Telegram, Signal and LinkedIn) this year. It worked, but it was a lot of surface area - multiple bridges, its own Postgres, its own registration secrets - for what turned out to be pretty marginal daily use. One less thing to keep patched.

AI agents

This replaced my old local-LLM setup entirely (previously llama.cpp in RPC mode, Open WebUI as the chat interface, and Planka wired up via a custom Python bridge). That stack was fun to build but experimental in the bad way - it fell over often and wasn’t something I actually relied on.

What’s there now is a small system I’ve built myself rather than deployed off the shelf - the images all live under ghcr.io/shano/*:

  • Hermes - a voice/chat AI agent gateway, with its own dashboard. This is the actual front door I talk to.
  • Honcho - the agent memory/context service behind Hermes. It has its own Postgres plus its own API, deriver, and Redis components.
  • family-ops - home automation and family reminders glue, tied into Hermes.
  • hamcp - an MCP bridge that exposes Home Assistant as tools the agent can call.

It’s still evolving, but it’s already more genuinely useful day-to-day than the old chat-window-in-front-of-a-model setup ever was.

Smart home

  • Home Assistant for home automation, with hamcp (above) exposing it to the AI agent stack

Tracking

Monitoring

I’d gone a bit deep on the monitoring stack at one point, then pulled some of it back out. I dropped Grafana, Loki, and Uptime Kuma - they were extra surface area for dashboards I rarely opened and log queries I rarely ran. Prometheus and Alertmanager stayed, untouched.

The core is kube-prometheus-stack which gives me Prometheus and Alertmanager. Tempo and the OpenTelemetry Collector (running as a DaemonSet) still handle distributed tracing.

For alerting, Alertmanager sends critical alerts to ntfy, which pushes notifications to my phone. I’ve set up inhibition rules so I don’t get spammed - critical alerts suppress warnings, warnings suppress info-level stuff. I recently added Telegram as a second delivery channel, mainly so a single notification path being down can’t silently mean I miss something critical.

Off-site backups are handled by Volsync, which replicates PVCs to Cloudflare R2. CloudNativePG databases get their own continuous backups to R2 as well. The Sealed Secrets controller key is also backed up to R2 monthly so I can recover the cluster from scratch if needed.

That backup path had a real incident recently worth writing down. My R2 storage bill started climbing for no obvious reason. Turned out spec.paused: true on a VolSync ReplicationSource doesn’t just pause the schedule the way the name implies - it also forces the generated backup Job’s parallelism to 0, so it silently creates zero pods, forever, with no error surfaced anywhere. Compounding that, a stale restic lock left over from a manual size-check I’d never cleaned up had been quietly blocking one app’s backups for 18 days.

Worse: the Prometheus alert that was supposed to catch exactly this kind of backup failure had never actually worked. It watched kube_pod_status_phase for failed pods, but VolSync garbage-collects those pods almost instantly, so the alert could structurally never fire. Fixed it by alerting on VolSync’s own volsync_volume_out_of_sync metric instead (had to add a missing ServiceMonitor just to get it scraped in the first place), and added a weekly job that diffs the live R2 bucket against known backup sources - so orphaned data, like the ~5GB the Matrix decommission left behind, gets caught automatically instead of by accident.

What’s next

Honestly this changes all the time but the things on my mind:

  • Proper network segmentation. Right now everything’s flat on the same LAN which isn’t great
  • Maybe a second cluster at some point for playing with multi-cluster tooling

Last updated: August 2026