Self-Hosted Platform Lab: Proxmox VE, Run as Code

Architecture

Guests fall into five classes: protected always-on, on-demand, disposable, templates and test. A documented scheme of ID ranges and resource pools keeps them apart, and the automation enforces it: provisioning scripts are idempotent and refuse to clobber an existing or protected guest.

Data volumes are bind-mounted into the guests that use them, so compute stays disposable, and a weekly timer trims the thin volumes.

Operating Principles

The repository sets out eight principles:

• Infrastructure is reproducible
• Configuration is version-controlled
• No secrets in Git
• Destructive operations need explicit confirmation
• Changes are tested before deployment
• Scripted, idempotent changes beat undocumented manual ones
• Persistent data stays separate from disposable compute
• Documentation is part of the infrastructure

Zero-Trust Access with Zero Port-Forwards

The only public way in is an outbound-only Cloudflare Tunnel. Its connector runs in a minimal container of its own, with SSH disabled and unattended security updates that cover the connector. The tunnel is managed remotely, so the container holds nothing but a connector token. Every published app sits behind an identity-aware Cloudflare Access policy at the edge, and nothing is reachable through a port-forward.

Administration has a separate plane: a Tailscale mesh built on WireGuard. The hypervisor is the mesh's subnet router and advertises one isolated internal network. I withdrew the home-LAN route on purpose. The mesh now routes only to the lab and the hypervisor, not the rest of the home LAN, and mesh devices no longer clash with foreign networks that use the same common private range. IP forwarding is managed as code, through a sysctl drop-in that the network script writes.

Services that live only on the mesh use Tailscale Serve for automatic HTTPS inside it, and admin SSH over the mesh is ACL-enforced: the policy refuses any user it doesn't authorise.

Segmentation That Fails Closed

Service-to-service traffic and the tunnel's origin connections run on an internal-only Linux bridge with no physical uplink, so nothing on it depends on addresses handed out by the ISP router. I cut it over live, in an expand-contract sequence:

1. 1 · Snapshot first
   Snapshot the affected guests and back up the host network config.
2. 2 · Add the bridge
   Create the internal-only bridge. Nothing uses it yet.
3. 3 · Attach a second NIC
   Give each affected guest a second interface on the new bridge. Purely additive: the old path keeps working.
4. 4 · Extend the allow-list
   The reverse proxy accepts the new network alongside the old one.
5. 5 · Repoint the tunnel origins
   The connector reaches the apps over the internal bridge.
6. 6 · Advertise the new route
   The mesh advertises it next to the old one, so both paths work during the change.
7. 7 · Verify, then contract
   Only after verification: remove the old allow-list entry and withdraw the old route.

The rollback is written out in reverse order, and a separate runbook covers replacing the router.

The on-demand app stack sits behind default-deny nftables that admit app ports only from the tunnel connector. Docker is told not to manage the firewall, so a published port can't open a hole that nftables doesn't know about, and Docker's systemd unit requires the firewall unit. A broken firewall means no service, never an open one.

A Private Git Forge on Gitea

Gitea gives me a private Git forge for my repositories, so private code can live on hardware I control, under my own policies, rather than on a public code-hosting service.

Build

A scripted, idempotent container build that refuses to run if the guest already exists. The guest is an unprivileged Debian container, with the TUN device passed through for an in-container mesh client.

Supply chain

The release version and its SHA-256 are pinned in the script, and the install aborts on a mismatch. Upgrades follow the runbook: verify the new checksum, take a dump, swap the binary, restart.

Exposure

Gitea binds to loopback, and Tailscale Serve terminates HTTPS inside the private mesh, with no port-forward. Inbound nftables is default-deny and admits only the mesh interface and WireGuard.

Identity

Self-registration is off and sign-in is required to view anything. Gitea's built-in SSH server is off, so clone and push go over HTTPS with per-user access tokens. The first admin password is random and must be changed at first login.

Data

Nightly application-level dumps on a systemd timer, kept locally, with a documented restore procedure. Copying them off-site is the next step.

Golden Images and Disaster-Recovery Drills

Recovery means rebuilding from blueprints: scripts and a 19-step runbook recreate the host and its guests. Backup coverage is deliberately narrow:

Off-site

The golden image and the host configuration. The drill pulls both back and verifies them.

Local only

The Git forge's nightly dumps, on the same disk as the forge. An off-site copy is the next step.

Not backed up

Guest and data volumes, by decision. Guests are rebuilt from scripts, but the data volumes, like any unpushed work, are a single copy on one disk.

Golden-image pipeline

1. 1
   InstallA pinned toolchain from checksum-verified downloads.
2. 2
   ScrubRemove host identity (SSH host keys, machine-id), secrets, caches and logs.
3. 3
   PromoteTurn it into a versioned template that new workstations clone from.
4. 4
   ArchiveKeep a copy off-site.

The DR drill

The drill is run by hand and planned quarterly. It refuses to start if its test IDs already exist, cleans up through an EXIT trap even when a step fails, and never touches protected guests or the live golden image.

1. 1 · Download and verify
   Pull the archived image back and verify its SHA-256.
2. 2 · Restore
   Restore it into a test guest with fresh MAC addresses.
3. 3 · Clone offline and boot
   Make a full clone, remove its NIC and boot it.
4. 4 · Check systemd
   It must report running. “Degraded” is accepted, because network-dependent units are expected to fail offline.
5. 5 · Check host identity and health
   Fresh SSH host keys and a fresh machine-id, a passing offline health check, and app config at 0600 permissions.
6. 6 · Validate the host-config bundle
   Fetch the latest bundle and validate it.
7. 7 · Destroy and log
   Tear everything down and append PASS or FAIL, with the duration, to a log.

First run: PASS in 336 seconds.

Host-config backup

A weekly timer bundles the hypervisor and network config, excluding private keys and tokens, and copies it off-site with rclone. It uploads only when the content changes, keeps the newest 8 versions, and re-uploads if the remote copy goes missing.

Least-Privilege Infrastructure Access for AI Agents

An AI coding agent works on the hypervisor through a scoped, privilege-separated API token. The hypervisor's ACLs are the security boundary, not the agent; the wrapper CLI and the agent's permission rules only add guard rails.

Scope

Read-only by default, with full lifecycle control only inside a disposable resource pool. It can't reach protected guests or platform configuration: host, storage, users and ACLs, console, guest-agent exec.

Escalation

Withholding user, ACL and pool-membership rights closes the obvious escalation paths: it can't grant itself rights or pull a protected guest into its own pool.

Guard rails

The wrapper CLI adds a protected-guest deny-list and a RAM admission check, and destructive actions need out-of-band human approval.

Secrets

The agent's permission rules deny it read access to the hypervisor's token file. Its own model-provider API key comes from a key helper, so the shells and tool servers it spawns never inherit it. Verified: child processes see 0 bytes of it.

Operations

Setup is idempotent and prints the effective permissions. Revoking the token takes one command, rotating it is a revoke plus a re-run of setup, and every task the agent starts lands in the hypervisor's task log under its own identity.

Alerting and Self-Healing

Alert on transitions, not states. A systemd timer runs every 10 minutes and keeps a state file of open problems. It pushes a phone notification only when a problem opens or clears.

Hypervisor notifications, such as backup jobs and updates, reach the same channel through a webhook, not email.

Self-healing. A watchdog timer restarts the web IDE after three consecutive failed health checks. A restart-on-failure drop-in covers the reverse proxy, journald is size-capped, and an hourly self-health check logs warnings to the journal; the same tool has a JSON mode for scripts.

On-Demand Workloads with Guard Rails

Self-hosted productivity apps run on demand as a version-pinned Docker Compose stack in an unprivileged LXC container. They're off by default.

Start guard. The start command refuses unless every hostname redirects to the SSO login, and refuses when the host lacks the RAM to run the stack.

Least privilege. The container mounts only a dedicated, small, size-capped data volume, so the apps can't see or fill anything else. Secrets are generated at deploy time with 0600 permissions and never go into Git. SSH is disabled.

Auto-off. After 60 minutes with no established connector sessions on an app port and CPU under 25%, the stack powers off and sends a push notification. Closed documents are already saved by then, and an open one holds a connection, which keeps the stack up.

The Developer Workstation as Code

A permanent Debian container, cloned from the golden template, is my development environment. Persistent tmux sessions run as lingering systemd user units, SSH is key-only, and security updates are automatic.

The toolchain is captured as code, with pinned versions and checksum-verified downloads: ShellCheck, actionlint, gitleaks, pre-commit and others. A capture script scans for secrets before it commits anything.

Recurring jobs moved from macOS launchd to systemd user timers, and a failed run alerts my phone.

Change Management

One repository holds the provisioning scripts, the architecture, the runbooks (disaster recovery, backup and restore, change management, network rebuild and rollback, credential rotation), a roadmap and audits.

Changes go through a branch and a pull request, and the repository defines these checks:

• ShellCheck on every script, detected by its shebang
• A pattern check for hard-coded secrets
• A guard against API-token literals
• A ban on committed Terraform state or tfvars

Automation runs in explicit operating modes (read, development, infrastructure, destructive), and protected resources need confirmation.

I also ran a read-only security audit of the developer workstation and fixed what it found: an IDE port reachable from the LAN, password SSH, and missing alerting.

Decisions and Trade-offs