Fronting Proxmox VE With Caddy

Load-Balanced Reverse Proxy

#

How DNS and Caddy Work in My Homelab covers the general Caddy setup: the wildcard certs, the core/apps split, all of it. This is a quick one, a single worked example for the 3-node Proxmox VE cluster.

Proxmox VE gives each cluster node its own web UI on port 8006. In my three-node setup, that’s node-01, node-02, and node-03, each with its own URL to remember. I wanted one hostname, proxmox.lab.example.com, covered by the same wildcard cert Caddy already issues for everything else, that reaches whichever node happens to answer. That’s a normal reverse-proxy job, except for one setting that isn’t optional the way it usually is.

The Caddyfile Fragment

#

This lives in conf.d/pve.caddy, one of the Ansible-managed fragments the general Caddy post already describes:

proxmox.lab.example.com {
    @notallowed not remote_ip 10.20.5.15 10.20.53.0/24
    respond @notallowed `
   ACCESS DENIED
  ┌───────────────────────────┐
  │  You did not reach the    │
  │  Management VLAN.         │
  │  That's the whole point.  │
  └───────────────────────────┘
   Try hop-1 or the management VPN.
` 403

    reverse_proxy https://node-01.lab.example.com:8006 https://node-02.lab.example.com:8006 https://node-03.lab.example.com:8006 {
        lb_policy ip_hash
        lb_try_duration 5s
        lb_try_interval 250ms
        health_uri /
        health_interval 10s
        health_timeout 5s
        fail_duration 30s
    }
    encode zstd gzip
}

The Proxmox VE cluster already sits on its own isolated management network at the firewall level, blocked from every network except a couple of specific admin paths and this proxy itself. The remote_ip matcher at the top of this fragment adds a second, independent check on top of that, and it only went in once both of those admin paths, hop-1 and the dedicated management VPN, were confirmed working end to end. It restricts this hostname to the same two paths and returns a 403 to everyone else before reverse_proxy ever runs. If the network-level rule ever had a gap, this one still wouldn’t.

The health_* settings poll each node’s / every 10 seconds and pull it out of rotation for 30 seconds if it stops answering, so a rebooting node doesn’t take the login page down with it. lb_try_duration and lb_try_interval control how long a single request keeps retrying against another upstream before giving up. Standard reverse-proxy settings, until you get to the actual lb_policy.

Upstream connections are plain HTTPS too, with normal certificate verification against each node’s own Let’s Encrypt certificate rather than skipping it. The proxy itself doesn’t terminate its own certificate either, it rides the same wildcard the rest of the Caddy setup already issues.

Update

Why It Has to Be ip_hash

#

My first instinct was to use round_robin, the normal preferred policy for stateless HTTPS. The Proxmox VE web UI isn’t stateless, at least not once you open a console.

The noVNC and xterm.js consoles run over WebSockets. When you click “Console,” the node that served that page creates a VNC or terminal ticket and hands the browser a WebSocket URL to connect back to. That WebSocket has to land on the same node that issued the ticket, because no other node knows about it. Round-robin doesn’t care which node issued anything, so roughly two times out of three the console WebSocket gets sent to a node which doesn’t know about the ticket, and the console drops the instant it tries to connect.

flowchart LR
    client["Browser"]
    caddy["Caddy 
 ip_hash"]
    node01["node-01 
 issued the ticket"]
    node02["node-02"]
    node03["node-03"]

    client -->|"page load"| caddy -->|"pinned by IP"| node01
    client -.->|"console WebSocket"| caddy -.->|"same node, every time"| node01
    caddy -.-x node02
    caddy -.-x node03

ip_hash fixes this by pinning each client IP to one backend for the life of that pinning, so the page and its console WebSocket always land on the same node. The trade-off: if that node goes down mid-session, the client re-hashes to a different one, and any console that was open at the time has to be reopened. The Proxmox VE auth cookie is cluster-signed, so you usually don’t have to log back in, just click “Console” again.

A load-balanced reverse proxy with health checks, console-aware pinning, and its own IP allowlist, for a cluster only I ever log into, is overkill. That’s more or less the whole point of having a home lab.