Locking Down the Proxmox Management VLAN

Why Isolate Proxmox Management at All

#

The Proxmox VE web interface and its underlying API are the keys to the whole cluster. Anything that can reach them can attempt to authenticate, and from there create, delete, or migrate virtual machines, read cluster configuration, and in most homelabs, do all of that with a single set of credentials that also has root on every node. Leaving that reachable from every device on a flat network is leaving a very wide door open for very little reason.

Proxmox VE has its own built-in firewall, at the datacenter, host, and VM level, and it can do most of what I wanted here. I still wanted the isolation to live at the network layer instead, in UniFi, not layered on top of the cluster it’s protecting. A rule enforced by the network doesn’t care whether Proxmox VE itself is misconfigured, mid-upgrade, or compromised.

I already wrote about what happens when a hypervisor’s own access control breaks down, from the other direction, in vSphere: It’s All Fun and Games Until Someone Gets Root. That post is about VMware ESXi and vSphere, and this one is entirely Proxmox VE and UniFi, but the underlying concern is the same one, and it keeps showing up in my own work: hypervisor-level access is worth controlling deliberately, on whatever platform you’re running, because the blast radius on the wrong side of that boundary is everything.

So the management network for my Proxmox VE cluster got its own VLAN, reachable from almost nowhere else on my network by default. This post is about how I actually got there without breaking anything that was supposed to keep working, and the two independent ways I built back in once it was locked down.

Finding the Real Dependencies First

#

The obvious way to isolate a network is to block everything, then start poking holes as things break. I did not do that, on purpose. A hypervisor cluster has automation touching it in ways that aren’t always obvious until they stop working, usually at an inconvenient moment, and I’d rather find those dependencies on my own schedule than have them find me.

Instead I went the other way: audit first, find every genuine dependency, carve out exactly those, verify that each carve-out actually works, and only then add the block rules. Two real dependencies turned up:

• A reverse proxy, proxy-1, the same Caddy setup from Fronting Proxmox VE With Caddy, needs to reach the Proxmox VE web interface for exactly one proxied hostname, proxmox.lab.example.com. TCP port 8006 only. Caddy itself only forwards that hostname to clients matching its own remote_ip allowlist, a rule I added to that config at the same time as this lockdown, on top of the network-level isolation this whole post is about.
• A power-monitoring service, pwr-1, polls the Proxmox VE API directly to check cluster state. TCP port 8006 again.

I built a narrow allow rule for each one, and confirmed both were working before I touched a single block rule. That ordering turned out to matter more than I realized at the time, for reasons that have nothing to do with being careful and everything to do with how the firewall itself evaluates rules.

The management network is also where I’d go to fix a bad rule. Get the order wrong and I lock myself out of the one place I’d need to be to undo it. The branch I was sawing through was the one I was sitting on.

Two Ways Back In

#

Locking a network down is only half the job. The other half is making sure I can still get into it, reliably, without depending on any single thing that could itself be the reason I need in during an actual problem. I built two access paths on purpose, and neither one depends on the other working.

flowchart LR
    me["Endpoint"]

    subgraph flat["Default home network"]
        hop["hop-1 
 hardened jump host"]
    end

    subgraph vpnctl["UniFi controller"]
        vpn["Dedicated Management VPN 
 narrow, purpose-built"]
    end

    mgmt["Management VLAN 
 node-01, node-02, node-03"]

    me -->|"SSH ProxyJump 
 sshuttle"| hop --> mgmt
    me -->|"VPN client"| vpn --> mgmt

Hardened Jump Host

#

The first path is a small virtual machine, hop-1, whose only job is being an SSH ProxyJump hop for terminal access and an sshuttle endpoint for reaching the web interface from a browser. Nothing else runs on it.

It sits on the default, untagged home network, not on any of the normal server VLANs. That’s deliberate. A jump host sitting on a network already isolated from management wouldn’t be a stepping stone. It would just be another isolated network. It has to start somewhere that can still reach the destination, on purpose, in a controlled way. There’s a trade-off here too, and I’ll be honest about it. hop-1 is itself a virtual machine, running as a guest on the very three-node cluster it exists to manage. Its network interface is on the default home VLAN, but the physical hardware underneath it is the same hardware it protects access to. That fact is exactly what causes trouble with the power monitor.

A few things about how it’s hardened, beyond the basics:

• A static IP set directly on the guest, not a DHCP reservation. A DHCP reservation is keyed to the guest’s MAC address, which is one dependency too many for something meant to be a reliable stepping stone. If the DHCP server has a bad day, or the guest’s virtual network interface ever gets recreated with a new MAC, the reservation stops applying and nothing tells you it happened. A static IP set on the guest itself doesn’t have that failure mode.
• Key-only SSH, no root login, and a dedicated admin account separate from the account my automation normally uses, with passwordless sudo. This produced a real gotcha. Adding that account to the sudo group still prompted for a password on every sudo command, a password that account was deliberately never given, since it only ever authenticates by key. Group membership alone doesn’t bypass the authentication step, it only grants authorization once you’ve passed it. The fix is an explicit sudoers drop-in:

  # /etc/sudoers.d/admin-nopasswd
  admin ALL=(ALL) NOPASSWD:ALL

• AllowTcpForwarding and AllowStreamLocalForwarding left enabled in sshd_config, against most hardening checklists’ default advice to turn both off. The entire point of this host is being an SSH ProxyJump hop and an sshuttle endpoint, and both of those depend on forwarding working. Disabling it here would harden the box straight out of doing its one job.
• Unattended security upgrades, layered on top of the fleet’s normal patch schedule, since this one host is a higher-value target than most of what I run.
• fail2ban, plus a default-deny inbound firewall that only allows SSH from the two legitimate admin paths and from the core services network. That last carve-out has its own small gotcha. Without it, the fleet’s own deploy tooling can’t reach hop-1 to patch it, and a hardened box that can’t receive its own security updates just falls behind on patches without anyone noticing.

Two more things cost me real troubleshooting time the first time I hit them.

IdentitiesOnly yes is not optional if your local SSH agent has more than one key loaded. Bitwarden, 1Password, and similar agents will happily offer every key they hold to any host that asks, in order, until one works or the server gives up. Most SSH servers cap the number of authentication attempts per connection, and if the right key isn’t first in line, the agent burns through the limit offering the wrong ones before it ever gets to try the one that would have worked. The result is Too many authentication failures, which reads like a configuration problem or a wrong key, and is actually neither. Setting IdentitiesOnly yes in your SSH client config for that host, alongside an explicit IdentityFile, stops the agent from offering anything except the key you actually meant to use.

Never run the sshuttle tunnel to hop-1 at the same time as the dedicated management VPN. Both create a route to the same management network, and the VPN’s kernel route always wins when both exist. Bring up sshuttle while the VPN is already connected, or the other way around, and management traffic doesn’t fail loudly, it just silently blackholes, since the kernel routes it down the connection to the tunnel that doesn’t actually work anymore. Use one or the other, never both.

Dedicated Management VPN

#

The second path is a separate, dedicated management VPN, configured directly in the UniFi network controller rather than as part of the Ansible-managed infrastructure. I already run a general-purpose VPN with full internal access, for everything else I do remotely. To be explicit about what that means here, that VPN is now blocked from the management network too, the same as every other network that isn’t one of the verified carve-outs. Being my own trusted, everyday remote-access path bought it no exception. This dedicated VPN is the one deliberate way around that, scoped to exactly four things: the management VLAN itself, the default home network hop-1 lives on, internal DNS, and the reverse proxy’s admin interface. Nothing else on my network is reachable through it.

Keeping it narrow, and separate from the VPN I use for everything else, is the entire point. If this VPN’s credentials or configuration were ever compromised, the blast radius stops at the management surface I’ve already gone to the trouble of isolating. If my everyday VPN were compromised instead, management is exactly the one thing it still couldn’t reach.

Having both paths configured, verified, and working independently means a problem with either one, a broken jump host, a VPN misconfiguration, doesn’t leave me locked out of my own cluster.

The Rule-Ordering Trap

#

UniFi’s zone-based firewall evaluates rules in index order, not in the order the logic of “this zone pair is now allowed for X” would suggest. That sounds obvious written down, and it still cost me real time to work out.

Partway through building this out, I needed one more carve-out on top of the block rule that was already in place for a zone pair. I added the new allow rule the way you’d add anything to a list, at the end. It did nothing. The traffic it was supposed to permit kept getting dropped, and there was no error, no log entry pointing at the problem, just a rule that looked correct and behaved as if it wasn’t there.

The block rule above it in the list was still matching first. A UniFi zone firewall stops at the first rule that matches a given connection, and an allow rule sitting after a block rule for the same source and destination zones never gets evaluated at all, no matter how specific or how recently added it is. The fix is rule order. The allow has to sit above the block, always, for that zone pair.

The general lesson is the same allow-then-block approach I used to find dependencies in the first place. Build and verify every allow rule first, then add the block rule after it. Never the other way around, and never assume a rule you add later will be re-evaluated against everything already in the list ahead of it. It won’t be.

When the UPS Monitor Broke

#

pwr-1 runs on a small single-board computer that lives outside the cluster entirely, on purpose. A device whose job is detecting a power incident and recovering the cluster afterward can’t depend on that same cluster to run. Its only network connection is Wi-Fi, and the management network deliberately isn’t reachable over Wi-Fi at all. Ideally pwr-1 would just sit inside the management network directly, no carve-out needed, but that hardware constraint took that option off the table on its own. Reaching in from the default home network was the next best option.

The first isolation rule I added blocked pwr-1 immediately. Its own logs made that obvious. They showed connection attempts to the Proxmox VE nodes failing, over and over.

The obvious fix was routing pwr-1’s access through hop-1, the same way I’d think about routing anything else that needed a path into the management network. I got about as far as thinking it through before it hit me. That would cut off the exact thing pwr-1 exists to handle.

pwr-1’s whole job is detecting a power incident, shutting the cluster down cleanly before the UPS runs out, and bringing services back up once power returns. hop-1, meanwhile, is a guest virtual machine running on the same three-node cluster that a power incident would take down. Routing pwr-1’s traffic through hop-1 would have built a circular dependency directly into the one service meant to work when things are already going wrong. A cluster-wide power event is also the scenario where hop-1 isn’t guaranteed to be up yet.

The actual fix was a narrow, direct carve-out. I added an explicit allow rule for pwr-1’s address, to the management network, TCP port 8006 only, built and verified the same way as the other dependency, then placed correctly above the existing block rule for that zone pair. It doesn’t touch the jump host or the VPN at all. It’s a specific, minimal, independently-verified path for the one service that has to keep working even when the more elaborate paths don’t.

It’s also a firewall poke straight to Proxmox VE’s own default management port, from a device I can’t put behind the same hardening as hop-1. If pwr-1 itself were ever compromised, that rule is exactly what it would use to gain further access. I know that, and I’m accepting it as the cost of this particular device for now rather than pretending the carve-out is free. pwr-1 itself isn’t undefended, at least. It runs headless, with key-only SSH and nothing else exposed, so the actual attack surface is smaller than the carve-out alone makes it sound.

Blocking One Network at a Time

#

Once every genuine dependency had a verified carve-out, I added the block rules one network at a time, not as a single blanket “block everything else” rule. Each network got confirmed as either safe to block outright or already covered by an existing allow, before its own explicit block rule went in. That’s slower than one sweeping deny-all rule, and it’s also the entire reason I didn’t break anything I hadn’t already accounted for. A single blanket rule is fast to write and impossible to reason about after the fact, since it doesn’t tell you which specific decision, for which specific network, is doing the blocking.

The Full Picture

#

This is what the finished policy set actually looks like:

And the exceptions themselves, all of them:

Eleven separate firewall policies came out of “isolate one network,” once it was done properly. Four allow rules, seven explicit per-network blocks. None of the individual pieces were hard on their own. Getting the count that high, on purpose, one verified piece at a time, was the actual work.

The management VLAN is locked down now. Both access paths, the jump host and the narrow VPN, are built, verified, and confirmed working independently of each other. I overengineer plenty of things in this home lab just because I enjoy it. This time it actually had a valid purpose. The UniFi rule-ordering trap and the UPS monitor incident are exactly what turns up when you take “isolate one VLAN” this seriously, and I’d rather find them while testing than during an actual incident.