↓ Skip to main content
  1. posts/

Getting Proxmox Backup Server Running, and Keeping a Copy Offsite

·1857 words·9 mins·
 Author
Author
Christian Mohn
IT veteran, podcaster, author, and blogger from Bergen, Norway.
Table of Contents
Home Lab - This article is part of a series.
Part 17 (V4.0): This Article

No VM-level backups
#

Most of what runs in my home lab regenerates from Git and Ansible. If a VM dies, I rebuild it from code. A few VMs don’t work that way, but have persistent data stored in them: Home Assistant, Plex, the CI runner that holds my deploy secrets, a media-automation VM, and a separate application-stack VM. Each holds live state that exists nowhere else, and until last week, none of it was backed up at the VM level. The cluster itself had been running for months before I got around to fixing that.

I’d already decided on Proxmox Backup Server (PBS) over plain vzdump to a storage target, for deduplication, incremental and encrypted backups, and a restore UI built into Proxmox VE itself. It took longer than I expected, and surfaced more real problems along the way.

Building the appliance
#

PBS ships its own ISO installer, the same shape as a Proxmox VE node itself. I built it the same way I already build Plex in this cluster: a VM installed straight from that ISO, with my standard ansible user and key added afterward so fleet-wide OS patching still reaches it, but with its own application configuration managed entirely by hand through proxmox-backup-manager and the PBS web UI. No Ansible role for the PBS configuration itself, no secrets file.

The VM runs 4GB of RAM and 2 vCPUs, registered in HA (High Availability), with its OS disk on the ceph-vm pool, the same Ceph-over-Thunderbolt storage the rest of this cluster already runs on. The datastore itself lives somewhere separate: a dedicated NFS mount from the NAS that already serves other storage in this cluster.

Three bugs before the first backup
#

Getting the appliance from installed to somewhere I actually trusted the backups it produced surfaced three real bugs, beyond the usual setup checklist.

Token permissions and the parent user
#

PBS API tokens are capped by their parent user’s own ACL as well as the role granted to the token itself. Granting a role to the token alone did nothing. The backup job kept failing with permission errors until I granted the identical role to the plain user behind that token too. Effective permission is the intersection of both:

proxmox-backup-manager acl update /datastore/primary DatastoreBackup --auth-id backup-job@pbs
proxmox-backup-manager acl update /datastore/primary DatastoreBackup --auth-id 'backup-job@pbs!pve'

Skip the first line and the token grant on the second line is silently worthless.

The missing audit permission
#

DatastoreBackup alone can’t list or audit a datastore. Adding the storage to Proxmox VE with pvesm add needs DatastoreAudit as well. Without it, the datastore list comes back empty, no error, and pvesm add surfaces that as a “cannot find datastore” message that points everywhere except the real cause. Once I added DatastoreAudit alongside DatastoreBackup, the same pvesm add command that had been failing worked immediately.

Datastore root permissions
#

datastore create left the datastore’s root directory at mode 000, owned by backup:backup. That’s invisible as a bug if you only ever check it as root, since root bypasses Unix permission checks entirely and everything looks fine. It only showed up once I tested as the actual unprivileged backup service user PBS itself runs as, and found it locked out of its own datastore, which seemed somewhat unfortunate for a backup system. A chmod to something the service account could actually traverse fixed it.

First backup, then two schedules
#

Before trusting any of this with something real, I ran one manual end-to-end test: backed up a disposable VM, then confirmed it landed with the PBS CLI and separately in the web UI.

proxmox-backup-client snapshot list --repository backup-job@pbs@pbs:primary

Two scheduled jobs went live after that: a daily job for the VMs that change constantly, and a weekly one for the VM with the largest disk, which changes far less often between runs.

Sometimes talking to yourself is good
#

Offsite replication was the other thing I still needed to build. PBS has a Remote Sync feature built for mirroring a datastore between two separate PBS servers, but I only have one appliance. The resolution: PBS can treat itself as a remote. Create a Remote object pointing at the appliance’s own API, then a Sync Job that pulls from the primary datastore into a second datastore that happens to live on a remote NFS mount instead of local storage.

graph LR
    vms["Backup jobs"] --> ds1[("Primary datastore 
local chunk store")] ds1 -->|prune + GC, nightly| ds1 ds1 -->|sync job| remote["Remote object
points at this same appliance"] remote --> ds2[("Offsite datastore
NFS mount, remote site")] ds2 -->|GC + verify, nightly| ds2

This keeps both copies inside PBS’s own data model: same deduplication, same verify tooling, same restore UI. Using rsync underneath it instead would have worked too, but it would have left a second, parallel backup mechanism, with its own restore path to keep track of.

Building the chunk store over NFS
#

PBS’s own storage documentation doesn’t mention NFS as a backing option for a datastore at all, and the datastore-creation UI has no way to point at one either. The NFS mount has to be set up by hand, outside PBS entirely, before datastore create ever runs against it. This is a hack that happens to work, not a documented configuration.

The naive version of this, creating the new datastore directly against the remote NFS path, caused a real outage on the first attempt.

A PBS chunk store needs exactly 65,536 subdirectories, 0000 through ffff, created one at a time, and the datastore-creation process holds PBS’s global configuration lock for the entire build. That’s intentional, to guarantee the chunk store is consistent before anything else can touch it. On a local disk that takes seconds. Over the site-to-site VPN link to the remote site, 65,536 sequential round-trip directory creations took hours, and the held lock blocked every other datastore operation on the appliance for that whole window, including that night’s real scheduled backup, prune, and garbage collection (GC) jobs. A half-built, abandoned structure left mounted kept the lock wedged even after I killed the stuck process. Full recovery needed restarting both PBS daemons and unmounting the half-built structure by hand. The production datastore itself was never touched and came back clean, but the jobs blocked that night were real, contained to this one appliance. I was already running an unsupported hack of a storage configuration. Getting bitten by it once felt less like bad luck and more like the price of my somewhat strange setup.

Build local, then ship the structure
#

The lock is only held while datastore create itself is running, so the fix was to stop running it against the slow destination at all. I attached a throwaway local disk, built the datastore there in seconds instead of hours, then used rsync -a to copy the finished skeleton onto the real remote path. That copy still took close to three hours, for a different reason: rsync -a’s per-file attribute-preservation calls (fstatat, fchownat, utimensat, fchmod) are just as round-trip-latency-bound over the VPN link as the original directory creation was, confirmed with strace on the stalled process. The difference is that the copy doesn’t hold PBS’s global lock, so it didn’t block anything real while it ran. Once the copy finished, I hand-edited the datastore’s configuration to repoint it at the real path.

I verified the repoint with a live GC run against the repointed path. It completed cleanly.

The general shape of this problem is worth remembering beyond PBS specifically: when a tool’s “create” path for something chunk-store-shaped, lots of small metadata-only filesystem operations, runs against a high-latency remote filesystem, don’t let it build in place remotely. Build fast and local, then ship the finished structure.

Quieter gotchas
#

The same parent-user ACL mistake from the first build happened again, in a different context. The dedicated sync token needed the identical “grant it to the user too, not just the token” fix as the original PVE-backup token did. Same root cause, same fix, different credential. Documenting a gotcha once doesn’t guarantee I’ll remember it, or even check the docs, the next time it shows up looking like something else entirely.

Renaming a Proxmox VE storage ID orphans its secret file. Proxmox VE keeps a PBS storage’s API token secret in a file keyed purely by the storage ID’s filename; the main configuration never references it by content. Rename the storage ID without also renaming that file and every node gets a 401 Unauthorized, propagating just as fast as the fix does once found, since both the config and the secret file live on the same cluster-replicated filesystem.

Two unattended runs
#

The real test was whether the whole chain, backup, prune, garbage-collect the primary datastore, sync offsite, garbage-collect the offsite copy, would run on its own overnight with nobody there to catch a failure. After two runs, proxmox-backup-manager task list --all has both nights on record:

StepNight 1 (first run)Night 2
Daily backup (sequential)41m 54s, some overlap5m 11s, cleanly sequential
GC on the primary datastore47m 21s50m 51s
Sync to the offsite copy2h 11m 48s, first full copy41m 55s, incremental
GC on the offsite copy8s, nothing there yet1h 28m 36s

Every stage reported clean both nights. The sync collapsed from over two hours to 42 minutes once the first full copy was done and later runs switched to incremental, and the daily backup dropped the same way, from 42 minutes with some overlap to 5 minutes cleanly sequential. GC is the one step whose cost doesn’t scale down with incrementality: it has to walk the entire chunk store checking reachability on every single chunk, so its runtime tracks the total size of the dataset rather than the nightly churn. That’s why the offsite copy’s GC went the other way, from 8 seconds with nothing yet to collect on the first night, to 1h 28m once there was a real dataset to walk on the second. The same full walk, over a remote NFS mount instead of local disk, pays a round-trip cost on every chunk too.

The numbers
#

That first full offsite sync, the 2h 11m 48s one, moved 37.26 GiB in 25,304 chunks, averaging 4.83 MiB/s, roughly 40 Mbps sustained. That’s almost certainly the real upload ceiling of the site-to-site VPN link rather than anything PBS-specific, and every sync after the first is incremental, so it only applies once. After GC, both the primary and offsite datastores report identical dedup stats: 452 GiB logical data, 36.7 GiB on disk, 25,050 chunks, a 12.31x dedup factor. That’s a clean, verified mirror, confirmed by the numbers matching on both sides.

What’s running now
#

Two PBS datastores, a primary and an offsite NFS-backed mirror, with scheduled backup jobs for every VM that needed one, a self-referential Remote and nightly Sync Job between them, GC and verify schedules on both datastores, and least-privilege dedicated credentials for each job. Recovery procedures exist for file-level restore, full-VM restore, and restoring directly from the offsite copy if the primary datastore is ever lost.

I now have working backups, stored both locally and remotely.

Home Lab - This article is part of a series.
Part 17 (V4.0): This Article

Related