Proxmox VE and Ceph Over a Thunderbolt Ring on Three NUCs

Nothing Went as Planned

#

I rebuilt my home lab this year around three ASUS NUC 15 Pro nodes running Proxmox VE, and the how and why of that whole project is its own post: How I GitOps’ed My Homelab. One piece of that rebuild is the storage layer: Ceph, replicated across all three nodes, running over a dedicated Thunderbolt 4 ring instead of the 2.5GbE management network. That got one paragraph there and a promise that it would get its own writeup once it existed. This is that writeup, and it exists because almost nothing about getting there went the way I expected.

The Plan

#

On paper this looked almost too simple: three cables, a routing protocol, and a storage backend that never has to touch the switch.

• Hardware: three ASUS NUC 15 Pro (NUC15CRHU7) boxes, Intel Core Ultra 7 255H, 96GB RAM, running Proxmox VE 9.2. Each node has a 2TB NVMe drive for Ceph and a separate 500GB NVMe for the OS.
• Topology: a physical ring, each node’s right-hand Thunderbolt port cabled to the next node’s left-hand port, turned into a logical any-to-any mesh by FRR running OpenFabric. Lose one cable and traffic reroutes around the ring instead of splitting the cluster.
• Storage: Ceph, with both its cluster_network and public_network pinned to the Thunderbolt fabric, so replication traffic never touches the management LAN.
• Safety: Corosync stays on its own separate network, so a Thunderbolt problem can never cause a split-brain.

None of that sounded hard. Then I actually tried to bring the links up, and spent the next several hours finding out exactly how wrong that assumption was.

The Ports Enumerated Nothing

#

Cables in, correct ports, genuine TB4 cables, and still no link on any node. No peer devices in sysfs, no interfaces created, nothing in dmesg beyond the local controller binding at boot.

I worked through the stack top to bottom, and the list of suspects is worth keeping around, because every one of them looked plausible at the time.

Then the test that actually cracked it: plugging a known-good USB-C thumb drive into a healthy node’s port produced nothing either, no kernel event, no device, no /dev/sd*. The port wasn’t refusing Thunderbolt specifically, it wasn’t detecting anything at all, which ruled out the cable, security, and power layers in one step and pointed straight at platform firmware.

The Fix Was the BIOS

#

The boxes had shipped with a BIOS that was 15 months old, and USB-C/TB4 attach detection was simply broken in that build. Flashing the current release and rebooting fixed it instantly. The same thumb drive test lit up with a real device node, and a Type-C partner appeared in sysfs.

Two things made this harder to get to than it should have been.

1. Getting the right BIOS for the exact silicon matters. The NUC15CRHU7 is the U-variant. ASUS’s own site also surfaces a “latest BIOS” for the C and I variants, different silicon entirely, wrong flash if you grab it without checking the supported-model list first.
2. The medium matters too. There is no Windows anywhere on these boxes to run the vendor’s update package on. I hooked up a monitor and keyboard just long enough to do it properly, and used the BIOS’s own flash tool instead: F7 at boot, plus a FAT32 USB stick holding the .cap file.

Flashed all three nodes, re-cabled the ring, and the links trained instantly. PCIe width jumped from x0 to x4, peers appeared, and thunderbolt-net interfaces came up on every node, each one discovering both neighbors by hostname in dmesg, matching the ring exactly as planned.

On new small form factor hardware, a BIOS update belongs at the top of the runbook, not somewhere you arrive at after ruling out cabling, power management, and ASPM one at a time.

Numbered Addressing

#

OpenFabric supports fully unnumbered links, which would have been the cleanest config possible. I tried it first, and it failed: FRR’s IPv4 OpenFabric won’t install loopback routes without an IPv4 next-hop on the links, you’d need an IPv6 underlay to do it properly. Only one of three nodes got routes, and every loopback ping failed.

So I went numbered instead, with two separate ranges doing two separate jobs.

• Loopbacks are the stable identity. Each node gets a /32 on a dummy interface, pve01 at 10.100.0.11, pve02 at .12, pve03 at .13, mirroring the management LAN octet so the mapping stays memorable, and this is what Ceph itself binds to. A dummy interface comes up at boot regardless and is immune to Thunderbolt reconnects, which turns out to matter a lot.
• The physical links get throwaway /30s in a separate 10.100.255.x range, pure OpenFabric next-hop plumbing that Ceph never touches.

graph LR
    subgraph N1["pve01 · lo 10.100.0.11"]
        p1e6["en06 LEFT/0d.3"]
        p1e5["en05 RIGHT/0d.2"]
    end
    subgraph N2["pve02 · lo 10.100.0.12"]
        p2e6["en06 LEFT/0d.3"]
        p2e5["en05 RIGHT/0d.2"]
    end
    subgraph N3["pve03 · lo 10.100.0.13"]
        p3e6["en06 LEFT/0d.3"]
        p3e5["en05 RIGHT/0d.2"]
    end
    p1e5 -- "Link A · 10.100.255.0/30" --> p2e6
    p2e5 -- "Link B · 10.100.255.4/30" --> p3e6
    p3e5 -- "Link C · 10.100.255.8/30 · ring-back" --> p1e6

MTU 65520 on the TB interfaces, net.ipv4.ip_forward=1 so each node can transit for the others, and a three-line OpenFabric configuration block per interface. I verified it with show openfabric neighbor (two adjacencies per node), a full loopback-to-loopback ping matrix, and a jumbo ping -M do -s 60000 for good measure.

IPs That Vanished on Reboot

#

Everything worked, until I rebooted a node to test durability and it came back with the Thunderbolt interfaces up but no IP addresses.

The root cause: Thunderbolt interfaces appear about 8 seconds into boot, well after networking.service has already run. The normal interface configuration never gets applied, because the interface doesn’t exist yet when Debian brings the network up. My udev-triggered bring-up script was only running ip link up, which sets the link and MTU but not the address.

It gets worse. The same thing happens whenever a neighbor reboots. When the node on the other end of a cable goes down, the local facing interface gets torn down and recreated, again without its address. OpenFabric’s adjacency masked this for a while, the mesh still looked fine, but the /30s were quietly gone underneath it.

The fix is ifup --force instead of ip link up. ifup reapplies the entire interface configuration, address and MTU both, every time the interface reappears, whether that’s at boot or when a peer reconnects. I wired it into both the udev move rule and a boot-time polling service, and the fabric became genuinely durable across reboots and reconnects. I confirmed it the right way too, rebooting a node and checking that its own IPs came back, then checking that both peers’ facing IPs auto-recovered on their side once it reconnected, and both held up.

There is one cosmetic quirk worth knowing about ahead of time: a thunderbolt-net interface can sit at operstate=down while it is actually up and passing traffic normally. The first time you see it, it looks like the link just failed. It has not, the interface itself is fine, only the reported state is wrong.

PCIe Active State Power Management (ASPM)

#

On these Meteor Lake and Arrow Lake class platforms, PCIe Active State Power Management will power-gate the Thunderbolt root ports and destabilize the links. The durable fix is pcie_aspm=off on the kernel command line. A udev rule pinning the specific root ports’ power/control to on holds things stable until the next reboot, when the kernel parameter takes over properly.

Setting Up Ceph

#

With a stable fabric, Ceph’s configuration was almost anticlimactic. On a no-subscription Proxmox VE install, repoint the Ceph repo and install Squid, then:

• pveceph init with both networks set to 10.100.0.0/24, so all Ceph traffic stays on Thunderbolt.
• Three monitors, in quorum, bound to the TB loopbacks, and three managers, one active and two standby.
• One OSD per node on the 2TB NVMe.
• A replicated pool: size 3, min_size 2, failure domain set to host, autoscaler on.

One gotcha along the way: pveceph pool create defaulted the PG autoscaler to warn and over-provisioned PGs, throwing POOL_TOO_MANY_PGS. Flipping autoscale to on cleared it.

The result: HEALTH_OK, 5.5TiB raw down to about 1.8TiB usable at size 3.

Does It Survive a Node Dying?

#

A storage cluster you haven’t failure-tested is a story, not a fact. I ran a proper single-node failure test on live data, about 51GiB across 13,000 objects, on two separate nodes, using the same procedure each time.

First, evacuate. Proxmox VE HA node-maintenance mode live-migrates every HA-managed VM off the target node in 20 to 30 seconds, and unlike a manual migration, it refuses to let guests land back on that node until maintenance is explicitly cleared.

Then reboot with the cluster live. The node’s OSD and monitor drop, Ceph goes HEALTH_WARN, and the PGs settle into active+undersized+degraded, running on two of three replicas with min_size 2 holding the line. A rados put, get, and rm round trip succeeds while degraded: read and write continuity with a whole host down.

Finally, watch it come home. The node returns in under half a minute with pcie_aspm=off active and every Thunderbolt IP auto-restored, including the peers’ facing links, the ifup --force fix earning its keep. The OSD and monitor rejoin, Ceph heals itself back to HEALTH_OK, and the PGs return to active+clean.

One detail worth calling out: HA maintenance evacuation is temporary. When each node cleared maintenance, its guests migrated back home on their own, even with the cluster’s dynamic rebalancer pinned to static for the duration of the maintenance window.

That is the part that actually mattered. The numbers below are just for the record.

The Numbers, For What They’re Worth

#

I ran a single client rados bench against the pool once things settled, and I want to be straight about what it shows rather than dress it up.

None of that is spectacular. 325 IOPS and 49ms of average latency on a replicated write is not what a link advertised at 26Gb/s sounds like it should deliver, and the random read number is modest for storage that is entirely NVMe underneath. Some of that is the cost of 3x replication riding the same fabric as everything else, some of it is a single client bench with nothing tuned to chase IOPS specifically. I did not build this to win a benchmark. I built it so the storage keeps working when a node goes down, which the previous section already proved, and which is the part I actually care about.

Worth the Detour

#

Three cables, three NUCs, about 1.3 GB/s of replicated storage that survives a node reboot without dropping a write. The BIOS update should have been step one, not something I arrived at after ruling out cabling, power management, and ASPM one at a time. If there’s a lesson here beyond the specific fixes, it’s that on brand new small form factor hardware, stale firmware presents as almost anything except a firmware problem. Everything else, the numbered addressing scheme, ifup --force, pcie_aspm=off, keeping Corosync off the storage fabric entirely, is exactly the kind of detail that only shows up once you actually fail-test the thing instead of trusting it because it came up green once.