Are these numbers actually good?#
The original Ceph-over-Thunderbolt post ended with a small benchmark table and a fairly modest verdict on it: not spectacular for a link rated at 26Gb/s. I wasn’t exactly selling it. That was a fair assessment at the time, but it was measured on a mostly empty pool right after the build, 51GiB and 13,000 objects, with nothing else on the pool competing for I/O. The fabric itself is always dedicated to Ceph, that part never changes. A few days later, with real VMs actually living on this storage and a fresh reason to hammer it, it seemed worth asking the more interesting question. Are these numbers actually good, or just good for a novelty setup?
Why now#
Two of the cluster’s three NVMe OSD drives had just been through a multi-day saga with an NVMe power-state bug: the drive controllers dropping into a low-power state and failing to wake back up, taking down OSDs and needing a physical power cycle every time. I wrote up the full incident separately: An NVMe Power-State Bug Kept Crashing My Ceph Cluster. The fix was a set of kernel boot parameters disabling the drives’ power-state negotiation, and so far it seemed to be holding. The natural next move was to load the cluster back up, moving VMs back onto it and then genuinely stressing it. Validating the fix under real I/O pressure and finding out how fast this thing actually is turned out to be the same experiment. Two birds, one very stressed cluster.
The numbers#
Same tools as before, rados bench plus Ceph’s own built-in per-OSD bench, same pool (ceph-vm, replicated size=3, min_size=2), now carrying real VM data and freshly stress-tested rather than freshly built.
| Workload | Throughput | IOPS | Avg latency |
|---|---|---|---|
| Per-OSD bench (single OSD, 4MB blocks) | 2.1 - 4.2 GB/s (varies by node) | 510 - 999 | – |
| Pool write (4MB blocks, 16 threads) | 1.67 GB/s | 416 | 38 ms |
| Pool sequential read | 3.21 GB/s | 802 | 19 ms |
| Pool random read | 3.29 GB/s | 822 | 19 ms |
| Small-block write (4K, 32 threads) | 17 MB/s | 4,382 (max 4,818) | 7.3 ms |
For reference, the original post’s numbers, on the same fabric right after the build:
| Then | Now | |
|---|---|---|
| Write | 1.30 GB/s | 1.67 GB/s |
| Sequential read | 2.03 GB/s | 3.21 GB/s |
| Random read | 2.55 GB/s | 3.29 GB/s |
Faster now than right after the build, which is a relief I wasn’t sure I’d get. That’s a good sign in itself, though some of the difference is probably just normal run-to-run variance. None of it points to any degradation.
Is that actually good?#
As with everything in life, it depends on what you’re comparing it to.
Reads are hitting the fabric’s ceiling. The Thunderbolt ring is rated at roughly 26Gb/s, about 3.25GB/s usable. That figure is only a rough estimate. Some of the link’s full 40Gb/s rating is reserved for DisplayPort tunneling whether it’s actually used or not, and protocol overhead eats further into what’s left for a data tunnel like this one. 3.21 to 3.29GB/s lands right around that estimate either way, close enough that the small difference is just measurement noise. A read only costs one network hop, client to whichever OSD holds the data, so this is about as good as this physical link gets.
Writes are about half that, and that’s the replication tax at work. With size=3, a healthy write has to reach the primary OSD and get acknowledged by both secondary replicas, all three copies, before it’s durable, more network round trips than a read, over the same link. min_size=2 is just a floor: it’s what still lets writes proceed if one of the three OSDs is down. It isn’t what this cluster is actually waiting on in the normal, healthy case. Landing at roughly 51% of the read ceiling for triple-replicated writes is a normal, healthy ratio. There’s no bottleneck here worth chasing.
The IOPS number is solid for a distributed system, and a long way from what the drives themselves can do. The OSD drives (Crucial P510, PCIe Gen 5) are individually capable of far higher IOPS and latency well under 100 microseconds. Getting 7.3ms average latency through the full stack says this is bound by network round trip, replication acknowledgment, and Ceph’s own software path. The media itself isn’t the bottleneck. That’s true of any replicated network storage compared to raw local disk. This isn’t a knock on the drives’ raw throughput, or on Thunderbolt. The firmware bug covered in the other post is real, it’s just a different failure mode than raw throughput.
Now to the comparison that actually matters. Ceph over 1GbE or 2.5GbE tops out around 110 to 280 MB/s. This setup is doing 1.67 to 3.29GB/s, roughly 6 to 12 times that, using three $40 Thunderbolt cables instead of a considerably more expensive 25/40GbE switch and matching NICs. That’s the real story here: homelab-budget hardware performing at a bandwidth tier that would otherwise cost real money to reach, a 25/40GbE switch and matching NICs instead of three cheap cables. It doesn’t beat an enterprise SAN. Comparing the two would be bringing a knife to a gunfight, and I already know which one’s the knife.
The bonus find#
Cleaning up after the benchmark turned up something unrelated but worth a footnote. The pool had roughly 368GiB of stale leftover benchmark data sitting in it from whatever testing happened during the original build, never cleaned up, quietly consuming replicated capacity the whole time. Whoops. rados cleanup doesn’t run itself. If you benchmark a pool, rados -p <pool> cleanup afterward is a required step, skip it and the test leaves a slow leak in your usable capacity behind.
Good enough, honestly#
Sustained multi-gigabyte-per-second I/O across all three nodes, for the whole length of this benchmark, produced zero recurrence of the NVMe fault. Honestly, the load itself is a weaker test than it sounds: the bug is triggered by a drive going idle and failing to wake back up, and a drive getting hammered with I/O doesn’t get much chance to go idle in the first place. The part that actually matters came right after, once the load dropped back down and the drives had normal idle gaps again: the fault didn’t come back. That’s the condition this bug actually needs to show up in, and it stayed clean. The more meaningful test going forward is still the ordinary, gappy production load the cluster has been carrying since the fix went in, without a single recurrence. I’m still watching for it the same way I said I would in the incident post.
It also answered the question this post opened with: these numbers hold up under real contention, novelty setup or not. 6 to 12 times what Ceph over 1GbE or 2.5GbE gets you is still the real headline here. A comparison to an enterprise SAN, or NFS, or a production-scale vSAN or Ceph deployment, was never the point: this is three NUCs and three NVMe drives over Thunderbolt in a home lab. A real production environment would be scaled and architected completely differently.
As long as the firmware issue really does stay behind me, this cheap Thunderbolt-based Ceph setup holds up fine for exactly what I need it to do.
