Part 1 was the design and the rollout; part 2 was the throughput hunt. By early October the new registry was fast, the dashboards were green, and we started flipping real CI load onto it.

This part is the ten days that followed. Remember the storage stack diagram from part 1, with Garage sitting on Longhorn, on a VM virtual disk, on Proxmox’s network storage, on physical disks somewhere in a rack? This is the part where that stack screwed us 😅

In part 1 we set up a Harbor registry backed by Garage and survived the first week. This part is about making it fast - or more precisely, about finding out, one layer at a time, what “fast” was even limited by.

TLDR: we fixed four genuinely broken or suboptimal things - the proxy path, hardware crypto, the overlay network, NIC queues - and the single-stream pull number barely improved. The reason was in Garage’s documentation all along, and it turned out not to matter much. But the fixes were still worth it, for reasons that only became clear once we could measure each layer by itself.

Over the past month and a half I’ve been rolling out a new OCI registry for our CI: Harbor as the registry, Garage as the S3 storage behind it, on Kubernetes. It was quite the journey - a couple of incidents, a lot of measuring, and a pile of lessons about what actually limits a registry under CI load.

This is part 1: why we built it, the design, and the things that broke before it even carried real traffic. Part 2 will be the throughput hunt (every layer we fixed, and the surprising number that didn’t improve). Part 3 is the storage war - including the weekend I already wrote about in the fstrim post.

For the past month or so I’ve been rolling out a scalable, high-throughput OCI registry for our CI: Harbor as the registry, Garage as the S3 storage behind it, on Kubernetes with Longhorn providing the volumes (that has since changed, but it’s the setup this story happened on). That journey is worth a few posts on its own (and I plan to write them, at some point), but this one is about a single thing that caught me off guard on a Saturday morning: deleting data does not free disk space. Not on Longhorn, and not on most thin-provisioned storage - until something explicitly tells the layer below which blocks are no longer in use.

I’ve been spending a bit more time lately - whatever little spare time I have outside work and family - on a few open source projects. One of them is persistent-queue: a durable, at-least-once, MPSC queue.

I wanted to take some time to explain the story behind it.

For the past few years I’ve been working on very similar systems, though in different industries: telemetry collection and ingestion from edge devices. And every single time, the same scenarios came up:

Our Rust CI got slow. Not “grab a coffee” slow - “go do something else and hope it’s green when you come back” slow. This is the story of how I got most of that time back, and the boring lesson underneath it: measure before you optimize.

Why a slow pipeline actually hurts

For a while I didn’t care that the Rust jobs were slow. There weren’t many contributions, and the pipeline was fast enough for the pace we shipped at. Then that changed - a lot more code started flowing into the workspace, a good chunk of it AI-generated, and the pipeline that was “fine” suddenly wasn’t.

I had a Rust service that ate memory for breakfast. The fix that helped the most, I stumbled into completely by accident - and it had nothing to do with the code I’d been sweating over 😅 .

Here’s the whole embarrassing, hopefully useful story.

The setup

The service was a data collector - high-throughput ingest with backpressure, checkpointing, retries, the usual. The design was simple: one thread reads batches of records off a source on a tick, hands them to another thread over a channel, and that thread ships them to a destination.

A confession to start: a big chunk of the work I used to do by hand is now done by an AI. Not all of it, but a lot - I’d guess 60 to 75% of what my job used to be day-to-day. Including, full disclosure, this very post: an AI interviewed me, I brain-dumped the answers, it did the writing - correcting and reshaping as it went - and I reviewed it and hit publish 😅 .

I’d been writing Go for 5+ years. I didn’t go looking for Rust - it got dragged into my world, and honestly, I was defensive about it at first.

How I got dragged in

Back at Transcelestial, a colleague of mine, was building a tool to flash Raspberry Pis using an A/B partition setup (partition A is active, B is passive; a failed boot falls back to the partition that still works). He wrote it in Rust.

It’s been a while. A long while 😅 .

The last time I wrote here I opened with almost the same line - back then the excuse was becoming a father. This time it’s bigger (well, nothing’s really bigger than becoming a father) this: I packed up my life in Singapore and moved to Tokyo. That was October 2024, so this update is running about two years late 😬 .

So here’s the catch-up: why I left, what the move actually took (spoiler: nothing like what you see on YouTube), and what I’m building now.