Over the past month and a half I’ve been rolling out a new OCI registry for our CI: Harbor as the registry, Garage as the S3 storage behind it, on Kubernetes. It was quite the journey - a couple of incidents, a lot of measuring, and a pile of lessons about what actually limits a registry under CI load.

This is part 1: why we built it, the design, and the things that broke before it even carried real traffic. Part 2 will be the throughput hunt (every layer we fixed, and the surprising number that didn’t improve). Part 3 is the storage war - including the weekend I already wrote about in the fstrim post.

For the past month or so I’ve been rolling out a scalable, high-throughput OCI registry for our CI: Harbor as the registry, Garage as the S3 storage behind it, on Kubernetes with Longhorn providing the volumes (that has since changed, but it’s the setup this story happened on). That journey is worth a few posts on its own (and I plan to write them, at some point), but this one is about a single thing that caught me off guard on a Saturday morning: deleting data does not free disk space. Not on Longhorn, and not on most thin-provisioned storage - until something explicitly tells the layer below which blocks are no longer in use.

I’ve been spending a bit more time lately - whatever little spare time I have outside work and family - on a few open source projects. One of them is persistent-queue: a durable, at-least-once, MPSC queue.

I wanted to take some time to explain the story behind it.

For the past few years I’ve been working on very similar systems, though in different industries: telemetry collection and ingestion from edge devices. And every single time, the same scenarios came up:

Our Rust CI got slow. Not “grab a coffee” slow - “go do something else and hope it’s green when you come back” slow. This is the story of how I got most of that time back, and the boring lesson underneath it: measure before you optimize.

Why a slow pipeline actually hurts

For a while I didn’t care that the Rust jobs were slow. There weren’t many contributions, and the pipeline was fast enough for the pace we shipped at. Then that changed - a lot more code started flowing into the workspace, a good chunk of it AI-generated, and the pipeline that was “fine” suddenly wasn’t.

I had a Rust service that ate memory for breakfast. The fix that helped the most, I stumbled into completely by accident - and it had nothing to do with the code I’d been sweating over 😅 .

Here’s the whole embarrassing, hopefully useful story.

The setup

The service was a data collector - high-throughput ingest with backpressure, checkpointing, retries, the usual. The design was simple: one thread reads batches of records off a source on a tick, hands them to another thread over a channel, and that thread ships them to a destination.

A confession to start: a big chunk of the work I used to do by hand is now done by an AI. Not all of it, but a lot - I’d guess 60 to 75% of what my job used to be day-to-day. Including, full disclosure, this very post: an AI interviewed me, I brain-dumped the answers, it did the writing - correcting and reshaping as it went - and I reviewed it and hit publish 😅 .

I’d been writing Go for 5+ years. I didn’t go looking for Rust - it got dragged into my world, and honestly, I was defensive about it at first.

How I got dragged in

Back at Transcelestial, a colleague of mine, was building a tool to flash Raspberry Pis using an A/B partition setup (partition A is active, B is passive; a failed boot falls back to the partition that still works). He wrote it in Rust.

It’s been a while. A long while 😅 .

The last time I wrote here I opened with almost the same line - back then the excuse was becoming a father. This time it’s bigger (well, nothing’s really bigger than becoming a father) this: I packed up my life in Singapore and moved to Tokyo. That was October 2024, so this update is running about two years late 😬 .

So here’s the catch-up: why I left, what the move actually took (spoiler: nothing like what you see on YouTube), and what I’m building now.

If you run CI on cloud runners while your git server lives somewhere else - on-prem, another region, the far end of a slow VPN - you have probably watched a lot of time (and egress) disappear into cloning the same repositories over and over.

The setup I kept running into: an on-prem GitLab, an elastic fleet of runners autoscaling in the cloud, and a handful of repos in the tens of gigabytes. Multiply that by hundreds or thousands of jobs a day and every single one does a fresh clone across the WAN. The link is the bottleneck, the egress is not free, and the runners end up spending more time fetching code than running it.

The chan type in Go is a mechanism which can be used to send or receive data from one function to another. E.g:

func main() {
    c := make(chan string)
    go printHello(c)
    sayHello(c)
}

func sayHello(msgChan chan<- string) {
    msgChan <- "hello"
}

func printHello(msgChan <-chan string) {
    fmt.Println(<-msgChan)
}

Channels can be bidirectional (chan string) or directional (chan <- string sender/<-chan string receiver). And they operate like FIFO queues.