Why a slow pipeline actually hurts

For a while I didn’t care that the Rust jobs were slow. There weren’t many contributions, and the pipeline was fast enough for the pace we shipped at. Then that changed - a lot more code started flowing into the workspace, a good chunk of it AI-generated, and the pipeline that was “fine” suddenly wasn’t.

Here’s the whole embarrassing, hopefully useful story.

The setup

The service was a data collector - high-throughput ingest with backpressure, checkpointing, retries, the usual. The design was simple: one thread reads batches of records off a source on a tick, hands them to another thread over a channel, and that thread ships them to a destination.

How I got dragged in

Back at Transcelestial, a colleague of mine, was building a tool to flash Raspberry Pis using an A/B partition setup (partition A is active, B is passive; a failed boot falls back to the partition that still works). He wrote it in Rust.

The last time I wrote here I opened with almost the same line - back then the excuse was becoming a father. This time it’s bigger (well, nothing’s really bigger than becoming a father) this: I packed up my life in Singapore and moved to Tokyo. That was October 2024, so this update is running about two years late 😬 .

So here’s the catch-up: why I left, what the move actually took (spoiler: nothing like what you see on YouTube), and what I’m building now.

func main() {
    c := make(chan string)
    go printHello(c)
    sayHello(c)
}

func sayHello(msgChan chan<- string) {
    msgChan <- "hello"
}

func printHello(msgChan <-chan string) {
    fmt.Println(<-msgChan)
}

Channels can be bidirectional (chan string) or directional (chan <- string sender/<-chan string receiver). And they operate like FIFO queues.

To add some context, I was working on a simple app, which is similar to a router dashboard, for one of the products the company I work (Transcelestial) for is developing. And one of the main features of it was realtime metrics from the device.

As described in the prev post, I could have went with either WebSockets or the Streams API (long polling was not an option because we deal with modern browsers), but I chose to go with the streams API, because there was no need for bidirectional comm b/t the client and the server and all the APIs were already in place, so it was easier to just add the extra handler to stream the data.

I’ve noticed some time in November that my blog was throwing a 500 and I just spent maybe 5 minutes to check the logs on the VM where it was running (GCP) to see why. Everything seemed ok (nginx was up and ghost showed no issues either).

I’ve also updated the system, firmware, apps, etc to the latest version:

sudo apt update
sudo apt full-upgrade
sudo rpi-update

Install Go

We need Go to install runc and containerd.