Query cycles: A compiler murder mystery

Summary

On the Ferrocene team, we deal with a lot of tricky problems, as Aïssata Maïga describes in her recent RustWeek talk, The trials of qualifying Rust. This post dives into a particularly thorny problem that crashed my computer, took me over a week to debug, and ended up being 3 different bugs combining into one very unhelpful error. It would be nice to have an error message … Ferrocene regularly merges in changes from the upstream Rust compiler. A few months ago, we encountered the following error in CI after a merge: test [ui] tests/ui/delegation/bad-resolve.rs has been running for a long time [ui] tests/ui/delegation/bad-resolve.rs ... F test [ui] tests/ui/delegation/body-identity-glob.rs has been running for a long time Too long with no output (exceeded 20m0s): context deadline exceeded I took a look at this on a local build of the compiler. I do a lot of my work on "devpods": these are similar to the dev-desktops of the upstream Rust project, but hosted on Ferrocene's private infrastructure, and with custom Docker images that set up a lot of the build requirements beforehand. Unlike dev-desktops, they're Kubernetes pods, not full virtual machines. I like them because they're much faster than my local MacBook Air, and also can run x86 natively rather than through emulation. I started the test without a problem, wandered away for lunch, and came back to this error: ci@devpod-pr-rotation-0 ~/ferrocene (automation/pull-upstream-HEAD/mscy1uah)> Connection to bastion.devpods.infra.ferrous-systems.net closed by remote host. client_loop: send disconnect: Broken pipe Connection to devpod-pr-rotation-jynn.ferrocene-devpods.svc.cluster.local closed by remote host. Connection to devpod-pr-rotation-jynn.ferrocene-devpods.svc.cluster.local closed. Well, that's odd. I tried to reconnect: $ ssh devpod-pr-rotation-jynn.ferrocene-devpods.svc.cluster.local -J bastion.devpods.infra.ferrous-systems.net channel 0: open failed: connect failed: Connection refused stdio forwarding failed ...