A flaky test exposed a Redis client use-after-free

Summary

It can be hard to fix a bug even when it happens deterministically; replicating the circumstances that trigger it is often the hardest step. When a bug is non-deterministic this becomes harder: we need to execute the code that reproduces it many times in order to observe the behaviour, which adds significant time to our feedback loop. Harder still is when the cause of the bug happens prior to its manifestation —when data is corrupted but we don’t immediately notice it. Our usual tools focus on capturing the state of the system at the time a crash happens, which doesn’t tell us why the corruption happened in the first place, just that it happened. This is the tale of a memory corruption bug we found in the redis-client library and what helped us get there. Chapter 1: Bounce Jump! Bounce! Down! Up! Our story starts with David, an engineering manager on our Scalability team. One idle Tuesday, David saw a build of his had failed due to what appeared to be a number of flaky tests. He started a thread in Slack to alert others and included a screenshot from the Test Engine dashboard. As a continuous integration company, we are intimately familiar with flaky tests —they are an inevitable part of any large codebase. Within 15 minutes, the team pulled the Andon Cord; these tests were so flaky they were preventing us deploying code to production. As David summarised it, “[the] priority is getting main unblocked”, so we worked to exclude the tests from our suite temporarily. With the urgency reduced, David and the many other developers blocked by this hiccup were free to go back to their original tasks. David, though, decided to take a slightly deeper look by inspecting the reliability of these tests using Buildkite’s Test Engine reliability score, to see if he could correlate when they suddenly became unstable. It seems the three worst flakeys went from ~100% reliable to… not around [2-4 days prior]. I’ve had a look through merged PRs to see if I can spot what might have contr...