Principles of Mechanical Sympathy

Summary

Over the past decade, hardware has seen tremendous advances, from unified memory that's redefined how consumer GPUs work, to neural engines that can run billion-parameter AI models on a laptop. And yet, software is still slow, from seconds-long cold starts for simple serverless functions, to hours-long ETL pipelines that merely transform CSV files into rows in a database. Back in 2011, a high-frequency trading engineer named Martin Thompson noticed these issues, attributing them to a lack of Mechanical Sympathy. He borrowed this phrase from a Formula 1 champion: You don't need to be an engineer to be a racing driver, but you do need Mechanical Sympathy. -- Sir Jackie Stewart, Formula 1 World Champion Although we're not (usually) driving race cars, this idea applies to software practitioners. By having “sympathy” for the hardware our software runs on, we can create surprisingly performant systems. The mechanically-sympathetic LMAX Architecture processes millions of events per second on a single Java thread. Inspired by Martin's work, I've spent the past decade creating performance-sensitive systems, from AI inference platforms serving millions of products at Wayfair, to novel binary encodings that outperform Protocol Buffers. In this article, I cover the principles of mechanical sympathy I use every day to create systems like these - principles that can be applied most anywhere, at any scale. Not-So-Random Memory Access Mechanical sympathy starts with understanding how CPUs store, access, and share memory. Figure 1: An abstract diagram of how CPU memory is organized Most modern CPUs - from Intel's chips to Apple's silicon - organize memory into a hierarchy of registers, buffers, and caches, each with different access latencies: Each CPU core has its own high-speed registers and buffers which are used for storing things like local variables and in-flight instructions. Each CPU core has its own Level 1 (L1) Cache which is much larger than the core's registers and buffe...