We should get rid of average CPU utilization

Summary

Why we should get rid of average CPU utilization2026-05-14 · Based on a talk at Cloud Native Aachen Meetup, April 2026A Go function in our application kept getting cancelled in production.The function had a tight timeout. The same code ran fine in our development setups, in our CI and CD pipelines, in every integration test we had. In production it would sometimes blow past the timeout and die with context deadline exceeded.What made it worse was the state machine library we used. When its context got cancelled, it wouldn’t recover on its own. It would crash and hang. We couldn’t reproduce it.When we talked to users, they reported the CPU utilization looked fine.It took us weeks to find the cause.Why average CPU isn’t enoughOne of the first things you learn when you work with computers is: when it gets slow, you open the task manager (or the equivalent) and check the CPU. If it’s high, you look at what the process is, and you stop it or you work on it, and you continue.On Linux servers you have top, htop, whatever, with a lot of other numbers on the screen. At least for me, when I was working with it, I always just looked at the average CPU. Then when you continue with your career, you provision VMs. Maybe on VMware or Hyper-V on-prem. Maybe on AWS, Azure, or Hetzner. You pick a number of vCPUs. If you look closer, it says something like “performance” or “dedicated” as a more expensive option. But at least I myself never really asked why. It was just cheaper without.The instinct, taught to you by every tool, by every vendor, by every dashboard, fails here.Every tool shows you average CPU utilization, but none of the tools help you interpret it. None of them tell you that CPU utilization isn’t linear to how much capacity you have. They just show you a percentage. The jump from 80% to 81% CPU utilization adds roughly 20× more wait time than the jump from 10% to 11%. So even with 20% “headroom” at 80% utilization, latencies have already started climbing:CPU utilization...