Caliper: Right-size your CI runners

Summary

Caliper: Right-size your CI runnersJanuary 16, 2026The problem: CI runners are a black boxHow do you know if you're overpaying for CI runners? Is it actually more expensive to run longer on a smaller runner than run shorter on a larger one? You pick a runner size sort of randomly, builds run, and you pay the bill. But is a 32-core runner actually faster than a 16-core one for your builds? Does more RAM help? Without data, you have to just make your best guess.We built Caliper to answer these questions with actual measurements.What Caliper doesCaliper is a CLI tool that benchmarks your build commands across different CPU/RAM configurations. It uses Docker containers with resource limits to simulate different runner sizes, runs multiple iterations with a warm-up run, and calculates build time statistics: mean, median, standard deviation, P90, P95, and success rate.The key feature is matrix mode: give Caliper a list of CPU and RAM values, and it will test every combination automatically and provide stats.Real results: Benchmarking InfluxDBWe benchmarked the InfluxDB Rust build (cargo clean && cargo build) across 25 configurations on a Hetzner AX162-R dedicated server, 10 runs per configuration:CPUsRAMMeanMedianStd DevMinMaxSuccess28 GB6m2s6m2s152ms6m1s6m2s100%216 GB6m0s6m1s142ms6m0s6m1s100%232 GB6m1s6m1s545ms5m59s6m1s100%264 GB6m0s6m0s184ms6m0s6m0s100%2128 GB6m1s6m2s637ms6m0s6m2s100%48 GB3m30s3m30s601ms3m29s3m31s100%416 GB3m28s3m28s684ms3m27s3m29s100%432 GB3m29s3m29s572ms3m28s3m30s100%464 GB3m29s3m30s966ms3m28s3m30s100%4128 GB3m29s3m29s861ms3m28s3m30s100%88 GB2m41s2m41s1.2s2m38s2m43s100%816 GB2m39s2m40s2.0s2m36s2m41s100%832 GB2m40s2m40s1.4s2m37s2m42s100%864 GB2m39s2m41s3.5s2m33s2m42s100%8128 GB2m41s2m41s2.2s2m34s2m42s100%168 GB2m14s2m14s829ms2m13s2m15s100%1616 GB2m13s2m12s901ms2m11s2m15s100%1632 GB2m12s2m12s499ms2m11s2m13s100%1664 GB2m13s2m14s761ms2m12s2m15s100%16128 GB2m13s2m13s800ms2m12s2m14s100%328 GB2m12s2m12s831ms2m11s2m13s100%3216 GB2m11s2m11s1.0s2m9s2m12s100%323...