Retry Storm Lab

Summary

A retry is not free. See when your recovery policy becomes the outage. Retry Storm Lab is a deterministic Python simulator for partial dependency failures. It models fresh requests and retries competing for the same capacity, then compares immediate, fixed, exponential, and full-jitter backoff policies. Retries improve reliability when failures are brief and spare capacity exists. During a sustained partial outage, synchronized clients can instead amplify demand, starve fresh requests, and keep a recovering service overloaded. The simulator makes that feedback loop visible: Generate a steady stream of fresh requests. Reduce successful capacity during a configurable failure window. Reschedule failed and over-capacity attempts using the selected policy. Measure amplification, peak demand, abandoned requests, and recovery time. Compare all four policies against the exact same scenario and random seed. This is an intentionally small queueing model, not a production capacity planner. It omits network latency distributions, client timeouts, autoscaling, circuit breakers, and correlated downstream failures. Its job is to make retry dynamics easy to inspect and discuss. Python 3.11 or newer is required. python -m venv .venv source .venv/bin/activate pip install -e . streamlit run app.py Open http://localhost:8501, choose a preset, and adjust the workload from the sidebar. retry-storm-lab --preset "Rate-limit spiral" --compare retry-storm-lab --preset "Slow recovery" --policy full_jitter --json Example comparison: immediate amplification=1.11x peak=2720 rps success=97.2% recovery=1s fixed amplification=1.08x peak=1600 rps success=99.9% recovery=3s exponential amplification=1.07x peak=1600 rps success=100.0% recovery=3s full_jitter amplification=1.10x peak=1651 rps success=99.9% recovery=3s Exact values depend on the selected preset and seed. Use the simulation engine from retry_storm_lab import BackoffPolicy, SimulationConfig, simulate result = simulate( SimulationConfig( ba...