Designing a Passively Safe API

Summary

I'm in the process of migrating Augno's monolithic API to a microservices architecture. It's been slow, largely because we're making every public endpoint passively safe. A passively safe system is one that is designed to fail gracefully. Crumple zones in cars, seismic zones in buildings, and gravity-driven cooling systems in nuclear reactors are all examples of passively safe designs. In APIs, passively safe means failures (crashes, timeouts, retries, partial outages) can't produce duplicate work, surprise side effects, or unrecoverable state. After any failure, the system either (a) completes the workflow exactly once, or (b) lands in a terminal, explicitly visible state that won't double-bill or duplicate work. Consider an endpoint that has the following characteristics: It must call an external API in-band and cause side effects. It must perform asynchronous work in response to the request. It must create a new resource and update several related resources across multiple services. Clients must be able to retry without creating duplicates or extra charges. Any failure at the wrong moment could leave the system in an unrecoverable state. So, how can we make such an endpoint passively safe? Let's think through a deliberately gnarly example and see if we can come up with a solution. Imagine an API endpoint that allows users to ship an order of goods: POST /shipments. There are many things that must happen to create this shipment. A third-party API must validate the shipping address. A third-party API must generate tracking information and labels. The shipment and shipping cases records must be updated with tracking information. An invoice record must be generated against the shipment record. The order record must be either marked as fulfilled or partially fulfilled. The customer, sales representative, and other interested parties must be notified of the shipment. The most straightforward approach would be to implement this endpoint in a monolith. Each step would be...