Oban, the job processing framework from Elixir, has come to Python

Summary

Setting the Stage# I’ve used Oban in Elixir for almost as long as I’ve been writing software in Elixir, and it has always been an essential tool for processing jobs. I always knew Oban was cool, but I never dug deeper. This article is a collection of my notes and observations on how the Python implementation of Oban works and what I’ve learned while exploring its codebase. I’ll also try to compare it with the Elixir version and talk about concurrency in general. Surface Level# Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back. Additionally, like most job processing frameworks, Oban has queues with local and global queue limits. But unlike others, it stores your completed jobs and can even keep their results if needed. It has built-in cron scheduling and many more features to control how your jobs are processed. Oban comes in two versions - Open Source Oban-py and commercial Oban-py-pro. OSS Oban has a few limitations, which are automatically lifted in the Pro version: Single-threaded asyncio execution - concurrent but not truly parallel, so CPU-bound jobs block the event loop. No bulk inserts - each job is inserted individually. No bulk acknowledgements - each job completion is persisted individually. Inaccurate rescues - jobs that are long-running might get rescued even if the producer is still alive. Pro version uses smarter heartbeats to track producer liveness. In addition, Oban-py-pro comes with a few extra features you’d configure separately, like workflows, relay, unique jobs, and smart concurrency. OSS Oban-py is a great start for your hobby project, or if you’d want to evaluate Oban philosophy itself, but for any bigger scale - I’d go with Oban Pro. The pricing seems very compelling, considering the amount of work put into making the above features work. I obviously can’t walk you th...