Scaling a Monolith to 1M LOC: 113 Pragmatic Lessons from Tech Lead to CTO

Summary

For context, I work in a Django/React (TS)/React Native codebase on a 20-person team at MorphMarket. I have 17 years experience with web development in other frameworks (Rails/Laravel/Express etc.) and similar ideas are applicable. Performance Page Counts Are a Major (but Surprising) Source of DB Performance Issues at Scale. Giving an accurate "total pages" count (or total "items" count) is a hard problem at scale. This is because the DB needs to count all matching rows—and, often, the queries used here select a great many related records and call many filters. This can potentially require millions of rows to be scanned. I don't have a full solution here. What we have done so far is automatically strip all "annotations" from paginators (extra data used to render items but not necessary for page counts). In some cases for very large tables, we use an estimated count paginator (which uses some DB stats instead of actual queries) instead of actually counting all rows. Lastly, for endpoints where the page count really matters, we use some systems to cache it. For example, if the page count is going to be less than the number on the page (or, more technically, per our "cache block," which is 1,000 items), then we instruct the paginator to not do an extra DB query since it can already infer the count of records by the fact of the block being incomplete. I could go on... Long Cron-Job Reads Can Cripple Your System. A single inefficient, long-running DB query (often in a cron job for reporting) can halve the performance of your entire system if they share a DB. Moreover, queries happening outside the web process are often not as effectively tracked in APMs like New Relic, so they can be invisible. The solution here is to treat the performance of these queries as something worth worrying about... or create a read-only follower of your DB for these purposes so as to insulate your "hot" DB serving web traffic. More generally, long queries OR large payloads are the chief troubl...