When Vectorized Arrays Aren't Enough

Summary

Or, Zipped Arrays Rule Everything Around Me (see followup post) I've been contributing on and off to the McFACTS simulation project for a few months now, and am currently in the process of testing and merging some pretty substantial optimizations to the core 'business logic' of the simulation, which mainly consists of long series of NumPy array operations. Pretty much all the optimizations in the pipeline right now are basically identical, to the point that I've considered just writing a macro to write the code for me. I figured it would be worth condensing what I've learned in the last few months about how these optimizations work concretely, as well as the intuition behind them. None of this is particularly arcane or hardware specific, and while we will be getting into some x86s assembly at the end, it's mainly illustrative and helpful for validating the intuitions we're building. This may be especially valuable if you're an author of scientific Python code, and you're struggling with slow-running programs despite being told in undergrad that NumPy arrays are really fast. But first, a tortured metaphor: Alice, Bob, and Carol are logging engineers in the forests of British Columbia. Day in and day out, they operate the TreeChopper 30000, a industrial miracle that renders raw material into construction-ready timber on the spot. However, there is one downside: the TreeChopper is large, heavy, and very difficult to move. Alice presses a button, watches the TreeChopper rip apart a tree in five seconds flat, and then spends half an hour moving it to the next felled trunk. Ratio of effective work to transit: 1:360. Bob presses the same button, and then hops into a truck to drag the next trunk over to the TreeChopper, taking just 5 minutes. Ratio: 1:60. Carol has set up a parallel trucking operation that brings newly felled trunks continuously to the TreeChopper, piling up a backlog nearby, with just ten seconds of downtime between trunks. Ratio: 1:2. In a laboratory in T...