The Zen of Parallel Programming: The Posture of a Kernel

Summary

As the journey through parallel programming continues, I shift my attention toward GPU kernels, and what better place to continue than the HipKittens: Fast and Furious AMD Kernels paper. A GPU kernel is a specialized function launched many times across the GPU. Each invocation follows the same general operation while receiving its own position and portion of the data. The kernel itself may appear simple, but its execution unfolds through threads, wavefronts, thread blocks, tiles, registers, memory, and the architecture responsible for holding all of that activity together. On the AMD hardware studied in the paper, threads are grouped into 64-thread waves, while multiple waves are scheduled together inside thread blocks on the GPU’s compute units. The hard truth is that it depends upon the threads that express it, the data that gives it something to transform, and the hardware that determines how that transformation can physically occur. Even when the same tile-based abstraction moves from NVIDIA to AMD, HipKittens finds that the algorithms used to express that abstraction must be redesigned for AMD’s architecture. The abstraction may travel, but it cannot remain untouched by the conditions receiving it. One of the paper’s central contributions is an eight-wave ping-pong schedule designed to overlap computation with memory movement. Eight waves are placed inside a thread block, with two waves residing on each of four SIMD units. Within each pair, one wave performs matrix computation while the other prefetches the data needed for the next stage. They then exchange roles, moving back and forth between computation and memory rather than remaining permanently attached to either identity.This brought me back to the “Posture” chapter of Zen Mind, Beginner’s Mind, where Suzuki describes us as both independent and dependent, “not two, and not one.” The paired waves remain distinguishable. One is computing while the other is moving memory, and each carries its own temporary r...