Building the Grace Cathedral Experience

Summary

Grace Cathedral is a San Francisco landmark on Nob Hill. After seeing Vincent Woo's high-quality capture of the cathedral and its surrounding streets as 3D Gaussian splats, we jumped at the chance to collaborate with him. Together, we turned his splats into a browser-based, interactive experience. You can see the finished app below: This article explores how I built this app. Let's dive in. Under the Hood​ The PlayCanvas Engine was the perfect runtime on which to build, thanks to its WebGPU hybrid renderer and streamed SOG format. WebGPU Hybrid Renderer​ The exterior view renders around 3.5 million Gaussian splats at a time. Because they must be sorted back to front, moving the camera means repeatedly sorting millions of splats in real time. Our first-generation Gaussian splat renderer used WebGL 2, which only provides vertex and fragment shaders. They are well suited to drawing the splats but not to sorting them, so a worker sorted the Gaussians on the CPU and uploaded the results to the GPU. Sorting and uploading could fall behind rendering, making artifacts visible as the camera moved. WebGPU lets us divide that work more effectively. Compute shaders cull, project and sort the active splat set on the GPU every frame, then vertex and fragment shaders render the result. Keeping the data on the GPU avoids the transfer bottleneck and makes the pipeline dramatically faster than the worker-based WebGL 2 renderer. The experience uses this WebGPU pipeline whenever the browser supports it and falls back to the WebGL 2 renderer automatically. To support both rendering paths, the cutout, flag animation and transition shaders each have WGSL and GLSL implementations so the experience looks the same on both backends. Streamed SOG​ The splat data is delivered as streamed SOG — our open compressed splat format, extended with chunked levels of detail that the engine fetches on demand. The experience contains separate splat sets for the exterior and interior. The exterior combines...