Noroboto: Lying fonts and mitigation in Rust

Summary

The most exciting phrase to hear in science, the one that heralds new discoveries, is not "Eureka!" but "That's funny..." What if your font is lying to your AI? Discovery Tritium has recently been under consideration from a number of "AI native" law firms for use in their legal tech pipelines. Most of these firms use web technologies for their UI and want Tritium as a part of that front-end. We've long used PDFium to render PDFs, but it requires a C binary and someone pointed out to us following a recent blog post that the hayro crate is getting good at PDF rendering. We agree. For a lot of reasons (e.g., multi-threading), we decided to switch. Switching required a new row segmentation algorithm. I had a flight to the US from London, and two hours in I was making great progress. I hit a bug. The new algorithm seemed, for some reason, to refuse to match a random character in a manner that broke our existing tests. In the above GIF, we try to select, copy and paste a portion of "The Art of War" only to have an arbitrary space land in the middle with our new hayro implementation. But, then, hmm, I noticed... PDFium seems to do it, too. Because we were now using a Rust crate rather than a C library binding, it was simple to step through the code in the VS Code debugger to see what was going wrong with the two "t" glyphs. Turns out, it's a double-t "tt" non-Unicode value! Our hayro Device implementation treats it as a non-breaking-space character. But, PDFium also just disregards it? ... "That's funny." ... I fixed our test, and it hit me. LegalTech's Mythos Moment Modern legal tech stacks in 2026 are Rube Goldberg machines of open-source and proprietary products from Word to LibreOffice, to python-docx and PDFium, to tesseract, node.js and dozens of UI libraries like SuperDoc, PDF.js and Office.js. Through those pipelines are pushed artifacts of decades-old written specifications which span tens of thousands of pages. In addition to the venerated OSS parts of these stac...