How AST-grep Rewrote Tree-sitter in Rust and Made It 30% Faster

Summary

How ast-grep Rewrote Tree-sitter in Rust and Made It 30% Faster ​Part 1 of 4 — the complete adventureVIDEOast-grep rewrote Tree-sitter's C core in Rust, with AI writing the code. The new core is faster at parsing, faster at reading the completed tree, and faster in ast-grep itself. (The title's “30%” is the parser-only number; end-to-end, ast-grep runs about 22% faster.)Source repository: HerringtonDarkholme/tree-sitter.Two quick introductions before the numbers. ast-grep — the structural code-search tool this blog belongs to — searches code by syntax rather than by text, so every file it touches must first become a syntax tree. Tree-sitter is the parser framework that builds that tree: you give it a grammar definition, and it generates a fast parser for that language. Born in the editor world, it now powers an enormous ecosystem of grammars and tools.Performance and peak RSS. Throughput is normalized so the unmodified C build (“C / normal”) scores 100; higher is better. RSS is peak resident memory, and the raw-parsing row shows it as a range because it varies across the benchmark's language fixtures. The outline row is ast-grep's real workload: parse every file in a repository, then walk each completed tree to extract a structural outline.BenchmarkC / normalRustDifferenceRaw parsingThroughput: 100RSS: 8.48–21.41 MiBThroughput: 129.74RSS: 8.42–25.70 MiB+29.74% throughput+20.0% RSS upper boundTree traversalThroughput: 100RSS: 20.38 MiBThroughput: 110.16RSS: 22.20 MiB+10.16% throughput+8.9% RSSComplete ast-grep outlineUser CPU: 1.233 sRSS: 26.52 MiBUser CPU: 0.960 sRSS: 34.43 MiB−22.2% user CPU+29.8% RSSRust won every parser and traversal fixture, and ast-grep produced exactly the same outline. Memory is the tradeoff: the Rust build uses about 8 MiB more in the ast-grep run. On the much larger TypeScript stress corpus — the TypeScript compiler repository's test-baseline tree, this project's memory torture test — it peaks at 91.2 MiB. That figure is a triumph, not a co...