Clojure on Fennel part three: parsing

Summary

The two previous posts were not related to the compiler itself, but were kicked off by the start of the compiler development. I’d say this project was the reason that I made proper immutable data structures for Fennel and Lua. But now we’re finally going to talk about the compiler itself! And we’ll start with the parsing stage - transforming Clojure code into something that can be operated on by the compiler. Manual single-pass parser and compiler When I started this project, as I usually do, I underestimated the complexity at hand. My first attempt was a single-pass compiler that accepted a stream of characters of Clojure code and re-compiled it on the fly. My idea was to write a simple recursive-descent parser that, when accepting a character would decide how to translate it to Fennel, assuming my fennel-cljlib library handled most of the semantics. So, for example, when encountering [ it would translate it to (cljlib.vector and thus [1 2 3] would descend to compile-vector and expand to (cljlib.vector 1 2 3). The idea is simple, and at first I thought that it would suffice, but then I remembered that there are a lot of other things in Clojure that I have to consider. For example #_ - the ignore form syntax, and ^foo - the metadata syntax. This complicated things a lot, so I added initial support for them and moved on - because I decided that this will be a bootstrap compiler for a proper Clojure parser. Clojure parsers After some time I had a compiler that could compile most of the arbitrary code I threw at it. Of course, it was not exactly to the Clojure spec, but it worked and I was going to replace it anyway. So I started looking at available Clojure parsers written in Clojure. Edamame I started with Edamame. The choice was based on the fact that the Squint project uses it and my project is similar to Squint, or so I thought. This parser is also used in SCI - a Clojure interpreter, so it seemed like a good fit. The first thing I did was make sure that Edamame a...