peon/docs/design.md

Peon design scratchpad

This is just a random doc I made to keep track of all the design changes I have in mind for Peon: with this being my first serious attempt at making a programming language that's actually useful, I want to get the design right the first time (no one wants to make JavaScript 2.0, right? Right?).

The basic idea is:

• Some peon code comes in (from a file or as command-line input, doesn't matter)
• It gets tokenized and parsed into a typeless AST
• The compiler processes the typeless AST into a typed one
• The typed AST is passed to an optional optimizer module, which spits out another (potentially identical) typed AST representing the optimized program. The optimizer is always run even when optimizations are disabled, as it takes care of performing closure conversion and other cool stuff
• The typed AST is passed to a code generator module that is specific to every backend/platform, which actually takes care of producing the code that will then be executed

The current design is fairly modular and some parts of the codebase are more final than others: for example, the lexer and parser are more or less complete and unlikely to undergo massive changes in the future as opposed to the compiler which has been subject to many major refactoring steps as the project went along, but I digress.

The typed AST format should ideally be serializable to binary files so that I can slot in different optimizer/code generator modules written in different languages without the need to use FFI. The format will serve a similar purpose to the IR used by gcc (GIMPLE), but instead of being an RTL-like language it'll operate on a much higher level since we don't really need to support any other programming language other than peon itself (while gcc has to be interoperable with FORTRAN and other stuff).