More changes to the README because yes

.gitignore

@@ -2,6 +2,7 @@
 nimcache/
 nimblecache/
 htmldocs/
-*.pbc  # Peon bytecode files
+tests/*.pbc  # Peon bytecode files
-stdin.pbc
+*.pbc
-tests.pbc
+bin/
+.vscode/

Makefile

@@ -1,5 +1,5 @@
-run:
+repl:
-	nim --hints:off --warnings:off r src/test.nim
+	nim --hints:off --warnings:off r src/main.nim
 
 pretty:
 	nimpretty src/*.nim src/backend/*.nim src/frontend/*.nim src/frontend/meta/*.nim src/memory/*.nim src/util/*.nim

README.md

@@ -24,11 +24,17 @@ Peon is a simple, functional, async-first programming language with a focus on c
 
 ## Credits
 
-- Araq, for creating the amazing language that is [Nim](https://nim-lang.org)
+- Araq, for creating the amazing language that is [Nim](https://nim-lang.org) (as well as all of its contributors!)
+- Guido Van Rossum, aka the chad who created [Python](https://python.org) and its awesome community and resources
 - The Nim community and contributors, for making Nim what it is today
 - Bob Nystrom, for his amazing [book](https://craftinginterpreters.com) that inspired me
-    and taught me how to actually make a programming language
+    and taught me how to actually make a programming language (kinda, I'm still very dumb)
 - [Njsmith](https://vorpus.org/), for his awesome articles on structured concurrency
+- All the amazing people in the [r/ProgrammingLanguages](https://reddit.com/r/ProgrammingLanguages) subreddit and its [Discord](https://discord.gg/tuFCPmB7Un) server
+- [Art](https://git.nocturn9x.space/art) <3
+- Everyone to listened (and still listens to) me ramble about compilers, programming languages and the likes (and for giving me ideas and testing peon!)
+- ... More? (I'd thank the contributors but it's just me :P)
+- Me! I guess
 
 
 ## Project State
@@ -52,28 +58,52 @@ In no particular order, here's a list of stuff that's done/to do (might be incom
 
 Toolchain:
 
-  - Tokenizer (with dynamic symbol table) [x]
+  - Tokenizer (with dynamic symbol table) [X]
-  - Parser (with support for custom operators, even builtins) [x]
+  - Parser (with support for custom operators, even builtins) [X]
-  - Compiler [ ]  (Work in Progress)
+  - Compiler [ ]  -> Being written
-  - VM [ ] (Work in Progress)
+  - VM [ ] -> Being written
-  - Bytecode (de-)serializer [x]
+  - Bytecode (de-)serializer [X]
-  - Static code debugger [x]
+  - Static code debugger [X]
   - Runtime debugger/inspection tool [ ]
 
 Type system:
 
   - Custom types [ ]
-  - Intrinsics [x]
+  - Intrinsics [X]
-  - Generics [ ] (Work in Progress)
+  - Generics [ ] -> WIP
-  - Function calls [ ] (Work in Progress)
+  - Functions [X]
 
 Misc:
 
-  - Pragmas [ ] (Work in Progress)
+  - Pragmas [ ] -> WIP (Some pragmas implemented)
   - Attribute resolution [ ]
+  - method-like call syntax without actual methods (dispatched at compile-time) [ ]
   - ... More?
 
+
+## Features
+
+Aside from the obvious basics like exceptions, a true import system with namespaces and a standard library (duh), here's a 
+random list of high-level features I plan peon to have and that I think are kinda neat:
+- References not being nullable by default (must use `#pragma[nullable]`)
+- Easy C/Nim interop via FFI
+- C/C++ backend
+- Nim backend (maybe)
+- Structured concurrency
+- Capability-based programming (i.e. functions are passed objects to act on the real world)
+- Parametric Polymorphism (with untagged typed unions and maybe interfaces)
+- Simple OOP (with multiple dispatch!)
+- RTTI, with methods that dispatch at runtime based on the true type of a value (maybe)
+- Limited compile-time evaluation (embed the Peon VM in the C/C++/Nim backend and use that to execute peon code at compile time)
+
+
 ## The name
 
 The name for peon comes from my and [Productive2's](https://git.nocturn9x.space/prod2) genius and is a result of shortening
-the name of the fastest animal on earth: the **Pe**regrine Falc**on**. I guess I wanted this to mean peon will be blazing fast
+the name of the fastest animal on earth: the **Pe**regrine Falc**on**. I guess I wanted this to mean peon will be blazing fast
+
+# Peon needs you.
+
+No, but really. I need help. This project is huge and (IMHO) awesome, but there's a lot of non-trivial work to do and doing
+it with other people is just plain more fun and rewarding. If you want to get involved, definitely try [contacting](https://nocturn9x.space/contact) me
+or open an issue/PR!

docs/bytecode.md

@@ -16,25 +16,27 @@ A peon bytecode dump contains:
 - Debugging information
 - File and version metadata 
 
-## Encoding
+## File Headers
-
-### Header
 
 A peon bytecode file starts with the header, which is structured as follows:
 
 - The literal string `PEON_BYTECODE`
-- A 3-byte version number (the major, minor and patch versions of the compiler that generated the file as per the SemVer versioning standard)
+- A 3-byte version number (the major, minor and patch version numbers of the compiler that generated the file)
 - The branch name of the repository the compiler was built from, prepended with its length as a 1 byte integer
-- The full commit hash (encoded as a 40-byte hex-encoded string) in the aforementioned branch from which the compiler was built from (particularly useful in development builds)
+- The commit hash (encoded as a 40-byte hex-encoded string) in the aforementioned branch from which the compiler was built from (particularly useful in development builds)
 - An 8-byte UNIX timestamp (with Epoch 0 starting at 1/1/1970 12:00 AM) representing the exact date and time of when the file was generated
-- A 32-byte, hex-encoded SHA256 hash of the source file's content, used to track file changes
 
-### Line data section
+## Debug information
 
-The line data section contains information about each instruction in the code section and associates them
+The following segments contain extra information and metadata about the compiled bytecode to aid debugging, but they may be missing
+in release builds.
+
+### Line data segment
+
+The line data segment contains information about each instruction in the code segment and associates them
 1:1 with a line number in the original source file for easier debugging using run-length encoding. The section's 
 size is fixed and is encoded at the beginning as a sequence of 4 bytes (i.e. a single 32 bit integer). The data
-in this section can be decoded as explained in [this file](../src/frontend/meta/bytecode.nim#L28), which is quoted
+in this segment can be decoded as explained in [this file](../src/frontend/meta/bytecode.nim#L28), which is quoted
 below:
 ```
 [...]
@@ -54,19 +56,41 @@ below:
 [...]
 ```
 
-### Constant section
+### CFI segment
 
-The constant section contains all the read-only values that the code will need at runtime, such as hardcoded
+The CFI segment (where CFI stands for **C**all **F**rame **I**nformation), contains details about each function in
+the original file. The segment's size is fixed and is encoded at the beginning as a sequence of 4 bytes (i.e. a single 32 bit integer).
+The data in this segment can be decoded as explained in [this file](../src/frontend/meta/bytecode.nim#L41), which is quoted
+below:
+
+```
+[...]
+## cfi represents Call Frame Information and encodes the following information:
+## - Function name
+## - Argument count
+## - Function boundaries
+## The encoding for CFI data is the following:
+## - First, the position into the bytecode where the function begins is encoded (as a 3 byte integer)
+## - Second, the position into the bytecode where the function ends is encoded (as a 3 byte integer)
+## - After that follows the argument count as a 1 byte integer
+## - Lastly, the function's name (optional) is encoded in ASCII, prepended with
+##   its size as a 2-byte integer
+[...]
+```
+
+## Constant segment
+
+The constant segment contains all the read-only values that the code will need at runtime, such as hardcoded
 variable initializers or constant expressions. It is similar to the `.rodata` section of Assembly files, although
 the implementation is different. Constants are encoded as a linear sequence of bytes with no type information about
 them whatsoever: it is the code that, at runtime, loads each constant (whose type is determined at compile time) onto 
 the stack accordingly. For example, a 32 bit integer constant would be encoded as a sequence of 4 bytes, which would
-then be loaded by the appropriate `LoadInt32` instruction at runtime. The section's size is fixed and is encoded at
+then be loaded by the appropriate `LoadInt32` instruction at runtime. The segment's size is fixed and is encoded at
-the beginning as a sequence of 4 bytes (i.e. a single 32 bit integer). The constant section may be empty, although in
+the beginning as a sequence of 4 bytes (i.e. a single 32 bit integer). The constant segment may be empty, although in
-real-world scenarios it's unlikely that it would.
+real-world scenarios likely won't.
 
-### Code section
+## Code segment
 
-The code section contains the linear sequence of bytecode instructions of a peon program. It is to be read directly 
+The code segment contains the linear sequence of bytecode instructions of a peon program. It is to be read directly 
-and without modifications. The section's size is fixed and is encoded at the beginning as a sequence of 3 bytes 
+and without modifications. The segment's size is fixed and is encoded at the beginning as a sequence of 3 bytes 
 (i.e. a single 24 bit integer).

docs/manual.md

@@ -33,20 +33,25 @@ to happen, we need:
 - C/Nim FFI
 - A package manager
 
-Peon ~~steals~~ borrows many ideas from Python and Nim (the latter being the language peon itself is written in).
+Peon ~~steals~~ borrows many ideas from Python, Nim (the the language peon itself is written in), C and many others.
 
 ## Peon by Example
 
-Here follow a few examples of peon code to make it clear what the end product should look like
+Here follow a few examples of peon code to make it clear what the end product should look like. Note that
+not all examples represent working functionality and some of these examples might not be up to date either.
+For somewhat updated tests, check the [tests](../tests/) directory.
 
 ### Variable declarations
 
 ```
-var x = 5;      # Inferred type is int64
+var x = 5;        # Inferred type is int64
-var y = 3'u16;  # Type is specified as uint16
+var y = 3'u16;    # Type is specified as uint16
-x = 6;          # Works: type matches
+x = 6;            # Works: type matches
-x = 3.0;        # Cannot assign float64 to x
+x = 3.0;          # Error: Cannot assign float64 to x
-var x = 3.14;   # Cannot re-declare x
+var x = 3.14;     # Error: Cannot re-declare x
+const z = 6.28;   # Constant declaration
+let a = "hi!";    # Cannot be reassigned/mutated
+var b: int32 = 5; # Explicit type declaration
 ```
 
 __Note__: Peon supports [name stropping](https://en.wikipedia.org/wiki/Stropping_(syntax)), meaning
@@ -57,7 +62,7 @@ __Note__: Peon supports [name stropping](https://en.wikipedia.org/wiki/Stropping
 
 ```
 fn fib(n: int): int {
-    if (n < 3) {
+    if n < 3 {
         return n;
     }
     return fib(n - 1) +  fib(n - 2);
@@ -78,7 +83,7 @@ type Foo = object { # Can also be "ref object" for reference types (managed auto
 ### Operator overloading
 
 ```
-operator `+`(a, b: Foo) {
+operator `+`(a, b: Foo): Foo {
     return Foo(fieldOne: a.fieldOne + b.fieldOne, fieldTwo: a.fieldTwo + b.fieldTwo);
 }
 
@@ -102,14 +107,14 @@ __Note__: Operators can be called as functions too. Just wrap their name in back
 ```
 
 __Note__: Code the likes of `a.b()` is desugared to `b(a)` if there exists a function `b` whose 
-    signature is compatible with the value of of `a` (assuming `a` doesn't have a `b` field, in
+    signature is compatible with the value of `a` (assuming `a` doesn't have a field named `b`, in
     which case the attribute resolution takes precedence)
 
 
-### Generic declarations
+### Generics
 
 ```
-fn genericSum[T](a, b: T): T {  # Note: "a, b: T" means that both a and b are of type T
+fn genericSum[T: Number](a, b: T): T {  # Note: "a, b: T" means that both a and b are of type T
     return a + b;
 }
 
@@ -120,16 +125,28 @@ genericSum(3.14, 0.1);
 genericSum(1'u8, 250'u8);
 ```
 
-#### Multiple generics
+#### More generics!
 
 ```
-fn genericSth[T, K](a: T, b: K) {  # Note: no return type == void function!
+fn genericSth[T: someInterface, K: someInterface2](a: T, b: K) {  # Note: no return type == void function!
     # code...
 }
 
 genericSth(1, 3.0);
 ```
 
+
+#### Even more generics?
+
+```
+type Box*[T: SomeNumber] = object {
+    num: T;
+}
+
+var boxFloat = Box[float](1.0);
+var boxInt = Box[int](1);
+```
+
 __Note__: The `*` modifier to make a name visible outside the current module must be put
 __before__ generics declarations, so only `fn foo*[T](a: T) {}` is the correct syntax
 
@@ -138,7 +155,7 @@ __before__ generics declarations, so only `fn foo*[T](a: T) {}` is the correct s
 ```
 fn someF: int;  # Semicolon, no body!
 
-someF();  # This works!
+print(someF());  # This works!
 
 fn someF: int {
     return 42;
@@ -149,7 +166,7 @@ fn someF: int {
 
 ```
 generator count(n: int): int {
-    while (n > 0) {
+    while n > 0 {
         yield n;
         n -= 1;
     }
@@ -175,12 +192,13 @@ coroutine req(url: string): string {
 
 coroutine main(urls: list[string]) {
     pool = concur.pool();  # Creates a task pool: like a nursery in njsmith's article
-    for (var i = 0; i < urls.len(); i += 1) {
+    foreach url in urls {
-        pool.spawn(req, urls[i]);
+        pool.spawn(req, urls);
     }
     # The pool has internal machinery that makes the parent
     # task wait until all child exit! When this function
-    # returns, ALL child tasks will have exited somehow
+    # returns, ALL child tasks will have exited somehow.
+    # Exceptions and return values propagate neatly, too.
 }
 
 

src/backend/types.nim

@@ -1,54 +0,0 @@
-# Copyright 2022 Mattia Giambirtone & All Contributors
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-type
-    ObjectKind* = enum
-        ## Enumeration of Peon
-        ## types
-        Int8, UInt8, Int16, UInt16, Int32,
-        UInt32, Int64, UInt64, Float32, Float64,
-        Char, Byte, String, Function, CustomType,
-        Nil, Nan, Bool, Inf
-    PeonObject* = object
-        ## A generic Peon object
-        case kind*: ObjectKind:
-            of Bool:
-                boolean*: bool
-            of Inf:
-                positive*: bool
-            of Byte:
-                `byte`*: byte
-            of Int8:
-                tiny*: uint8
-            of UInt8:
-                uTiny*: uint8
-            of Int16:
-                short*: int16
-            of UInt16:
-                uShort*: uint16
-            of Int32:
-                `int`*: int32
-            of UInt32:
-                uInt*: uint32
-            of Int64:
-                long*: int64
-            of UInt64:
-                uLong*: uint64
-            of Nil, Nan:
-                discard
-            of CustomType:
-                fields*: seq[PeonObject]
-            else:
-                discard # TODO

src/backend/vm.nim

@@ -11,35 +11,59 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 ## The Peon runtime environment
-import types
+{.push checks:off.}   # The VM is a critical point where checks are deleterious
-import strformat
+
+import std/monotimes
+import std/math
+
+
 import ../config
 import ../frontend/meta/bytecode
 import ../util/multibyte
 
 
+import strutils
+when debugVM:
+    import std/strformat
+    import std/terminal
+
+
+
 type
     PeonVM* = ref object
-        ## The Peon Virtual Machine
+        ## The Peon Virtual Machine.
-        stack: seq[PeonObject]
+        ## Note how the only data
-        ip: int                     # Instruction pointer
+        ## type we handle here is
-        cache: array[6, PeonObject] # Singletons cache
+        ## a 64-bit unsigned integer:
+        ## This is to allow the use
+        ## of unboxed primitive types.
+        ## For more complex types, the
+        ## value represents a pointer to
+        ## some stack- or heap-allocated
+        ## object. The VM has no concept
+        ## of type by itself and it relies
+        ## on the compiler to produce the
+        ## correct results
+        ip: uint64                  # Instruction pointer
         chunk: Chunk                # Piece of bytecode to execute
-        frames: seq[int]            # Stores the initial index of stack frames
+        calls: seq[uint64]          # Our call stack
-        heapVars: seq[PeonObject]   # Stores variables that do not have stack semantics (i.e. "static")
+        operands: seq[uint64]       # Our operand stack
+        cache: array[6, uint64]     # Singletons cache
+        frames: seq[uint64]         # Stores the bottom of stack frames
+        closedOver: seq[uint64]     # Stores variables that do not have stack semantics
+        results: seq[uint64]        # Stores function's results (return values)
 
 
 proc initCache*(self: PeonVM) =
     ## Initializes the VM's
     ## singletons cache
-    self.cache[0] = PeonObject(kind: Nil)
+    self.cache[0] = 0x0  # Nil
-    self.cache[1] = PeonObject(kind: Bool, boolean: true)
+    self.cache[1] = 0x1  # True
-    self.cache[2] = PeonObject(kind: Bool, boolean: false)
+    self.cache[2] = 0x2  # False
-    self.cache[3] = PeonObject(kind: ObjectKind.Inf, positive: true)
+    self.cache[3] = 0x3  # Positive inf
-    self.cache[4] = PeonObject(kind: ObjectKind.Inf, positive: false)
+    self.cache[4] = 0x4  # Negative inf
-    self.cache[5] = PeonObject(kind: ObjectKind.Nan)
+    self.cache[5] = 0x5  # NaN
 
 
 proc newPeonVM*: PeonVM =
@@ -48,66 +72,114 @@ proc newPeonVM*: PeonVM =
     new(result)
     result.ip = 0
     result.frames = @[]
-    result.stack = newSeq[PeonObject]()
+    result.calls = newSeq[uint64]()
+    result.operands = newSeq[uint64]()
     result.initCache()
 
 
-## Getters for singleton types (they are cached!)
+# Getters for singleton types
+{.push inline.}
 
-proc getNil*(self: PeonVM): PeonObject = self.cache[0]
+proc getNil*(self: PeonVM): uint64 = self.cache[2]
 
-proc getBool*(self: PeonVM, value: bool): PeonObject =
+proc getBool*(self: PeonVM, value: bool): uint64 =
     if value:
         return self.cache[1]
-    return self.cache[2]
+    return self.cache[0]
 
-proc getInf*(self: PeonVM, positive: bool): PeonObject =
+proc getInf*(self: PeonVM, positive: bool): uint64 =
     if positive:
         return self.cache[3]
     return self.cache[4]
 
-proc getNan*(self: PeonVM): PeonObject = self.cache[5]
+proc getNan*(self: PeonVM): uint64 = self.cache[5]
-
-## Stack primitives. Note: all stack accessing that goes
-## through the get/set wrappers is frame-relative, meaning
-## that the index is added to the current stack frame's 
-## bottom to obtain an absolute stack index.
-
-proc push(self: PeonVM, obj: PeonObject) =
-    ## Pushes a Peon object onto the
-    ## stack
-    self.stack.add(obj)
 
 
-proc pop(self: PeonVM): PeonObject =
+# Thanks to nim's *genius* idea of making x !> y a template
-    ## Pops a Peon object off the
+# for y < x (which by itself is fine) together with the fact
-    ## stack, decreasing the stack
+# that the order of evaluation of templates with the same
-    ## pointer. The object is returned
+# expression is fucking stupid (see https://nim-lang.org/docs/manual.html#order-of-evaluation
-    return self.stack.pop()
+# and https://github.com/nim-lang/Nim/issues/10425 and try not to 
+# bang your head against the nearest wall), we need a custom operator
+# that preserves the natural order of evaluation
+proc `!>`[T](a, b: T): auto {.inline.} = 
+    b < a
 
 
-proc peek(self: PeonVM): PeonObject =
+proc `!>=`[T](a, b: T): auto {.inline, used.} = 
-    ## Returns the Peon object at the top
+    b <= a
-    ## of the stack without consuming
-    ## it
-    return self.stack[^1]
 
 
-proc get(self: PeonVM, idx: int): PeonObject =
+# Stack primitives. Note: all accesses to the call stack
+# that go through the getc/setc wrappers is frame-relative, 
+# meaning that the index is added to the current stack frame's 
+# bottom to obtain an absolute stack index
+proc push(self: PeonVM, obj: uint64)  =
+    ## Pushes a value object onto the
+    ## operand stack
+    self.operands.add(obj)
+
+
+proc pop(self: PeonVM): uint64  =
+    ## Pops a value off the
+    ## operand stack and
+    ## returns it
+    return self.operands.pop()
+
+
+proc peekb(self: PeonVM, distance: BackwardsIndex = ^1): uint64  =
+    ## Returns the value at the
+    ## given (backwards) distance from the top of
+    ## the operand stack without consuming it
+    return self.operands[distance]
+
+
+proc peek(self: PeonVM, distance: int = 0): uint64  =
+    ## Returns the value at the
+    ## given distance from the top of
+    ## the operand stack without consuming it
+    if distance < 0:
+        return self.peekb(^(-distance))
+    return self.operands[self.operands.high() + distance]
+
+
+
+proc pushc(self: PeonVM, val: uint64)  =
+    ## Pushes a value to the
+    ## call stack
+    self.calls.add(val)
+
+
+proc popc(self: PeonVM): uint64  =
+    ## Pops a value off the call
+    ## stack and returns it
+    return self.calls.pop()
+
+
+proc peekc(self: PeonVM, distance: int = 0): uint64 {.used.} =
+    ## Returns the value at the
+    ## given distance from the top of
+    ## the call stack without consuming it
+    return self.calls[self.calls.high() + distance]
+
+
+proc getc(self: PeonVM, idx: uint): uint64  =
     ## Accessor method that abstracts
-    ## stack accessing through stack
+    ## indexing our call stack through stack
     ## frames
-    return self.stack[idx + self.frames[^1]]
+    return self.calls[idx + self.frames[^1]]
 
 
-proc set(self: PeonVM, idx: int, val: PeonObject) =
+proc setc(self: PeonVM, idx: uint, val: uint64)  =
     ## Setter method that abstracts
-    ## stack accessing through stack
+    ## indexing our call stack through stack
     ## frames
-    self.stack[idx + self.frames[^1]] = val
+    self.calls[idx + self.frames[^1]] = val
 
+# Byte-level primitives to read and decode
+# bytecode
 
-proc readByte(self: PeonVM): uint8 =
+proc readByte(self: PeonVM): uint8  =
     ## Reads a single byte from the
     ## bytecode and returns it as an
     ## unsigned 8 bit integer
@@ -115,7 +187,7 @@ proc readByte(self: PeonVM): uint8 =
     return self.chunk.code[self.ip - 1]
 
 
-proc readShort(self: PeonVM): uint16 =
+proc readShort(self: PeonVM): uint16  =
     ## Reads two bytes from the
     ## bytecode and returns them
     ## as an unsigned 16 bit
@@ -123,7 +195,7 @@ proc readShort(self: PeonVM): uint16 =
     return [self.readByte(), self.readByte()].fromDouble()
 
 
-proc readLong(self: PeonVM): uint32 =
+proc readLong(self: PeonVM): uint32  =
     ## Reads three bytes from the
     ## bytecode and returns them
     ## as an unsigned 32 bit
@@ -133,69 +205,170 @@ proc readLong(self: PeonVM): uint32 =
     return uint32([self.readByte(), self.readByte(), self.readByte()].fromTriple())
 
 
-proc readInt64(self: PeonVM, idx: int): PeonObject =
+proc readUInt(self: PeonVM): uint32  =
+    ## Reads three bytes from the
+    ## bytecode and returns them
+    ## as an unsigned 32 bit
+    ## integer
+    return uint32([self.readByte(), self.readByte(), self.readByte(), self.readByte()].fromQuad())
+
+
+# Functions to read primitives from the chunk's
+# constants table
+
+proc constReadInt64(self: PeonVM, idx: int): int64  =
     ## Reads a constant from the
     ## chunk's constant table and
-    ## returns a Peon object. Assumes
+    ## returns it as an int64
-    ## the constant is an Int64
     var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1],
                self.chunk.consts[idx + 2], self.chunk.consts[idx + 3],
                self.chunk.consts[idx + 4], self.chunk.consts[idx + 5],
                self.chunk.consts[idx + 6], self.chunk.consts[idx + 7],
             ]
-    result = PeonObject(kind: Int64)
+    copyMem(result.addr, arr.addr, sizeof(arr))
-    copyMem(result.long.addr, arr.addr, sizeof(arr))
 
 
-proc readUInt64(self: PeonVM, idx: int): PeonObject =
+proc constReadUInt64(self: PeonVM, idx: int): uint64  =
     ## Reads a constant from the
     ## chunk's constant table and
-    ## returns a Peon object. Assumes
+    ## returns it as an uint64
-    ## the constant is an UInt64
     var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1],
                self.chunk.consts[idx + 2], self.chunk.consts[idx + 3],
                self.chunk.consts[idx + 4], self.chunk.consts[idx + 5],
                self.chunk.consts[idx + 6], self.chunk.consts[idx + 7],
             ]
-    result = PeonObject(kind: UInt64)
+    copyMem(result.addr, arr.addr, sizeof(arr))
-    copyMem(result.uLong.addr, arr.addr, sizeof(arr))
 
 
-proc readUInt32(self: PeonVM, idx: int): PeonObject =
+proc constReadUInt32(self: PeonVM, idx: int): uint32  =
     ## Reads a constant from the
     ## chunk's constant table and
-    ## returns a Peon object. Assumes
+    ## returns it as an int32
-    ## the constant is an UInt32
     var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1],
                self.chunk.consts[idx + 2], self.chunk.consts[idx + 3]]
-    result = PeonObject(kind: UInt32)
+    copyMem(result.addr, arr.addr, sizeof(arr))
-    copyMem(result.uInt.addr, arr.addr, sizeof(arr))
 
 
-proc readInt32(self: PeonVM, idx: int): PeonObject =
+proc constReadInt32(self: PeonVM, idx: int): int32  =
     ## Reads a constant from the
     ## chunk's constant table and
-    ## returns a Peon object. Assumes
+    ## returns it as an uint32
-    ## the constant is an Int32
     var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1],
                self.chunk.consts[idx + 2], self.chunk.consts[idx + 3]]
-    result = PeonObject(kind: Int32)
+    copyMem(result.addr, arr.addr, sizeof(arr))
-    copyMem(result.`int`.addr, arr.addr, sizeof(arr))
+
+
+proc constReadInt16(self: PeonVM, idx: int): int16  =
+    ## Reads a constant from the
+    ## chunk's constant table and
+    ## returns it as an int16
+    var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1]]
+    copyMem(result.addr, arr.addr, sizeof(arr))
+
+
+proc constReadUInt16(self: PeonVM, idx: int): uint16  =
+    ## Reads a constant from the
+    ## chunk's constant table and
+    ## returns it as an uint16
+    var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1]]
+    copyMem(result.addr, arr.addr, sizeof(arr))
+
+
+proc constReadInt8(self: PeonVM, idx: int): int8  =
+    ## Reads a constant from the
+    ## chunk's constant table and
+    ## returns it as an int8
+    result = int8(self.chunk.consts[idx])
+
+
+proc constReadUInt8(self: PeonVM, idx: int): uint8  =
+    ## Reads a constant from the
+    ## chunk's constant table and
+    ## returns it as an uint8
+    result = self.chunk.consts[idx]
+
+
+proc constReadFloat32(self: PeonVM, idx: int): float32 =
+    ## Reads a constant from the
+    ## chunk's constant table and
+    ## returns it as a float32
+    var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1],
+               self.chunk.consts[idx + 2], self.chunk.consts[idx + 3]]
+    copyMem(result.addr, arr.addr, sizeof(arr))
+
+
+proc constReadFloat64(self: PeonVM, idx: int): float =
+    ## Reads a constant from the
+    ## chunk's constant table and
+    ## returns it as a float
+    var arr = [self.chunk.consts[idx], self.chunk.consts[idx + 1],
+               self.chunk.consts[idx + 2], self.chunk.consts[idx + 3],
+               self.chunk.consts[idx + 4], self.chunk.consts[idx + 5], 
+               self.chunk.consts[idx + 6], self.chunk.consts[idx + 7]]
+    copyMem(result.addr, arr.addr, sizeof(arr))
+{.pop.}
+
+
+when debugVM:   # So nim shuts up 
+    proc debug(self: PeonVM) =
+        ## Implements the VM's runtime
+        ## debugger
+        styledEcho fgMagenta, "IP: ", fgYellow, &"{self.ip}"
+        styledEcho fgBlue, "Instruction: ", fgRed, &"{OpCode(self.chunk.code[self.ip])} (", fgYellow, $self.chunk.code[self.ip], fgRed, ")"
+        if self.calls.len() !> 0:
+            stdout.styledWrite(fgGreen, "Call Stack: ", fgMagenta, "[")
+            for i, e in self.calls:
+                stdout.styledWrite(fgYellow, $e)
+                if i < self.calls.high():
+                    stdout.styledWrite(fgYellow, ", ")
+            styledEcho fgMagenta, "]"
+        if self.operands.len() !> 0:
+            stdout.styledWrite(fgBlue, "Operand Stack: ", fgMagenta, "[")
+            for i, e in self.operands:
+                stdout.styledWrite(fgYellow, $e)
+                if i < self.operands.high():
+                    stdout.styledWrite(fgYellow, ", ")
+            styledEcho fgMagenta, "]"
+        if self.frames.len() !> 0:
+            stdout.styledWrite(fgCyan, "Current Frame: ", fgMagenta, "[")
+            for i, e in self.calls[self.frames[^1]..^1]:
+                stdout.styledWrite(fgYellow, $e)
+                if i < self.calls.high():
+                    stdout.styledWrite(fgYellow, ", ")
+            styledEcho fgMagenta, "]"
+            stdout.styledWrite(fgRed, "Live stack frames: ", fgMagenta, "[")
+            for i, e in self.frames:
+                stdout.styledWrite(fgYellow, $e)
+                if i < self.frames.high():
+                    stdout.styledWrite(fgYellow, ", ")
+            styledEcho fgMagenta, "]"
+        if self.closedOver.len() !> 0:
+            stdout.styledWrite(fgGreen, "Closure Array: ", fgMagenta, "[")
+            for i, e in self.closedOver:
+                stdout.styledWrite(fgYellow, $e)
+                if i < self.closedOver.high():
+                    stdout.styledWrite(fgYellow, ", ")
+            styledEcho fgMagenta, "]"
+        if self.results.len() !> 0:
+            stdout.styledWrite(fgYellow, "Function Results: ", fgMagenta, "[")
+            for i, e in self.results:
+                stdout.styledWrite(fgYellow, $e)
+                if i < self.results.high():
+                    stdout.styledWrite(fgYellow, ", ")
+            styledEcho fgMagenta, "]"
+        discard readLine stdin
 
 
 proc dispatch*(self: PeonVM) =
     ## Main bytecode dispatch loop
-    var instruction: OpCode
+    var instruction {.register.}: OpCode
     while true:
+        {.computedgoto.}  # https://nim-lang.org/docs/manual.html#pragmas-computedgoto-pragma
+        when debugVM:
+            self.debug()
         instruction = OpCode(self.readByte())
-        when DEBUG_TRACE_VM:
-            echo &"IP: {self.ip}"
-            echo &"SP: {self.stack.high()}"
-            echo &"Stack: {self.stack}"
-            echo &"Instruction: {instruction}"
-            discard readLine stdin
         case instruction:
-            # Constant loading
+            # Constant loading instructions
             of LoadTrue:
                 self.push(self.getBool(true))
             of LoadFalse:
@@ -207,106 +380,316 @@ proc dispatch*(self: PeonVM) =
             of LoadInf:
                 self.push(self.getInf(true))
             of LoadInt64:
-                self.push(self.readInt64(int(self.readLong())))
+                self.push(uint64(self.constReadInt64(int(self.readLong()))))
             of LoadUInt64:
-                self.push(self.readUInt64(int(self.readLong())))
+                self.push(uint64(self.constReadUInt64(int(self.readLong()))))
             of LoadUInt32:
-                self.push(self.readUInt32(int(self.readLong())))
+                self.push(uint64(self.constReadUInt32(int(self.readLong()))))
+            of LoadInt32:
+                self.push(uint64(self.constReadInt32(int(self.readLong()))))
+            of LoadInt16:
+                self.push(uint64(self.constReadInt16(int(self.readLong()))))
+            of LoadUInt16:
+                self.push(uint64(self.constReadUInt16(int(self.readLong()))))
+            of LoadInt8:
+                self.push(uint64(self.constReadInt8(int(self.readLong()))))
+            of LoadUInt8:
+                self.push(uint64(self.constReadUInt8(int(self.readLong()))))
+            of LoadString:
+                # TODO: Use constReadString with own memory manager
+                # Strings are broken rn!!
+                let size = int(self.readLong())
+                let idx = int(self.readLong())
+                var str = self.chunk.consts[idx..<idx + size].fromBytes()
+                self.push(cast[uint64](str.addr))
+            # We cast instead of converting because, unlike with integers,
+            # we don't want nim to touch any of the bits of the underlying
+            # value!
+            of LoadFloat32:
+                self.push(cast[uint64](self.constReadFloat32(int(self.readLong()))))
+            of LoadFloat64:
+                self.push(cast[uint64](self.constReadFloat64(int(self.readLong()))))
+            of LoadFunction:
+                # Loads a function address onto the operand stack
+                self.push(uint64(self.readLong()))
+            of LoadReturnAddress:
+                # Loads a 32-bit unsigned integer onto the operand stack.
+                # Used to load function return addresses
+                self.push(uint64(self.readUInt()))
             of Call:
-                # Calls a function. The calling convention for peon
+                # Calls a peon function. The calling convention here
-                # functions is pretty simple: the return address sits
+                # is pretty simple: the first value in the frame is
-                # at the bottom of the stack frame, then follow the
+                # the new instruction pointer to jump to, then a
-                # arguments and all temporaries/local variables
+                # 32-bit return address follows. After that, all
-                let newIp = self.readLong()
+                # arguments and locals follow. Note that, due to
-                # We do this because if we immediately changed
+                # how the stack works, all arguments before the call 
-                # the instruction pointer, we'd read the wrong
+                # are in the reverse order in which they are passed
-                # value for the argument count. Storing it and
+                # to the function
-                # changing it later fixes this issue
+                let argc = self.readLong().int
-                self.frames.add(int(self.readLong()))
+                let retAddr = self.peek(-argc - 1)    # Return address
-                self.ip = int(newIp)
+                let jmpAddr = self.peek(-argc - 2)    # Function address
-            of OpCode.Return:
+                self.ip = jmpAddr
-                # Returns from a void function or terminates the
+                self.pushc(jmpAddr)
-                # program entirely if we're at the topmost frame
+                self.pushc(retAddr)
-                if self.frames.len() > 1:
+                # Creates a new result slot for the
-                    let frame = self.frames.pop()
+                # function's return value
-                    for i in countdown(self.stack.high(), frame):
+                self.results.add(self.getNil())
-                        discard self.pop()
+                # Creates a new call frame
-                    self.ip = int(self.pop().uInt)
+                self.frames.add(uint64(self.calls.len() - 2))
-                else:
+                # Loads the arguments onto the stack
-                    return
+                for _ in 0..<argc:
-            of ReturnValue:
+                    self.pushc(self.pop())
-                # Returns from a function which has a return value,
+                # Pops the function and return address 
-                # pushing it on the stack
+                # off the operand stack since they're 
-                let retVal = self.pop()
+                # not needed there anymore
-                let frame = self.frames.pop()
-                for i in countdown(self.stack.high(), frame):
-                    discard self.pop()
-                self.ip = int(self.pop().uInt)
-                self.push(retVal)
-            of StoreVar:
-                # Stores the value at the top of the stack
-                # into the given stack index
-                self.set(int(self.readLong()), self.pop())
-            of StoreHeap:
-                self.heapVars.add(self.pop())
-            of LoadHeap:
-                self.push(self.heapVars[self.readLong()])
-            of LoadVar:
-                self.push(self.get(int(self.readLong())))
-            of NoOp:
-                continue
-            of Pop:
                 discard self.pop()
-            of PopN:
+                discard self.pop()
-                for _ in 0..<int(self.readLong()):
+            of Return:
+                # Returns from a function.
+                # Every peon program is wrapped
+                # in a hidden function, so this 
+                # will also exit the VM if we're 
+                # at  the end of the program
+                while self.calls.len().uint64 !> self.frames[^1] + 2'u64:
+                    # Discards the function's local variables,
+                    # if there is any
+                    discard self.popc()
+                let ret = self.popc() # Return address
+                discard self.popc()   # Function address
+                if self.readByte() == 1:
+                    # Function is non-void!
+                    self.push(self.results.pop())
+                else:
+                    discard self.results.pop()
+                # Discard the topmost stack frame
+                discard self.frames.pop()
+                if self.frames.len() == 0:
+                    # End of the program!
+                    return
+                self.ip = ret.uInt
+            of SetResult:
+                # Sets the result of the
+                # current function. A Return
+                # instruction will pop this
+                # off the results array and
+                # onto the operand stack when
+                # the current function exits.
+                self.results[self.frames.high()] = self.pop()
+            of StoreVar:
+                # Stores the value at the top of the operand stack
+                # into the given call stack index
+                let idx = self.readLong()
+                when debugVM:
+                    assert idx.int - self.calls.high() <= 1, "StoreVar index is bigger than the length of the call stack"
+                if idx + self.frames[^1] <= self.calls.high().uint:
+                    self.setc(idx, self.pop())
+                else:
+                    self.pushc(self.pop())
+            of StoreClosure:
+                # Stores/updates the value of a closed-over
+                # variable
+                let idx = self.readLong().int
+                if idx !> self.closedOver.high():
+                    # Note: we *peek* the stack, but we
+                    # don't pop!
+                    self.closedOver.add(self.peek())
+                else:
+                    self.closedOver[idx] = self.peek()
+            of LoadClosure:
+                # Loads a closed-over variable onto the
+                # stack
+                self.push(self.closedOver[self.readLong()])
+            of LoadVar:
+                # Pushes a variable onto the operand
+                # stack
+                self.push(self.getc(self.readLong()))
+            of NoOp:
+                # Does nothing
+                continue
+            of PopC:
+                # Pops a value off the call stack
+                discard self.popc()
+            of Pop:
+                # Pops a value off the operand stack
+                discard self.pop()
+            of PushC:
+                # Pushes a value from the operand stack
+                # onto the call stack
+                self.pushc(self.pop())
+            of PopRepl:
+                # Pops a peon object off the
+                # operand stack and prints it.
+                # Used in interactive REPL mode
+                if self.frames.len() !> 1:
                     discard self.pop()
+                    continue
+                echo self.pop()
+            of PopN:
+                # Pops N elements off the call stack
+                for _ in 0..<int(self.readShort()):
+                    discard self.popc()
+            # Jump opcodes
             of Jump:
-                self.ip = int(self.readShort())
+                # Absolute jump
+                self.ip = self.readLong()
             of JumpForwards:
-                self.ip += int(self.readShort())
+                # Relative, forward-jump
+                self.ip += self.readLong()
             of JumpBackwards:
-                self.ip -= int(self.readShort())
+                # Relative, backward-jump
+                self.ip -= self.readLong()
             of JumpIfFalse:
-                if not self.peek().boolean:
+                # Conditional positive jump
-                    self.ip += int(self.readShort())
+                if not self.peek().bool:
+                    self.ip += self.readLong()
             of JumpIfTrue:
-                if self.peek().boolean:
+                # Conditional positive jump
-                    self.ip += int(self.readShort())
+                if self.peek().bool:
+                    self.ip += self.readLong()
             of JumpIfFalsePop:
-                if not self.peek().boolean:
+                let ip = self.readLong()
-                    self.ip += int(self.readShort())
+                if not self.peek().bool:
+                    self.ip += ip
                 discard self.pop()
             of JumpIfFalseOrPop:
-                if not self.peek().boolean:
+                if not self.peek().bool:
-                    self.ip += int(self.readShort())
+                    self.ip += self.readLong()
                 else:
                     discard self.pop()
-            of LongJumpIfFalse:
+            # Built-in operations on primitive types.
-                if not self.peek().boolean:
+            # Note: for operations where the order of
-                    self.ip += int(self.readLong())
+            # the operands matters, we don't need to
-            of LongJumpIfFalsePop:
+            # swap the order of the calls to pop: this 
-                if not self.peek().boolean:
+            # is because operators are handled like peon 
-                    self.ip += int(self.readLong())
+            # functions, which means the arguments are 
-                discard self.pop()
+            # already reversed on the stack when we 
-            of LongJumpForwards:
+            # execute the instruction
-                self.ip += int(self.readLong())
+            of Negate:
-            of LongJumpBackwards:
+                self.push(uint64(-int64(self.pop())))
-                self.ip -= int(self.readLong())
+            of NegateFloat64:
-            of LongJump:
+                self.push(cast[uint64](-cast[float](self.pop())))
-                self.ip = int(self.readLong())
+            of NegateFloat32:
-            of LongJumpIfFalseOrPop:
+                self.push(cast[uint64](-cast[float32](self.pop())))
-                if not self.peek().boolean:
+            of Add:
-                    self.ip += int(self.readLong())
+                self.push(self.pop() + self.pop())
+            of Subtract:
+                self.push(self.pop() - self.pop())
+            of Multiply:
+                self.push(self.pop() * self.pop())
+            of Divide:
+                self.push(self.pop() div self.pop())
+            of SignedDivide:
+                self.push(uint64(int64(self.pop()) div int64(self.pop())))
+            of AddFloat64:
+                self.push(cast[uint64](cast[float](self.pop()) + cast[float](self.pop())))
+            of SubtractFloat64:
+                self.push(cast[uint64](cast[float](self.pop()) - cast[float](self.pop())))
+            of MultiplyFloat64:
+                self.push(cast[uint64](cast[float](self.pop()) * cast[float](self.pop())))
+            of DivideFloat64:
+                self.push(cast[uint64](cast[float](self.pop()) / cast[float](self.pop())))
+            of AddFloat32:
+                self.push(cast[uint64](cast[float32](self.pop()) + cast[float32](self.pop())))
+            of SubtractFloat32:
+                self.push(cast[uint64](cast[float32](self.pop()) - cast[float32](self.pop())))
+            of MultiplyFloat32:
+                self.push(cast[uint64](cast[float32](self.pop()) * cast[float32](self.pop())))
+            of DivideFloat32:
+                self.push(cast[uint64](cast[float32](self.pop()) / cast[float32](self.pop())))
+            of Pow:
+                self.push(uint64(self.pop() ^ self.pop()))
+            of SignedPow:
+                self.push(uint64(int64(self.pop()) ^ int64(self.pop())))
+            of PowFloat64:
+                self.push(cast[uint64](pow(cast[float](self.pop()), cast[float](self.pop()))))
+            of PowFloat32:
+                self.push(cast[uint64](pow(cast[float](self.pop()), cast[float](self.pop()))))
+            of Mod:
+                self.push(uint64(self.pop() mod self.pop()))
+            of SignedMod:
+                self.push(uint64(int64(self.pop()) mod int64(self.pop())))
+            of ModFloat64:
+                self.push(cast[uint64](floorMod(cast[float](self.pop()), cast[float](self.pop()))))
+            of ModFloat32:
+                self.push(cast[uint64](floorMod(cast[float](self.pop()), cast[float](self.pop()))))
+            of LShift:
+                self.push(self.pop() shl self.pop())
+            of RShift:
+                self.push(self.pop() shr self.pop())
+            of Xor:
+                self.push(self.pop() xor self.pop())
+            of Not:
+                self.push(not self.pop())
+            of And:
+                self.push(self.pop() and self.pop())
+            # Comparison opcodes
+            of Equal:
+                self.push(self.getBool(self.pop() == self.pop()))
+            of NotEqual:
+                self.push(self.getBool(self.pop() != self.pop()))
+            of GreaterThan:
+                self.push(self.getBool(self.pop() !> self.pop()))
+            of LessThan:
+                self.push(self.getBool(self.pop() < self.pop()))
+            of GreaterOrEqual:
+                self.push(self.getBool(self.pop() !>= self.pop()))
+            of LessOrEqual:
+                self.push(self.getBool(self.pop() <= self.pop()))
+            # Print opcodes
+            of PrintInt64:
+                echo int64(self.pop())
+            of PrintUInt64:
+                echo self.pop()
+            of PrintInt32:
+                echo int32(self.pop())
+            of PrintUInt32:
+                echo uint32(self.pop())
+            of PrintInt16:
+                echo int16(self.pop())
+            of PrintUInt16:
+                echo uint16(self.pop())
+            of PrintInt8:
+                echo int8(self.pop())
+            of PrintUInt8:
+                echo uint8(self.pop())
+            of PrintFloat32:
+                echo cast[float32](self.pop())
+            of PrintFloat64:
+                echo cast[float](self.pop())
+            of PrintHex:
+                echo "0x" & self.pop().toHex().strip(chars={'0'})
+            of PrintBool:
+                if self.pop().bool:
+                    echo "true"
                 else:
-                    discard self.pop()
+                    echo "false"
+            of PrintInf:
+                if self.pop() == 0x3:
+                    echo "-inf"
+                else:
+                    echo "inf"
+            of PrintNan:
+                echo "nan"
+            of PrintString:
+                echo cast[ptr string](self.pop())[]  # TODO
+            of SysClock64:
+                # Pushes the value of a monotonic clock
+                # onto the operand stack. This can be used
+                # to track system time accurately, but it 
+                # cannot be converted to a date. The number
+                # is in seconds
+                self.push(cast[uint64](getMonoTime().ticks().float() / 1_000_000_000))
             else:
                 discard
 
 
 proc run*(self: PeonVM, chunk: Chunk) =
-    ## Executes a piece of Peon bytecode.
+    ## Executes a piece of Peon bytecode
     self.chunk = chunk
-    self.frames = @[0]
+    self.frames = @[]
-    self.stack = @[]
+    self.calls = @[]
+    self.operands = @[]
     self.ip = 0
     self.dispatch()
+
+{.pop.}

src/config.nim

@@ -14,25 +14,28 @@
 
 import strformat
 
-
+# Debug various components of peon
-const BYTECODE_MARKER* = "PEON_BYTECODE"
+const debugLexer* {.booldefine.} = false
-const HEAP_GROW_FACTOR* = 2 # How much extra memory to allocate for dynamic arrays and garbage collection when resizing
+const debugParser* {.booldefine.} = false
-when HEAP_GROW_FACTOR <= 1:
+const debugCompiler* {.booldefine.} = false
+const debugVM* {.booldefine.} = false
+const debugGC* {.booldefine.} = false
+const debugMem* {.booldefine.} = false
+const debugSerializer* {.booldefine.} = false
+const PeonBytecodeMarker* = "PEON_BYTECODE"
+const HeapGrowFactor* = 2 # How much extra memory to allocate for dynamic arrays and garbage collection when resizing
+when HeapGrowFactor <= 1:
     {.fatal: "Heap growth factor must be > 1".}
-const PEON_VERSION* = (major: 0, minor: 4, patch: 0)
+const PeonVersion* = (major: 0, minor: 1, patch: 0)
-const PEON_RELEASE* = "alpha"
+const PeonRelease* = "alpha"
-const PEON_COMMIT_HASH* = "ed79385e2a93100331697f26a4a90157e60ad27a"
+const PeonCommitHash* = "b273cd744883458a4a6354a0cc5f4f5d0f560c31"
-when len(PEON_COMMIT_HASH) != 40:
+when len(PeonCommitHash) != 40:
     {.fatal: "The git commit hash must be exactly 40 characters long".}
-const PEON_BRANCH* = "master"
+const PeonBranch* = "unboxed-types"
-when len(PEON_BRANCH) > 255:
+when len(PeonBranch) > 255:
     {.fatal: "The git branch name's length must be less than or equal to 255 characters".}
-const DEBUG_TRACE_VM* = true # Traces VM execution
+const PeonVersionString* = &"Peon {PeonVersion.major}.{PeonVersion.minor}.{PeonVersion.patch} {PeonRelease} ({PeonBranch}, {CompileDate}, {CompileTime}, {PeonCommitHash[0..8]}) [Nim {NimVersion}] on {hostOS} ({hostCPU})"
-const DEBUG_TRACE_GC* = false # Traces the garbage collector (TODO)
+const HelpMessage* = """The peon programming language, Copyright (C) 2022 Mattia Giambirtone & All Contributors
-const DEBUG_TRACE_ALLOCATION* = false # Traces memory allocation/deallocation
-const DEBUG_TRACE_COMPILER* = false # Traces the compiler
-const PEON_VERSION_STRING* = &"Peon {PEON_VERSION.major}.{PEON_VERSION.minor}.{PEON_VERSION.patch} {PEON_RELEASE} ({PEON_BRANCH}, {CompileDate}, {CompileTime}, {PEON_COMMIT_HASH[0..8]}) [Nim {NimVersion}] on {hostOS} ({hostCPU})"
-const HELP_MESSAGE* = """The peon programming language, Copyright (C) 2022 Mattia Giambirtone & All Contributors
 
 This program is free software, see the license distributed with this program or check
 http://www.apache.org/licenses/LICENSE-2.0 for more info.

src/frontend/lexer.nim

@@ -15,10 +15,10 @@
 ## A simple and modular tokenizer implementation with arbitrary lookahead
 ## using a customizable symbol table
 
-import strutils
+import std/strutils
-import parseutils
+import std/parseutils
-import strformat
+import std/strformat
-import tables
+import std/tables
 
 
 import meta/token
@@ -51,9 +51,17 @@ type
         file: string
         lines: seq[tuple[start, stop: int]]
         lastLine: int
+        spaces: int
+    LexingError* = ref object of PeonException
+        ## A lexing error
+        lexer*: Lexer
+        file*: string
+        lexeme*: string
+        line*: int
 
 
 proc newSymbolTable: SymbolTable =
+    ## Initializes a new symbol table
     new(result)
     result.keywords = newTable[string, TokenType]()
     result.symbols = newTable[string, TokenType]()
@@ -143,6 +151,7 @@ proc isAlphaNumeric(s: string): bool =
             return false
     return true
 
+# Forward declaration
 proc incLine(self: Lexer)
 
 # Simple public getters used for error
@@ -151,6 +160,7 @@ proc getStart*(self: Lexer): int = self.start
 proc getFile*(self: Lexer): string = self.file
 proc getCurrent*(self: Lexer): int = self.current
 proc getLine*(self: Lexer): int = self.line
+proc getLines*(self: Lexer): seq[tuple[start, stop: int]] = self.lines
 proc getSource*(self: Lexer): string = self.source
 proc getRelPos*(self: Lexer, line: int): tuple[start, stop: int] = 
         if self.tokens.len() == 0 or self.tokens[^1].kind != EndOfFile:
@@ -173,6 +183,7 @@ proc newLexer*(self: Lexer = nil): Lexer =
     result.lines = @[]
     result.lastLine = 0
     result.symbols = newSymbolTable()
+    result.spaces = 0
 
 
 proc done(self: Lexer): bool =
@@ -182,8 +193,8 @@ proc done(self: Lexer): bool =
 
 proc incLine(self: Lexer) =
     ## Increments the lexer's line
-    ## and updates internal line
+    ## counter and updates internal 
-    ## metadata
+    ## line metadata
     self.lines.add((self.lastLine, self.current))
     self.lastLine = self.current
     self.line += 1
@@ -211,7 +222,8 @@ proc peek(self: Lexer, distance: int = 0, length: int = 1): string =
     ## previously consumed tokens. If the
     ## distance and/or the length are beyond
     ## EOF (even partially), the resulting string
-    ## will be shorter than length bytes
+    ## will be shorter than length bytes. The string
+    ## may be empty
     var i = distance
     while len(result) < length:
         if self.done() or self.current + i > self.source.high() or
@@ -223,9 +235,9 @@ proc peek(self: Lexer, distance: int = 0, length: int = 1): string =
 
 
 proc error(self: Lexer, message: string) =
-    ## Raises a lexing error with a formatted
+    ## Raises a lexing error with info
-    ## error message
+    ## for error messages
-    raise LexingError(msg: message, line: self.line, file: self.file, lexeme: self.peek())
+    raise LexingError(msg: message, line: self.line, file: self.file, lexeme: self.peek(), lexer: self)
 
 
 proc check(self: Lexer, s: string, distance: int = 0): bool =
@@ -282,9 +294,9 @@ proc createToken(self: Lexer, tokenType: TokenType) =
     tok.kind = tokenType
     tok.lexeme = self.source[self.start..<self.current]
     tok.line = self.line
-    tok.pos = (start: self.start, stop: self.current)
+    tok.spaces = self.spaces
-    if len(tok.lexeme) != tok.pos.stop - tok.pos.start:
+    self.spaces = 0
-        self.error("invalid state: len(tok.lexeme) != tok.pos.stop - tok.pos.start (this is most likely a compiler bug!)")
+    tok.pos = (start: self.start, stop: self.current - 1)
     self.tokens.add(tok)
 
 
@@ -295,7 +307,7 @@ proc parseEscape(self: Lexer) =
     # likely be soon. Another notable limitation is that
     # \xhhh and \nnn are limited to the size of a char
     # (i.e. uint8, or 256 values)
-    case self.peek()[0]: # We use a char instead of a string because of how case statements handle ranges with strings
+    case self.peek()[0]:   # We use a char instead of a string because of how case statements handle ranges with strings
                            # (i.e. not well, given they crash the C code generator)
         of 'a':
             self.source[self.current] = cast[char](0x07)
@@ -555,13 +567,15 @@ proc next(self: Lexer) =
         return
     elif self.match(" "):
         # Whitespaces
-        self.createToken(TokenType.Whitespace)
+        inc(self.spaces)
     elif self.match("\r"):
-        # Tabs
+        self.error("tabs are not allowed in peon code")
-        self.createToken(TokenType.Tab)
     elif self.match("\n"):
         # New line
         self.incLine()
+        # TODO: Broken
+        #[if not self.getToken("\n").isNil():
+            self.createToken(Semicolon)]#
     elif self.match("`"):
         # Stropped token
         self.parseBackticks()
@@ -576,7 +590,7 @@ proc next(self: Lexer) =
         self.parseString(self.peek(-1), mode)
     elif self.peek().isDigit():
         discard self.step() # Needed because parseNumber reads the next
-                             # character to tell the base of the number
+                            # character to tell the base of the number
         # Number literal
         self.parseNumber()
     elif self.peek().isAlphaNumeric() and self.check(["\"", "'"], 1):
@@ -594,10 +608,14 @@ proc next(self: Lexer) =
         # Keywords and identifiers
         self.parseIdentifier()
     elif self.match("#"):
-        # Inline comments, pragmas, etc.
+        if not self.match("pragma["):
-        while not (self.check("\n") or self.done()):
+            # Inline comments
-            discard self.step()
+            while not (self.match("\n") or self.done()):
-        self.createToken(Comment)
+                discard self.step()
+            self.createToken(Comment)
+            self.incLine()
+        else:
+            self.createToken(Pragma)
     else:
         # If none of the above conditions matched, there's a few
         # other options left:
@@ -607,7 +625,7 @@ proc next(self: Lexer) =
         # We handle all of these cases here by trying to
         # match the longest sequence of characters possible
         # as either an operator or a statement/expression
-        # delimiter, erroring out if there's no match
+        # delimiter
         var n = self.symbols.getMaxSymbolSize()
         while n > 0:
             for symbol in self.symbols.getSymbols(n):

src/frontend/meta/ast.nim

@@ -16,8 +16,8 @@
 ## top-down parser. For more info, check out docs/grammar.md
 
 
-import strformat
+import std/strformat
-import strutils
+import std/strutils
 
 
 import token
@@ -30,10 +30,10 @@ type
         ## precedence
 
         # Declarations
-        funDecl = 0'u8,
+        typeDecl = 0'u8
+        funDecl,
         varDecl,
         # Statements
-        forStmt, # Unused for now (for loops are compiled to while loops)
         ifStmt,
         returnStmt,
         breakStmt,
@@ -61,7 +61,7 @@ type
         sliceExpr,
         callExpr,
         getItemExpr, # Get expressions like a.b
-                         # Primary expressions
+        # Primary expressions
         groupingExpr, # Parenthesized expressions such as (true) and (3 + 4)
         trueExpr,
         falseExpr,
@@ -76,7 +76,10 @@ type
         nanExpr,
         infExpr,
         identExpr, # Identifier
-        pragmaExpr
+        pragmaExpr,
+        varExpr,
+        refExpr,
+        ptrExpr
 
     # Here I would've rather used object variants, and in fact that's what was in
     # place before, but not being able to re-declare a field of the same type in
@@ -97,6 +100,7 @@ type
     # work properly
     Declaration* = ref object of ASTNode
         ## A declaration
+        isPrivate*: bool
         pragmas*: seq[Pragma]
         generics*: seq[tuple[name: IdentExpr, cond: Expression]]
 
@@ -132,6 +136,7 @@ type
 
     IdentExpr* = ref object of Expression
         name*: Token
+        depth*: int
 
     GroupingExpr* = ref object of Expression
         expression*: Expression
@@ -150,6 +155,7 @@ type
         callee*: Expression # The object being called
         arguments*: tuple[positionals: seq[Expression], keyword: seq[tuple[
                 name: IdentExpr, value: Expression]]]
+        genericArgs*: seq[Expression]
 
     UnaryExpr* = ref object of Expression
         operator*: Token
@@ -169,15 +175,15 @@ type
 
     LambdaExpr* = ref object of Expression
         body*: Statement
-        arguments*: seq[tuple[name: IdentExpr, valueType: Expression,
+        arguments*: seq[tuple[name: IdentExpr, valueType: Expression]]
-                mutable: bool, isRef: bool, isPtr: bool]]
         defaults*: seq[Expression]
         isGenerator*: bool
         isAsync*: bool
         isPure*: bool
         returnType*: Expression
         hasExplicitReturn*: bool
-
+        freeVars*: seq[IdentExpr]
+        depth*: int
 
     SliceExpr* = ref object of Expression
         expression*: Expression
@@ -245,25 +251,40 @@ type
         name*: IdentExpr
         value*: Expression
         isConst*: bool
-        isPrivate*: bool
         isLet*: bool
         valueType*: Expression
 
     FunDecl* = ref object of Declaration
         name*: IdentExpr
         body*: Statement
-        arguments*: seq[tuple[name: IdentExpr, valueType: Expression,
+        arguments*: seq[tuple[name: IdentExpr, valueType: Expression]]
-                mutable: bool, isRef: bool, isPtr: bool]]
         defaults*: seq[Expression]
         isAsync*: bool
         isGenerator*: bool
-        isPrivate*: bool
         isPure*: bool
         returnType*: Expression
         hasExplicitReturn*: bool
+        freeVars*: seq[IdentExpr]
+        depth*: int
+
+    TypeDecl* = ref object of Declaration
+        name*: IdentExpr
+        fields*: seq[tuple[name: IdentExpr, valueType: Expression, isPrivate: bool]]
+        defaults*: seq[Expression]
+        valueType*: Expression
+        
     Pragma* = ref object of Expression
         name*: IdentExpr
         args*: seq[LiteralExpr]
+    
+    Var* = ref object of Expression
+        value*: Expression
+    
+    Ref* = ref object of Expression
+        value*: Expression
+    
+    Ptr* = ref object of Expression
+        value*: Expression
 
 
 proc isConst*(self: ASTNode): bool =
@@ -300,6 +321,28 @@ proc newPragma*(name: IdentExpr, args: seq[LiteralExpr]): Pragma =
     result.kind = pragmaExpr
     result.args = args
     result.name = name
+    result.token = name.token
+
+
+proc newVarExpr*(expression: Expression, token: Token): Var =
+    new(result)
+    result.kind = varExpr
+    result.value = expression
+    result.token = token
+
+
+proc newRefExpr*(expression: Expression, token: Token): Ref =
+    new(result)
+    result.kind = refExpr
+    result.value = expression
+    result.token = token
+
+
+proc newPtrExpr*(expression: Expression, token: Token): Ptr =
+    new(result)
+    result.kind = ptrExpr
+    result.value = expression
+    result.token = token
 
 
 proc newIntExpr*(literal: Token): IntExpr =
@@ -356,10 +399,11 @@ proc newCharExpr*(literal: Token): CharExpr =
     result.token = literal
 
 
-proc newIdentExpr*(name: Token): IdentExpr =
+proc newIdentExpr*(name: Token, depth: int = 0): IdentExpr =
     result = IdentExpr(kind: identExpr)
     result.name = name
     result.token = name
+    result.depth = depth
 
 
 proc newGroupingExpr*(expression: Expression, token: Token): GroupingExpr =
@@ -368,11 +412,11 @@ proc newGroupingExpr*(expression: Expression, token: Token): GroupingExpr =
     result.token = token
 
 
-proc newLambdaExpr*(arguments: seq[tuple[name: IdentExpr, valueType: Expression,
+proc newLambdaExpr*(arguments: seq[tuple[name: IdentExpr, valueType: Expression]], defaults: seq[Expression],
-        mutable: bool, isRef: bool, isPtr: bool]], defaults: seq[Expression],
+                    body: Statement, isAsync, isGenerator: bool,
-        body: Statement, isGenerator: bool, isAsync: bool, token: Token,
+                    token: Token, depth: int, pragmas: seq[Pragma] = @[], 
-                returnType: Expression, pragmas: seq[Pragma],
+                    returnType: Expression, generics: seq[tuple[name: IdentExpr, cond: Expression]] = @[], 
-                generics: seq[tuple[name: IdentExpr, cond: Expression]]): LambdaExpr =
+                    freeVars: seq[IdentExpr] = @[]): LambdaExpr =
     result = LambdaExpr(kind: lambdaExpr)
     result.body = body
     result.arguments = arguments
@@ -384,6 +428,8 @@ proc newLambdaExpr*(arguments: seq[tuple[name: IdentExpr, valueType: Expression,
     result.isPure = false
     result.pragmas = pragmas
     result.generics = generics
+    result.freeVars = freeVars
+    result.depth = depth
 
 
 proc newGetItemExpr*(obj: Expression, name: IdentExpr,
@@ -405,15 +451,15 @@ proc newSetItemExpr*(obj: Expression, name: IdentExpr, value: Expression,
 
 proc newCallExpr*(callee: Expression, arguments: tuple[positionals: seq[
         Expression], keyword: seq[tuple[name: IdentExpr, value: Expression]]],
-        token: Token): CallExpr =
+        token: Token, genericArgs: seq[Expression] = @[]): CallExpr =
     result = CallExpr(kind: callExpr)
     result.callee = callee
     result.arguments = arguments
     result.token = token
+    result.genericArgs = @[]
 
 
-proc newSliceExpr*(expression: Expression, ends: seq[Expression],
+proc newSliceExpr*(expression: Expression, ends: seq[Expression], token: Token): SliceExpr =
-        token: Token): SliceExpr =
     result = SliceExpr(kind: sliceExpr)
     result.expression = expression
     result.ends = ends
@@ -570,10 +616,11 @@ proc newVarDecl*(name: IdentExpr, value: Expression, isConst: bool = false,
     result.pragmas = pragmas
 
 
-proc newFunDecl*(name: IdentExpr, arguments: seq[tuple[name: IdentExpr, valueType: Expression, mutable: bool, isRef: bool, isPtr: bool]], defaults: seq[Expression],
+proc newFunDecl*(name: IdentExpr, arguments: seq[tuple[name: IdentExpr, valueType: Expression]], defaults: seq[Expression],
                  body: Statement, isAsync, isGenerator: bool,
-                 isPrivate: bool, token: Token, pragmas: seq[Pragma],
+                 isPrivate: bool, token: Token, depth: int,
-                 returnType: Expression, generics: seq[tuple[name: IdentExpr, cond: Expression]]): FunDecl =
+                 pragmas: seq[Pragma] = @[], returnType: Expression, 
+                 generics: seq[tuple[name: IdentExpr, cond: Expression]] = @[], freeVars: seq[IdentExpr] = @[]): FunDecl =
     result = FunDecl(kind: funDecl)
     result.name = name
     result.arguments = arguments
@@ -587,6 +634,22 @@ proc newFunDecl*(name: IdentExpr, arguments: seq[tuple[name: IdentExpr, valueTyp
     result.returnType = returnType
     result.isPure = false
     result.generics = generics
+    result.freeVars = freeVars
+    result.depth = depth
+
+
+proc newTypeDecl*(name: IdentExpr, fields: seq[tuple[name: IdentExpr, valueType: Expression, isPrivate: bool]], 
+                  defaults: seq[Expression], isPrivate: bool, token: Token, pragmas: seq[Pragma],
+                  generics: seq[tuple[name: IdentExpr, cond: Expression]], valueType: Expression): TypeDecl =
+    result = TypeDecl(kind: typeDecl)
+    result.name = name
+    result.fields = fields
+    result.defaults = defaults
+    result.isPrivate = isPrivate
+    result.token = token
+    result.pragmas = pragmas
+    result.generics = generics
+    result.valueType = valueType
 
 
 proc `$`*(self: ASTNode): string =
@@ -669,13 +732,16 @@ proc `$`*(self: ASTNode): string =
             result &= &"AwaitStmt({self.expression})"
         of varDecl:
             var self = VarDecl(self)
-            result &= &"Var(name={self.name}, value={self.value}, const={self.isConst}, private={self.isPrivate}, type={self.valueType})"
+            result &= &"Var(name={self.name}, value={self.value}, const={self.isConst}, private={self.isPrivate}, type={self.valueType}, pragmas={self.pragmas})"
         of funDecl:
             var self = FunDecl(self)
-            result &= &"""FunDecl(name={self.name}, body={self.body}, type={self.returnType}, arguments=[{self.arguments.join(", ")}], defaults=[{self.defaults.join(", ")}], generics=[{self.generics.join(", ")}], async={self.isAsync}, generator={self.isGenerator}, private={self.isPrivate})"""
+            result &= &"""FunDecl(name={self.name}, body={self.body}, type={self.returnType}, arguments=[{self.arguments.join(", ")}], defaults=[{self.defaults.join(", ")}], generics=[{self.generics.join(", ")}], async={self.isAsync}, generator={self.isGenerator}, private={self.isPrivate}, pragmas={self.pragmas}, vars=[{self.freeVars.join(", ")}])"""
+        of typeDecl:
+            var self = TypeDecl(self)
+            result &= &"""TypeDecl(name={self.name}, fields={self.fields}, defaults={self.defaults}, private={self.isPrivate}, pragmas={self.pragmas}, generics={self.generics}, pragmas={self.pragmas}, type={self.valueType})"""
         of lambdaExpr:
             var self = LambdaExpr(self)
-            result &= &"""Lambda(body={self.body}, type={self.returnType}, arguments=[{self.arguments.join(", ")}], defaults=[{self.defaults.join(", ")}], generator={self.isGenerator}, async={self.isAsync})"""
+            result &= &"""Lambda(body={self.body}, type={self.returnType}, arguments=[{self.arguments.join(", ")}], defaults=[{self.defaults.join(", ")}], generator={self.isGenerator}, async={self.isAsync}, pragmas={self.pragmas}, vars=[{self.freeVars.join(", ")}])"""
         of deferStmt:
             var self = DeferStmt(self)
             result &= &"Defer({self.expression})"
@@ -694,6 +760,15 @@ proc `$`*(self: ASTNode): string =
             else:
                 result &= ", elseClause=nil"
             result &= ")"
+        of pragmaExpr:
+            var self = Pragma(self)
+            result &= &"Pragma(name={self.name}, args={self.args})"
+        of varExpr:
+            result &= &"Var({Var(self).value})"
+        of refExpr:
+            result &= &"Ptr({Ref(self).value})"
+        of ptrExpr:
+            result &= &"Ptr({Ptr(self).value})"
         else:
             discard
 

src/frontend/meta/bytecode.nim

@@ -14,8 +14,8 @@
 
 ## Low level bytecode implementation details
 
-import strutils
+import std/strutils
-import strformat
+import std/strformat
 
 import ../../util/multibyte
 
@@ -23,7 +23,7 @@ import ../../util/multibyte
 type
     Chunk* = ref object
         ## A piece of bytecode.
-        ## consts is used when serializing to/from a bytecode stream.
+        ## consts is the code's constants table.
         ## code is the linear sequence of compiled bytecode instructions.
         ## lines maps bytecode instructions to line numbers using Run
         ## Length Encoding. Instructions are encoded in groups whose structure
@@ -38,9 +38,20 @@ type
         ## are 3 and 4"
         ## This is more efficient than using the naive approach, which would encode
         ## the same line number multiple times and waste considerable amounts of space.
+        ## cfi represents Call Frame Information and encodes the following information:
+        ## - Function name
+        ## - Argument count
+        ## - Function boundaries
+        ## The encoding for CFI data is the following:
+        ## - First, the position into the bytecode where the function begins is encoded (as a 3 byte integer)
+        ## - Second, the position into the bytecode where the function ends is encoded (as a 3 byte integer)
+        ## - After that follows the argument count as a 1 byte integer
+        ## - Lastly, the function's name (optional) is encoded in ASCII, prepended with
+        ##   its size as a 2-byte integer
         consts*: seq[uint8]
         code*: seq[uint8]
         lines*: seq[int]
+        cfi*: seq[uint8]
 
     OpCode* {.pure.} = enum
         ## Enum of Peon's bytecode opcodes
@@ -57,7 +68,8 @@ type
         # or 24 bit numbers that are defined statically
         # at compilation time into the bytecode
 
-        # These push a constant onto the stack
+        # These push a constant at position x in the
+        # constant table onto the stack
         LoadInt64 = 0u8,
         LoadUInt64,
         LoadInt32,
@@ -69,22 +81,77 @@ type
         LoadFloat64,
         LoadFloat32,
         LoadString,
+        LoadFunction,
+        LoadReturnAddress,
         ## Singleton opcodes (each of them pushes a constant singleton on the stack)
         LoadNil,
         LoadTrue,
         LoadFalse,
         LoadNan,
         LoadInf,
+        ## Operations on primitive types
+        Negate,
+        NegateFloat64,
+        NegateFloat32,
+        Add,
+        Subtract,
+        Multiply,
+        Divide,
+        SignedDivide,
+        AddFloat64,
+        SubtractFloat64,
+        MultiplyFloat64,
+        DivideFloat64,
+        AddFloat32,
+        SubtractFloat32,
+        MultiplyFloat32,
+        DivideFloat32,
+        Pow,
+        SignedPow,
+        Mod,
+        SignedMod,
+        PowFloat64,
+        PowFloat32,
+        ModFloat64,
+        ModFloat32,
+        LShift,
+        RSHift,
+        Xor,
+        Or,
+        And,
+        Not,
+        Equal,
+        NotEqual,
+        GreaterThan,
+        LessThan,
+        GreaterOrEqual,
+        LessOrEqual,
+        ## Print opcodes
+        PrintInt64,
+        PrintUInt64,
+        PrintInt32,
+        PrintUInt32,
+        PrintInt16,
+        PrintUint16,
+        PrintInt8,
+        PrintUInt8,
+        PrintFloat64,
+        PrintFloat32,
+        PrintHex,
+        PrintBool,
+        PrintNan,
+        PrintInf,
+        PrintString,
         ## Basic stack operations
         Pop, # Pops an element off the stack and discards it
-        Push, # Pushes x onto the stack
+        PopRepl, # Same as Pop, but also prints the value of what's popped (used in REPL mode)
-        PopN, # Pops x elements off the stack (optimization for exiting local scopes which usually pop many elements)
+        PopN, # Pops x elements off the call stack (optimization for exiting local scopes which usually pop many elements)
         ## Name resolution/handling
         LoadAttribute, # Pushes the attribute b of object a onto the stack
         LoadVar, # Pushes the object at position x in the stack onto the stack
         StoreVar, # Stores the value of b at position a in the stack
-        LoadHeap, # Pushes the object position x in the closure array onto the stack
+        LoadClosure, # Pushes the object position x in the closure array onto the stack
-        StoreHeap, # Stores the value of b at position a in the closure array
+        StoreClosure, # Stores the value of b at position a in the closure array
         ## Looping and jumping
         Jump, # Absolute, unconditional jump into the bytecode
         JumpForwards, # Relative, unconditional, positive jump in the bytecode
@@ -93,18 +160,10 @@ type
         JumpIfTrue, # Jumps to a relative index in the bytecode if x is true
         JumpIfFalsePop, # Like JumpIfFalse, but also pops off the stack (regardless of truthyness). Optimization for if statements
         JumpIfFalseOrPop, # Jumps to an absolute index in the bytecode if x is false and pops otherwise (used for logical and)
-        ## Long variants of jumps (they use a 24-bit operand instead of a 16-bit one)
-        LongJump,
-        LongJumpIfFalse,
-        LongJumpIfTrue,
-        LongJumpIfFalsePop,
-        LongJumpIfFalseOrPop,
-        LongJumpForwards,
-        LongJumpBackwards,
         ## Functions
         Call, # Calls a function and initiates a new stack frame
-        Return, # Terminates the current function without popping off the stack
+        Return, # Terminates the current function
-        ReturnValue, # Pops a return value off the stack and terminates the current function
+        SetResult,  # Sets the result of the current function
         ## Exception handling
         Raise, # Raises exception x or re-raises active exception if x is nil
         BeginTry, # Initiates an exception handling context
@@ -114,21 +173,75 @@ type
         ## Coroutines
         Await, # Calls an asynchronous function
         ## Misc
-        Assert, # Raises an AssertionFailed exception if x is false
+        Assert,   # Raises an AssertionFailed exception if x is false
-        NoOp, # Just a no-op
+        NoOp,     # Just a no-op
+        PopC,     # Pop off the call stack onto the operand stack
+        PushC,    # Pop off the operand stack onto the call stack
+        SysClock64  # Pushes the output of a monotonic clock on the stack
 
 
 # We group instructions by their operation/operand types for easier handling when debugging
 
 # Simple instructions encompass instructions that push onto/pop off the stack unconditionally (True, False, Pop, etc.)
-const simpleInstructions* = {OpCode.Return, LoadNil,
+const simpleInstructions* = {Return, LoadNil,
                              LoadTrue, LoadFalse,
                              LoadNan, LoadInf,
-                             Pop, OpCode.Raise,
+                             Pop, Raise, 
-                             BeginTry, FinishTry,
+                             BeginTry, FinishTry, Yield, 
-                             OpCode.Yield, OpCode.Await,
+                             Await, NoOp, SetResult,
-                             OpCode.NoOp, OpCode.Return,
+                             PopC, PushC, SysClock64,
-                             OpCode.ReturnValue}
+                             Negate,
+                             NegateFloat64,
+                             NegateFloat32,
+                             Add,
+                             Subtract,
+                             Multiply,
+                             Divide,
+                             SignedDivide,
+                             AddFloat64,
+                             SubtractFloat64,
+                             MultiplyFloat64,
+                             DivideFloat64,
+                             AddFloat32,
+                             SubtractFloat32,
+                             MultiplyFloat32,
+                             DivideFloat32,
+                             Pow,
+                             SignedPow,
+                             Mod,
+                             SignedMod,
+                             PowFloat64,
+                             PowFloat32,
+                             ModFloat64,
+                             ModFloat32,
+                             LShift,
+                             RSHift,
+                             Xor,
+                             Or,
+                             And,
+                             Not,
+                             Equal,
+                             NotEqual,
+                             GreaterThan,
+                             LessThan,
+                             GreaterOrEqual,
+                             LessOrEqual,
+                             PrintInt64,
+                             PrintUInt64,
+                             PrintInt32,
+                             PrintUInt32,
+                             PrintInt16,
+                             PrintUint16,
+                             PrintInt8,
+                             PrintUInt8,
+                             PrintFloat64,
+                             PrintFloat32,
+                             PrintHex,
+                             PrintBool,
+                             PrintNan,
+                             PrintInf,
+                             PrintString,
+                            }
 
 # Constant instructions are instructions that operate on the bytecode constant table
 const constantInstructions* = {LoadInt64, LoadUInt64,
@@ -140,7 +253,7 @@ const constantInstructions* = {LoadInt64, LoadUInt64,
 
 # Stack triple instructions operate on the stack at arbitrary offsets and pop arguments off of it in the form
 # of 24 bit integers
-const stackTripleInstructions* = {StoreVar, LoadVar, LoadHeap, StoreHeap}
+const stackTripleInstructions* = {StoreVar, LoadVar, LoadCLosure, }
 
 # Stack double instructions operate on the stack at arbitrary offsets and pop arguments off of it in the form
 # of 16 bit integers
@@ -150,22 +263,20 @@ const stackDoubleInstructions* = {}
 const argumentDoubleInstructions* = {PopN, }
 
 # Argument double argument instructions take hardcoded arguments as 24 bit integers
-const argumentTripleInstructions* = {}
+const argumentTripleInstructions* = {StoreClosure}
 
 # Instructions that call functions
 const callInstructions* = {Call, }
 
 # Jump instructions jump at relative or absolute bytecode offsets
-const jumpInstructions* = {Jump, LongJump, JumpIfFalse, JumpIfFalsePop,
+const jumpInstructions* = {Jump, JumpIfFalse, JumpIfFalsePop,
                            JumpForwards, JumpBackwards,
-                           LongJumpIfFalse, LongJumpIfFalsePop,
+                           JumpIfTrue}
-                           LongJumpForwards, LongJumpBackwards,
-                           JumpIfTrue, LongJumpIfTrue}
 
 
 proc newChunk*: Chunk =
     ## Initializes a new, empty chunk
-    result = Chunk(consts: @[], code: @[], lines: @[])
+    result = Chunk(consts: @[], code: @[], lines: @[], cfi: @[])
 
 
 proc `$`*(self: Chunk): string = &"""Chunk(consts=[{self.consts.join(", ")}], code=[{self.code.join(", ")}], lines=[{self.lines.join(", ")}])"""

src/frontend/meta/errors.nim

@@ -11,24 +11,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import token
-import ast
 
 
 type
     ## Nim exceptions for internal Peon failures
     PeonException* = ref object of CatchableError
-    LexingError* = ref object of PeonException
-        file*: string
-        lexeme*: string
-        line*: int
-    ParseError* = ref object of PeonException
-        file*: string
-        token*: Token
-        module*: string
-    CompileError* = ref object of PeonException
-        node*: ASTNode
-        file*: string
-        module*: string
     SerializationError* = ref object of PeonException
         file*: string

src/frontend/meta/token.nim

@@ -12,7 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import strformat
+import std/strformat
+import std/strutils
 
 
 type
@@ -39,7 +40,7 @@ type
     Raise, Assert, Await, Foreach,
     Yield, Defer, Try, Except,
     Finally, Type, Operator, Case,
-    Enum, From, Ptr, Ref
+    Enum, From, Ptr, Ref, Object
 
     # Literal types
     Integer, Float, String, Identifier,
@@ -59,11 +60,9 @@ type
     NoMatch, # Used internally by the symbol table
     Comment, # Useful for documentation comments, pragmas, etc.
     Symbol, # A generic symbol
-              # These are not used at the moment but may be
+    Pragma, 
-              # employed to enforce indentation or other neat
+
-              # stuff I haven't thought about yet
+
-    Whitespace,
-    Tab,
 
 
   Token* = ref object
@@ -73,13 +72,14 @@ type
     line*: int                    # The line where the token appears
     pos*: tuple[start, stop: int] # The absolute position in the source file
                                   # (0-indexed and inclusive at the beginning)
+    spaces*: int                  # Number of spaces before this token
 
 
 
 proc `$`*(self: Token): string =
   ## Strinfifies
   if self != nil:
-    result = &"Token(kind={self.kind}, lexeme='{$(self.lexeme)}', line={self.line}, pos=({self.pos.start}, {self.pos.stop}))"
+    result = &"Token(kind={self.kind}, lexeme={self.lexeme.escape()}, line={self.line}, pos=({self.pos.start}, {self.pos.stop}), spaces={self.spaces})"
   else:
     result = "nil"
 

src/main.nim

@@ -1,8 +1,11 @@
 # Builtins & external libs
-import strformat
+import std/strformat
-import strutils
+import std/strutils
-import terminal
+import std/terminal
-import os
+import std/parseopt
+import std/times
+import std/os
+
 # Thanks art <3
 import jale/editor as ed
 import jale/templates
@@ -11,32 +14,19 @@ import jale/plugin/editor_history
 import jale/keycodes
 import jale/multiline
 
-
 # Our stuff
 import frontend/lexer as l
 import frontend/parser as p
 import frontend/compiler as c
 import backend/vm as v
 import util/serializer as s
+import util/debugger
+import util/symbols
+import config
 
 # Forward declarations
-proc fillSymbolTable(tokenizer: Lexer)
 proc getLineEditor: LineEditor
 
-# Handy dandy compile-time constants
-const debugLexer = false
-const debugParser = false
-const debugCompiler = true
-const debugSerializer = false
-const debugRuntime = false
-
-when debugSerializer:
-    import nimSHA2
-    import times
-
-when debugCompiler:
-    import util/debugger
-
 
 proc repl =
     styledEcho fgMagenta, "Welcome into the peon REPL!"
@@ -44,29 +34,44 @@ proc repl =
         keep = true
         tokens: seq[Token] = @[]
         tree: seq[Declaration] = @[]
-        compiled: Chunk
+        compiled: Chunk = newChunk()
         serialized: Serialized
         tokenizer = newLexer()
         parser = newParser()
-        compiler = newCompiler()
+        compiler = newCompiler(replMode=true)
-        serializer = newSerializer()
         vm = newPeonVM()
+        debugger = newDebugger()
+        serializer = newSerializer()
         editor = getLineEditor()
         input: string
+        current: string
+        incremental: bool = false
     tokenizer.fillSymbolTable()
     editor.bindEvent(jeQuit):
         stdout.styledWriteLine(fgGreen, "Goodbye!")
         editor.prompt = ""
         keep = false
+        input = ""
     editor.bindKey("ctrl+a"):
         editor.content.home()
     editor.bindKey("ctrl+e"):
         editor.content.`end`()
     while keep:
         try:
+            # We incrementally add content to the input
+            # so that you can, for example, define a function
+            # then press enter and use it at the next iteration
+            # of the read loop
             input = editor.read()
             if input.len() == 0:
                 continue
+            elif input == "#reset":
+                compiled = newChunk()
+                incremental = false
+                continue
+            elif input == "#clear":
+                stdout.write("\x1Bc")
+                continue
             tokens = tokenizer.lex(input, "stdin")
             if tokens.len() == 0:
                 continue
@@ -78,7 +83,7 @@ proc repl =
                         break
                     styledEcho fgGreen, "\t", $token
                 echo ""
-            tree = parser.parse(tokens, "stdin")
+            tree = parser.parse(tokens, "stdin", tokenizer.getLines(), input, persist=true)
             if tree.len() == 0:
                 continue
             when debugParser:
@@ -86,22 +91,17 @@ proc repl =
                 for node in tree:
                     styledEcho fgGreen, "\t", $node
                 echo ""
-            compiled = compiler.compile(tree, "stdin")
+            discard compiler.compile(tree, "stdin", tokenizer.getLines(), input, chunk=compiled, incremental=incremental, terminateScope=false)
+            incremental = true
             when debugCompiler:
-                styledEcho fgCyan, "Compilation step:"
+                styledEcho fgCyan, "Compilation step:\n"
-                styledEcho fgCyan, "\tRaw byte stream: ", fgGreen, "[", fgYellow, compiled.code.join(", "),  fgGreen, "]"
+                debugger.disassembleChunk(compiled, "stdin")
-                styledEcho fgCyan, "\tConstant table:  ", fgGreen, "[", fgYellow, compiled.consts.join(", "), fgGreen, "]"
-                styledEcho fgCyan, "\nBytecode disassembler output below:\n"
-                disassembleChunk(compiled, "stdin")
                 echo ""
 
-            serializer.dumpFile(compiled, input, "stdin", "stdin.pbc")
+            serialized = serializer.loadBytes(serializer.dumpBytes(compiled, "stdin"))
-            serialized = serializer.loadFile("stdin.pbc")
             when debugSerializer:
-                var hashMatches = computeSHA256(input).toHex().toLowerAscii() == serialized.fileHash
                 styledEcho fgCyan, "Serialization step: "
-                styledEcho fgBlue, &"\t- File hash: ", fgYellow, serialized.fileHash, fgBlue, " (", if hashMatches: fgGreen else: fgRed, if hashMatches: "OK" else: "Fail", fgBlue, ")"
+                styledEcho fgBlue, "\t- Peon version: ", fgYellow, &"{serialized.version.major}.{serialized.version.minor}.{serialized.version.patch}", fgBlue, " (commit ", fgYellow, serialized.commit[0..8], fgBlue, ") on branch ", fgYellow, serialized.branch
-                styledEcho fgBlue, "\t- Peon version: ", fgYellow, &"{serialized.peonVer.major}.{serialized.peonVer.minor}.{serialized.peonVer.patch}", fgBlue, " (commit ", fgYellow, serialized.commitHash[0..8], fgBlue, ") on branch ", fgYellow, serialized.peonBranch
                 stdout.styledWriteLine(fgBlue, "\t- Compilation date & time: ", fgYellow, fromUnix(serialized.compileDate).format("d/M/yyyy HH:mm:ss"))
                 stdout.styledWrite(fgBlue, &"\t- Constants segment: ")
                 if serialized.chunk.consts == compiled.consts:
@@ -118,46 +118,57 @@ proc repl =
                     styledEcho fgGreen, "OK"
                 else:
                     styledEcho fgRed, "Corrupted"
-            when debugRuntime:
+                stdout.styledWrite(fgBlue, "\t- CFI segment: ")
-                styledEcho fgCyan, "\n\nExecution step: "
+                if serialized.chunk.cfi == compiled.cfi:
+                    styledEcho fgGreen, "OK"
+                else:
+                    styledEcho fgRed, "Corrupted"
             vm.run(serialized.chunk)
         except LexingError:
-            let exc = LexingError(getCurrentException())
+            input = ""
-            let relPos = tokenizer.getRelPos(exc.line)
+            var exc = LexingError(getCurrentException())
-            let line = tokenizer.getSource().splitLines()[exc.line - 1].strip()
+            if exc.lexeme == "":
+                exc.line -= 1
+            let relPos = exc.lexer.getRelPos(exc.line)
+            let line = exc.lexer.getSource().splitLines()[exc.line - 1].strip(chars={'\n'})
             stderr.styledWriteLine(fgRed, "A fatal error occurred while parsing ", fgYellow, &"'{exc.file}'", fgRed, ", module ",
-                                   fgYellow, &"'{exc.file.extractFilename()}'", fgRed, ", line ", fgYellow, $exc.line, fgRed, " at ", fgYellow, &"'{exc.lexeme}'",
+                                    fgYellow, &"'{exc.file.extractFilename()}'", fgRed, ", line ", fgYellow, $exc.line, fgRed, " at ", fgYellow, &"'{exc.lexeme.escape()}'",
-                                   fgRed, ": ", fgGreen , getCurrentExceptionMsg())
+                                    fgRed, ": ", fgGreen , getCurrentExceptionMsg())
             styledEcho fgBlue, "Source line: " , fgDefault, line
             styledEcho fgCyan, " ".repeat(len("Source line: ")) & "^".repeat(relPos.stop - relPos.start)
         except ParseError:
+            input = ""
             let exc = ParseError(getCurrentException())
             let lexeme = exc.token.lexeme
-            let lineNo = exc.token.line
+            var lineNo = exc.token.line
-            let relPos = tokenizer.getRelPos(lineNo)
+            if exc.token.kind == EndOfFile:
+                lineNo -= 1
+            let relPos = exc.parser.getRelPos(lineNo)
             let fn = parser.getCurrentFunction()
-            let line = tokenizer.getSource().splitLines()[lineNo - 1].strip()
+            let line = exc.parser.getSource().splitLines()[lineNo - 1].strip(chars={'\n'})
             var fnMsg = ""
             if fn != nil and fn.kind == funDecl:
                 fnMsg &= &"in function '{FunDecl(fn).name.token.lexeme}'"
             stderr.styledWriteLine(fgRed, "A fatal error occurred while parsing ", fgYellow, &"'{exc.file}'", fgRed, ", module ",
-                                   fgYellow, &"'{exc.file}'", fgRed, ", line ", fgYellow, $lineNo, fgRed, " at ", fgYellow, &"'{lexeme}'",
+                                    fgYellow, &"'{exc.module}'", fgRed, ", line ", fgYellow, $lineNo, fgRed, " at ", fgYellow, &"'{lexeme}'",
-                                   fgRed, ": ", fgGreen , getCurrentExceptionMsg())
+                                    fgRed, ": ", fgGreen , getCurrentExceptionMsg())
             styledEcho fgBlue, "Source line: " , fgDefault, line
             styledEcho fgCyan, " ".repeat(len("Source line: ")) & "^".repeat(relPos.stop - relPos.start)
         except CompileError:
             let exc = CompileError(getCurrentException())
             let lexeme = exc.node.token.lexeme
-            let lineNo = exc.node.token.line
+            var lineNo = exc.node.token.line
-            let relPos = tokenizer.getRelPos(lineNo)
+            if exc.node.token.kind == EndOfFile:
-            let line = tokenizer.getSource().splitLines()[lineNo - 1].strip()
+                lineNo -= 1
-            var fn = compiler.getCurrentFunction()
+            let relPos = exc.compiler.getRelPos(lineNo)
+            let line = exc.compiler.getSource().splitLines()[lineNo - 1].strip(chars={'\n'})
+            var fn = exc.compiler.getCurrentFunction()
             var fnMsg = ""
             if fn != nil and fn.kind == funDecl:
                 fnMsg &= &"in function '{FunDecl(fn).name.token.lexeme}'"
             stderr.styledWriteLine(fgRed, "A fatal error occurred while compiling ", fgYellow, &"'{exc.file}'", fgRed, ", module ",
-                                   fgYellow, &"'{exc.module}'", fgRed, ", line ", fgYellow, $lineNo, fgRed, " at ", fgYellow, &"'{lexeme}'",
+                                    fgYellow, &"'{exc.module}'", fgRed, ", line ", fgYellow, $lineNo, fgRed, " at ", fgYellow, &"'{lexeme}'",
-                                   fgRed, ": ", fgGreen , getCurrentExceptionMsg())
+                                    fgRed, ": ", fgGreen , getCurrentExceptionMsg())
             styledEcho fgBlue, "Source line: " , fgDefault, line
             styledEcho fgCyan, " ".repeat(len("Source line: ")) & "^".repeat(relPos.stop - relPos.start)
         except SerializationError:
@@ -166,7 +177,7 @@ proc repl =
     quit(0)
 
 
-proc runFile(f: string) =
+proc runFile(f: string, interactive: bool = false, fromString: bool = false) =
     var
         tokens: seq[Token] = @[]
         tree: seq[Declaration] = @[]
@@ -175,12 +186,20 @@ proc runFile(f: string) =
         tokenizer = newLexer()
         parser = newParser()
         compiler = newCompiler()
+        debugger {.used.} = newDebugger()
         serializer = newSerializer()
         vm = newPeonVM()
         input: string
     tokenizer.fillSymbolTable()
     try:
-        input = readFile(f)
+        var f = f
+        if not fromString:
+            if not f.endsWith(".pn"):
+                f &= ".pn"
+            input = readFile(f)
+        else:
+            input = f
+            f = "<string>"
         tokens = tokenizer.lex(input, f)
         if tokens.len() == 0:
             return
@@ -192,7 +211,7 @@ proc runFile(f: string) =
                     break
                 styledEcho fgGreen, "\t", $token
             echo ""
-        tree = parser.parse(tokens, f)
+        tree = parser.parse(tokens, f, tokenizer.getLines(), input)
         if tree.len() == 0:
             return
         when debugParser:
@@ -200,22 +219,22 @@ proc runFile(f: string) =
             for node in tree:
                 styledEcho fgGreen, "\t", $node
             echo ""
-        compiled = compiler.compile(tree, f)
+        compiled = compiler.compile(tree, f, tokenizer.getLines(), input)
         when debugCompiler:
-            styledEcho fgCyan, "Compilation step:"
+            styledEcho fgCyan, "Compilation step:\n"
-            styledEcho fgCyan, "\tRaw byte stream: ", fgGreen, "[", fgYellow, compiled.code.join(", "),  fgGreen, "]"
+            debugger.disassembleChunk(compiled, f)
-            styledEcho fgCyan, "\tConstant table:  ", fgGreen, "[", fgYellow, compiled.consts.join(", "), fgGreen, "]"
-            styledEcho fgCyan, "\nBytecode disassembler output below:\n"
-            disassembleChunk(compiled, f)
             echo ""
-
+        var path = splitFile(f).dir
-        serializer.dumpFile(compiled, input, f, splitFile(f).name & ".pbc")
+        if path.len() > 0:
-        serialized = serializer.loadFile(splitFile(f).name & ".pbc")
+            path &= "/"
+        path &= splitFile(f).name & ".pbc"
+        serializer.dumpFile(compiled, f, path)
+        serialized = serializer.loadFile(path)
+        if fromString:
+            discard tryRemoveFile("<string>.pbc")
         when debugSerializer:
-            var hashMatches = computeSHA256(input).toHex().toLowerAscii() == serialized.fileHash
             styledEcho fgCyan, "Serialization step: "
-            styledEcho fgBlue, &"\t- File hash: ", fgYellow, serialized.fileHash, fgBlue, " (", if hashMatches: fgGreen else: fgRed, if hashMatches: "OK" else: "Fail", fgBlue, ")"
+            styledEcho fgBlue, "\t- Peon version: ", fgYellow, &"{serialized.version.major}.{serialized.version.minor}.{serialized.version.patch}", fgBlue, " (commit ", fgYellow, serialized.commit[0..8], fgBlue, ") on branch ", fgYellow, serialized.branch
-            styledEcho fgBlue, "\t- Peon version: ", fgYellow, &"{serialized.peonVer.major}.{serialized.peonVer.minor}.{serialized.peonVer.patch}", fgBlue, " (commit ", fgYellow, serialized.commitHash[0..8], fgBlue, ") on branch ", fgYellow, serialized.peonBranch
             stdout.styledWriteLine(fgBlue, "\t- Compilation date & time: ", fgYellow, fromUnix(serialized.compileDate).format("d/M/yyyy HH:mm:ss"))
             stdout.styledWrite(fgBlue, &"\t- Constants segment: ")
             if serialized.chunk.consts == compiled.consts:
@@ -232,40 +251,49 @@ proc runFile(f: string) =
                 styledEcho fgGreen, "OK"
             else:
                 styledEcho fgRed, "Corrupted"
-        when debugRuntime:
+            stdout.styledWrite(fgBlue, "\t- CFI segment: ")
-            styledEcho fgCyan, "\n\nExecution step: "
+            if serialized.chunk.cfi == compiled.cfi:
+                styledEcho fgGreen, "OK"
+            else:
+                styledEcho fgRed, "Corrupted"
         vm.run(serialized.chunk)
     except LexingError:
-        let exc = LexingError(getCurrentException())
+        var exc = LexingError(getCurrentException())
-        let relPos = tokenizer.getRelPos(exc.line)
+        if exc.lexeme == "":
-        let line = tokenizer.getSource().splitLines()[exc.line - 1].strip()
+            exc.line -= 1
+        let relPos = exc.lexer.getRelPos(exc.line)
+        let line = exc.lexer.getSource().splitLines()[exc.line - 1].strip(chars={'\n'})
         stderr.styledWriteLine(fgRed, "A fatal error occurred while parsing ", fgYellow, &"'{exc.file}'", fgRed, ", module ",
-                                fgYellow, &"'{exc.file}'", fgRed, ", line ", fgYellow, $exc.line, fgRed, " at ", fgYellow, &"'{exc.lexeme}'",
+                                fgYellow, &"'{exc.file.extractFilename()}'", fgRed, ", line ", fgYellow, $exc.line, fgRed, " at ", fgYellow, &"'{exc.lexeme.escape()}'",
                                 fgRed, ": ", fgGreen , getCurrentExceptionMsg())
         styledEcho fgBlue, "Source line: " , fgDefault, line
         styledEcho fgCyan, " ".repeat(len("Source line: ")) & "^".repeat(relPos.stop - relPos.start)
     except ParseError:
         let exc = ParseError(getCurrentException())
         let lexeme = exc.token.lexeme
-        let lineNo = exc.token.line
+        var lineNo = exc.token.line
-        let relPos = tokenizer.getRelPos(lineNo)
+        if exc.token.kind == EndOfFile:
+            lineNo -= 1
+        let relPos = exc.parser.getRelPos(lineNo)
         let fn = parser.getCurrentFunction()
-        let line = tokenizer.getSource().splitLines()[lineNo - 1].strip()
+        let line = exc.parser.getSource().splitLines()[lineNo - 1].strip(chars={'\n'})
         var fnMsg = ""
         if fn != nil and fn.kind == funDecl:
             fnMsg &= &"in function '{FunDecl(fn).name.token.lexeme}'"
         stderr.styledWriteLine(fgRed, "A fatal error occurred while parsing ", fgYellow, &"'{exc.file}'", fgRed, ", module ",
-                                fgYellow, &"'{exc.file}'", fgRed, ", line ", fgYellow, $lineNo, fgRed, " at ", fgYellow, &"'{lexeme}'",
+                                fgYellow, &"'{exc.module}'", fgRed, ", line ", fgYellow, $lineNo, fgRed, " at ", fgYellow, &"'{lexeme}'",
                                 fgRed, ": ", fgGreen , getCurrentExceptionMsg())
         styledEcho fgBlue, "Source line: " , fgDefault, line
         styledEcho fgCyan, " ".repeat(len("Source line: ")) & "^".repeat(relPos.stop - relPos.start)
     except CompileError:
         let exc = CompileError(getCurrentException())
         let lexeme = exc.node.token.lexeme
-        let lineNo = exc.node.token.line
+        var lineNo = exc.node.token.line
-        let relPos = tokenizer.getRelPos(lineNo)
+        if exc.node.token.kind == EndOfFile:
-        let line = tokenizer.getSource().splitLines()[lineNo - 1].strip()
+            lineNo -= 1
-        var fn = compiler.getCurrentFunction()
+        let relPos = exc.compiler.getRelPos(lineNo)
+        let line = exc.compiler.getSource().splitLines()[lineNo - 1].strip(chars={'\n'})
+        var fn = exc.compiler.getCurrentFunction()
         var fnMsg = ""
         if fn != nil and fn.kind == funDecl:
             fnMsg &= &"in function '{FunDecl(fn).name.token.lexeme}'"
@@ -283,73 +311,59 @@ proc runFile(f: string) =
         stderr.styledWriteLine(fgRed, "An error occurred while trying to read ", fgYellow, &"'{f}'", fgGreen, &": {osErrorMsg(osLastError())} [errno {osLastError()}]")
 
 
+
 when isMainModule:
     setControlCHook(proc () {.noconv.} = quit(0))
-    let args = commandLineParams()
+    var optParser = initOptParser(commandLineParams())
-    if args.len() == 0:
+    var file: string = ""
+    var fromString: bool = false
+    var interactive: bool = false
+    for kind, key, value in optParser.getopt():
+        case kind:
+            of cmdArgument:
+                file = key
+            of cmdLongOption:
+                case key:
+                    of "help":
+                        echo HELP_MESSAGE
+                        quit()
+                    of "version":
+                        echo PEON_VERSION_STRING
+                        quit()
+                    of "string":
+                        file = key
+                        fromString = true
+                    of "interactive":
+                        interactive = true
+                    else:
+                        echo &"error: unkown option '{key}'"
+                        quit()
+            of cmdShortOption:
+                case key:
+                    of "h":
+                        echo HELP_MESSAGE
+                        quit()
+                    of "v":
+                        echo PEON_VERSION_STRING
+                        quit()
+                    of "s":
+                        file = key
+                        fromString = true
+                    of "i":
+                        interactive = true
+                    else:
+                        echo &"error: unkown option '{key}'"
+                        quit()
+            else:
+                echo "usage: peon [options] [filename.pn]"
+                quit()
+    # TODO: Use interactive
+    if file == "":
         repl()
     else:
-        runFile(args[0])
+        runFile(file, interactive, fromString)
 
 
-proc fillSymbolTable(tokenizer: Lexer) =
-    ## Initializes the Lexer's symbol
-    ## table with the builtin symbols
-    ## and keywords
-
-    # 1-byte symbols
-    tokenizer.symbols.addSymbol("{", LeftBrace)
-    tokenizer.symbols.addSymbol("}", RightBrace)
-    tokenizer.symbols.addSymbol("(", LeftParen)
-    tokenizer.symbols.addSymbol(")", RightParen)
-    tokenizer.symbols.addSymbol("[", LeftBracket)
-    tokenizer.symbols.addSymbol("]", RightBracket)
-    tokenizer.symbols.addSymbol(".", Dot)
-    tokenizer.symbols.addSymbol(",", Comma)
-    tokenizer.symbols.addSymbol(";", Semicolon)
-    # Keywords
-    tokenizer.symbols.addKeyword("type", TokenType.Type)
-    tokenizer.symbols.addKeyword("enum", Enum)
-    tokenizer.symbols.addKeyword("case", Case)
-    tokenizer.symbols.addKeyword("operator", Operator)
-    tokenizer.symbols.addKeyword("generator", Generator)
-    tokenizer.symbols.addKeyword("fn", TokenType.Function)
-    tokenizer.symbols.addKeyword("coroutine", Coroutine)
-    tokenizer.symbols.addKeyword("break", TokenType.Break)
-    tokenizer.symbols.addKeyword("continue", Continue)
-    tokenizer.symbols.addKeyword("while", While)
-    tokenizer.symbols.addKeyword("for", For)
-    tokenizer.symbols.addKeyword("foreach", Foreach)
-    tokenizer.symbols.addKeyword("if", If)
-    tokenizer.symbols.addKeyword("else", Else)
-    tokenizer.symbols.addKeyword("await", TokenType.Await)
-    tokenizer.symbols.addKeyword("defer", Defer)
-    tokenizer.symbols.addKeyword("try", Try)
-    tokenizer.symbols.addKeyword("except", Except)
-    tokenizer.symbols.addKeyword("finally", Finally)
-    tokenizer.symbols.addKeyword("raise", TokenType.Raise)
-    tokenizer.symbols.addKeyword("assert", TokenType.Assert)
-    tokenizer.symbols.addKeyword("const", Const)
-    tokenizer.symbols.addKeyword("let", Let)
-    tokenizer.symbols.addKeyword("var", Var)
-    tokenizer.symbols.addKeyword("import", Import)
-    tokenizer.symbols.addKeyword("yield", TokenType.Yield)
-    tokenizer.symbols.addKeyword("return", TokenType.Return)
-    # These are more like expressions with a reserved
-    # name that produce a value of a builtin type,
-    # but we don't need to care about that until
-    # we're in the parsing/ compilation steps so
-    # it's fine
-    tokenizer.symbols.addKeyword("nan", NotANumber)
-    tokenizer.symbols.addKeyword("inf", Infinity)
-    tokenizer.symbols.addKeyword("nil", TokenType.Nil)
-    tokenizer.symbols.addKeyword("true", True)
-    tokenizer.symbols.addKeyword("false", False)
-    tokenizer.symbols.addKeyword("ref", Ref)
-    tokenizer.symbols.addKeyword("ptr", Ptr)
-    for sym in [">", "<", "=", "~", "/", "+", "-", "_", "*", "?", "@", ":"]:
-        tokenizer.symbols.addSymbol(sym, Symbol)
-
 
 proc getLineEditor: LineEditor =
     result = newLineEditor()

src/memory/allocator.nim

@@ -15,29 +15,29 @@
 ## Memory allocator from JAPL
 
 
-import segfaults
+import std/segfaults
 import ../config
 
-when DEBUG_TRACE_ALLOCATION:
+when debugMem:
-    import strformat
+    import std/strformat
 
 
 proc reallocate*(p: pointer, oldSize: int, newSize: int): pointer =
-    ## Wrapper around realloc/dealloc
+    ## Simple wrapper around realloc/dealloc
     try:
         if newSize == 0 and p != nil:
-            when DEBUG_TRACE_ALLOCATION:
+            when debugMem:
                 if oldSize > 1:
                     echo &"DEBUG - Memory manager: Deallocating {oldSize} bytes"
                 else:
                     echo "DEBUG - Memory manager: Deallocating 1 byte"
             dealloc(p)
             return nil
-        when DEBUG_TRACE_ALLOCATION:
+        when debugMem:
             if pointr == nil and newSize == 0:
                 echo &"DEBUG - Memory manager: Warning, asked to dealloc() nil pointer from {oldSize} to {newSize} bytes, ignoring request"
         if oldSize > 0 and p != nil or oldSize == 0:
-            when DEBUG_TRACE_ALLOCATION:
+            when debugMem:
                 if oldSize == 0:
                     if newSize > 1:
                         echo &"DEBUG - Memory manager: Allocating {newSize} bytes of memory"
@@ -46,40 +46,35 @@ proc reallocate*(p: pointer, oldSize: int, newSize: int): pointer =
                 else:
                     echo &"DEBUG - Memory manager: Resizing {oldSize} bytes of memory to {newSize} bytes"
             result = realloc(p, newSize)
-        when DEBUG_TRACE_ALLOCATION:
+        when debugMem:
             if oldSize > 0 and pointr == nil:
                 echo &"DEBUG - Memory manager: Warning, asked to realloc() nil pointer from {oldSize} to {newSize} bytes, ignoring request"
     except NilAccessDefect:
-        stderr.write("JAPL: could not manage memory, segmentation fault\n")
+        stderr.write("Peon: could not manage memory, segmentation fault\n")
         quit(139) # For now, there's not much we can do if we can't get the memory we need, so we exit
 
 
-template resizeArray*(kind: untyped, pointr: pointer, oldCount,
+template resizeArray*(kind: untyped, p: pointer, oldCount, newCount: int): untyped =
-        newCount: int): untyped =
+    ## Handy template to resize a dynamic array
-    ## Handy macro (in the C sense of macro, not nim's) to resize a dynamic array
+    cast[ptr UncheckedArray[kind]](reallocate(p, sizeof(kind) * oldCount, sizeof(kind) * newCount))
-    cast[ptr UncheckedArray[kind]](reallocate(pointr, sizeof(kind) * oldCount,
-            sizeof(kind) * newCount))
 
 
-template freeArray*(kind: untyped, pointr: pointer, oldCount: int): untyped =
+template freeArray*(kind: untyped, p: pointer, oldCount: int): untyped =
     ## Frees a dynamic array
-    reallocate(pointr, sizeof(kind) * oldCount, 0)
+    reallocate(p, sizeof(kind) * oldCount, 0)
 
 
-template free*(kind: untyped, pointr: pointer): untyped =
+template free*(kind: untyped, p: pointer): untyped =
     ## Frees a pointer by reallocating its
     ## size to 0
-    reallocate(pointr, sizeof(kind), 0)
+    reallocate(p, sizeof(kind), 0)
 
 
 template growCapacity*(capacity: int): untyped =
-    ## Handy macro used to calculate how much
+    ## Handy template used to calculate how much
     ## more memory is needed when reallocating
     ## dynamic arrays
-    if capacity < 8:
+    if capacity < 8: 8 else: capacity * HeapGrowFactor
-        8
-    else:
-        capacity * ARRAY_GROW_FACTOR
 
 
 template allocate*(castTo: untyped, sizeTo: untyped, count: int): untyped =

src/peon/stdlib/arithmetics.pn

@@ -1,193 +0,0 @@
-## Builtin arithmetic operators for Peon
-
-
-operator `+`(a, b: int): int {
-    #pragma[magic: AddInt64, pure]
-    return;
-}
-
-
-operator `+`(a, b: uint): uint {
-    #pragma[magic: AddUInt64, pure]
-    return;
-}
-
-
-operator `+`(a, b: int32): int32 {
-    #pragma[magic: AddInt32, pure]
-    return;
-}
-
-
-operator `+`(a, b: uint32): uint32 {
-    #pragma[magic: AddUInt32, pure]
-    return;
-}
-
-
-operator `+`(a, b: int16): int16 {
-    #pragma[magic: AddInt16, pure]
-    return;
-}
-
-
-operator `+`(a, b: uint16): uint16 {
-    #pragma[magic: AddUInt16, pure]
-    return;
-}
-
-
-operator `+`(a, b: int8): int8 {
-    #pragma[magic: AddInt8, pure]
-    return;
-}
-
-
-operator `+`(a, b: uint8): uint8 {
-    #pragma[magic: AddUInt8, pure]
-    return;
-}
-
-
-operator `-`(a, b: int): int {
-    #pragma[magic: SubInt64, pure]
-    return;
-}
-
-
-operator `-`(a, b: uint): uint {
-    #pragma[magic: SubUInt64, pure]
-    return;
-}
-
-
-operator `-`(a, b: int32): int32 {
-    #pragma[magic: SubInt32, pure]
-    return;
-}
-
-
-operator `-`(a, b: uint32): uint32 {
-    #pragma[magic: SubUInt32, pure]
-    return;
-}
-
-
-operator `-`(a, b: int16): int16 {
-    #pragma[magic: SubInt16, pure]
-    return;
-}
-
-
-operator `-`(a, b: uint16): uint16 {
-    #pragma[magic: SubUInt16, pure]
-    return;
-}
-
-
-operator `-`(a, b: int8): int8 {
-    #pragma[magic: SubInt8, pure]
-    return;
-}
-
-
-operator `-`(a, b: uint8): uint8 {
-    #pragma[magic: SubUInt8, pure]
-    return;
-}
-
-
-operator `*`(a, b: int): int {
-    #pragma[magic: MulInt64, pure]
-    return;
-}
-
-
-operator `*`(a, b: uint): uint {
-    #pragma[magic: MulUInt64, pure]
-    return;
-}
-
-
-operator `*`(a, b: int32): int32 {
-    #pragma[magic: MulInt32, pure]
-    return;
-}
-
-
-operator `*`(a, b: uint32): uint32 {
-    #pragma[magic: MulUInt32, pure]
-    return;
-}
-
-
-operator `*`(a, b: int16): int16 {
-    #pragma[magic: MulInt16, pure]
-    return;
-}
-
-
-operator `*`(a, b: uint16): uint16 {
-    #pragma[magic: MulUInt16, pure]
-    return;
-}
-
-
-operator `*`(a, b: int8): int8 {
-    #pragma[magic: MulInt8, pure]
-    return;
-}
-
-
-operator `*`(a, b: uint8): uint8 {
-    #pragma[magic: MulUInt8, pure]
-    return;
-}
-
-
-operator `/`(a, b: int): int {
-    #pragma[magic: DivInt64, pure]
-    return;
-}
-
-
-operator `/`(a, b: uint): uint {
-    #pragma[magic: DivUInt64, pure]
-    return;
-}
-
-
-operator `/`(a, b: int32): int32 {
-    #pragma[magic: DivInt32, pure]
-    return;
-}
-
-
-operator `/`(a, b: uint32): uint32 {
-    #pragma[magic: DivUInt32, pure]
-    return;
-}
-
-
-operator `/`(a, b: int16): int16 {
-    #pragma[magic: DivInt16, pure]
-    return;
-}
-
-
-operator `/`(a, b: uint16): uint16 {
-    #pragma[magic: DivUInt16, pure]
-    return;
-}
-
-
-operator `/`(a, b: int8): int8 {
-    #pragma[magic: DivInt8, pure]
-    return;
-}
-
-
-operator `/`(a, b: uint8): uint8 {
-    #pragma[magic: DivUInt8, pure]
-    return;
-}

src/util/debugger.nim

@@ -16,9 +16,27 @@ import ../frontend/meta/bytecode
 import multibyte
 
 
-import strformat
+import std/strformat
-import strutils
+import std/strutils
-import terminal
+import std/terminal
+
+
+type 
+    CFIElement = ref object
+        start, stop, bottom, argc: int
+        name: string
+        started, stopped: bool
+    Debugger* = ref object
+        chunk: Chunk
+        cfiData: seq[CFIElement]
+        current: int
+
+
+proc newDebugger*: Debugger =
+    ## Initializes a new, empty
+    ## debugger object
+    new(result)
+    result.cfiData = @[]
 
 
 proc nl = stdout.write("\n")
@@ -44,131 +62,195 @@ proc printInstruction(instruction: OpCode, newline: bool = false) =
         nl()
 
 
-proc simpleInstruction(instruction: OpCode, offset: int): int =
+proc checkFrameStart(self: Debugger, n: int) =
-    printInstruction(instruction)
+    ## Checks if a call frame begins at the given
-    nl()
+    ## bytecode offset
-    return offset + 1
+    for i, e in self.cfiData:
+        if n == e.start and not (e.started or e.stopped):
+            e.started = true
+            styledEcho fgBlue, "\n==== Peon Bytecode Debugger - Begin Frame ", fgYellow, &"'{e.name}' ", fgBlue, "(", fgYellow, $i, fgBlue, ") ===="
+            styledEcho fgGreen, "\t- Start offset: ", fgYellow, $e.start
+            styledEcho fgGreen, "\t- End offset: ", fgYellow, $e.stop
+            styledEcho fgGreen, "\t- Argument count: ", fgYellow, $e.argc
 
 
-proc stackTripleInstruction(instruction: OpCode, chunk: Chunk,
+proc checkFrameEnd(self: Debugger, n: int) =
-        offset: int): int =
+    ## Checks if a call frame ends at the given
+    ## bytecode offset
+    for i, e in self.cfiData:
+        if n == e.stop and e.started and not e.stopped:
+            e.stopped = true
+            styledEcho fgBlue, "\n==== Peon Bytecode Debugger - End Frame ", fgYellow, &"'{e.name}' ", fgBlue, "(", fgYellow, $i, fgBlue, ") ===="
+    
+
+proc simpleInstruction(self: Debugger, instruction: OpCode) =
+    ## Debugs simple instructions
+    printInstruction(instruction, true)
+    self.current += 1
+    if instruction == Return:
+        printDebug("Void: ")
+        if self.chunk.code[self.current] == 0:
+            stdout.styledWriteLine(fgYellow, "Yes")
+        else:
+            stdout.styledWriteLine(fgYellow, "No")
+        self.current += 1
+        self.checkFrameEnd(self.current - 2)
+        self.checkFrameEnd(self.current - 1)
+        self.checkFrameEnd(self.current)
+
+
+
+proc stackTripleInstruction(self: Debugger, instruction: OpCode) =
     ## Debugs instructions that operate on a single value on the stack using a 24-bit operand
-    var slot = [chunk.code[offset + 1], chunk.code[offset + 2], chunk.code[
+    var slot = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple()
-            offset + 3]].fromTriple()
     printInstruction(instruction)
-    stdout.styledWrite(fgGreen, &", points to index ")
+    stdout.styledWriteLine(fgGreen, &", points to index ", fgYellow, $slot)
-    stdout.styledWriteLine(fgYellow, &"{slot}")
+    self.current += 4
-    return offset + 4
 
 
-proc stackDoubleInstruction(instruction: OpCode, chunk: Chunk,
+proc stackDoubleInstruction(self: Debugger, instruction: OpCode) =
-        offset: int): int =
     ## Debugs instructions that operate on a single value on the stack using a 16-bit operand
-    var slot = [chunk.code[offset + 1], chunk.code[offset + 2]].fromDouble()
+    var slot = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2]].fromDouble()
     printInstruction(instruction)
     stdout.write(&", points to index ")
-    stdout.styledWrite(fgGreen, &", points to index ")
+    stdout.styledWriteLine(fgGreen, &", points to index ", fgYellow, $slot)
-    stdout.styledWriteLine(fgYellow, &"{slot}")
+    self.current += 3
-    return offset + 3
 
 
-proc argumentDoubleInstruction(instruction: OpCode, chunk: Chunk, offset: int): int =
+proc argumentDoubleInstruction(self: Debugger, instruction: OpCode) =
     ## Debugs instructions that operate on a hardcoded value on the stack using a 16-bit operand
-    var slot = [chunk.code[offset + 1], chunk.code[offset + 2]].fromDouble()
+    var slot = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2]].fromDouble()
     printInstruction(instruction)
-    stdout.styledWrite(fgGreen, &", has argument ")
+    stdout.styledWriteLine(fgGreen, &", has argument ", fgYellow, $slot)
-    stdout.styledWriteLine(fgYellow, $slot)
+    self.current += 3
-    return offset + 3
 
 
-proc argumentTripleInstruction(instruction: OpCode, chunk: Chunk, offset: int): int =
+proc argumentTripleInstruction(self: Debugger, instruction: OpCode) =
     ## Debugs instructions that operate on a hardcoded value on the stack using a 24-bit operand
-    var slot = [chunk.code[offset + 1], chunk.code[offset + 2], chunk.code[offset + 3]].fromTriple()
+    var slot = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple()
     printInstruction(instruction)
-    stdout.styledWrite(fgGreen, ", has argument ")
+    stdout.styledWriteLine(fgGreen, ", has argument ", fgYellow, $slot)
-    stdout.styledWriteLine(fgYellow, $slot)
+    self.current += 4
-    return offset + 4
 
 
-proc callInstruction(instruction: OpCode, chunk: Chunk, offset: int): int =
+proc callInstruction(self: Debugger, instruction: OpCode) =
     ## Debugs function calls
-    var slot = [chunk.code[offset + 1], chunk.code[offset + 2], chunk.code[offset + 3]].fromTriple()
+    var size = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple()
-    var args = [chunk.code[offset + 4], chunk.code[offset + 5], chunk.code[offset + 6]].fromTriple()
     printInstruction(instruction)
-    stdout.styledWrite(fgGreen, &", jumps to address ", fgYellow, $slot, fgGreen, " with ", fgYellow, $args, fgGreen, " argument")
+    styledEcho fgGreen, &", creates frame of size ", fgYellow, $(size + 2)
-    if args > 1:
+    self.current += 4
-        stdout.styledWrite(fgYellow, "s")
-    nl()
-    return offset + 7
 
 
-proc constantInstruction(instruction: OpCode, chunk: Chunk, offset: int): int =
+proc functionInstruction(self: Debugger, instruction: OpCode) =
+    ## Debugs function calls
+    var address = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple()
+    printInstruction(instruction)
+    styledEcho fgGreen, &", loads function at address ", fgYellow, $address
+    self.current += 4
+
+
+proc loadAddressInstruction(self: Debugger, instruction: OpCode) =
+    ## Debugs LoadReturnAddress instructions
+    var address = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3],
+                   self.chunk.code[self.current + 4]].fromQuad()
+    printInstruction(instruction)
+    styledEcho fgGreen, &", loads address ", fgYellow, $address
+    self.current += 5
+
+
+proc constantInstruction(self: Debugger, instruction: OpCode) =
     ## Debugs instructions that operate on the constant table
-    var constant = [chunk.code[offset + 1], chunk.code[offset + 2], chunk.code[
+    var size: uint
-            offset + 3]].fromTriple()
+    if instruction == LoadString:
+        size = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple()
+        self.current += 3
+    var constant = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple()
     printInstruction(instruction)
     stdout.styledWrite(fgGreen, &", points to constant at position ", fgYellow, $constant)
-    nl()
+    self.current += 4
-    printDebug("Operand: ")
+    if instruction == LoadString:
-    stdout.styledWriteLine(fgYellow, &"{chunk.consts[constant]}")
+        stdout.styledWriteLine(fgGreen, " of length ", fgYellow, $size)
-    return offset + 4
+    else:
+        stdout.write("\n")
 
 
-proc jumpInstruction(instruction: OpCode, chunk: Chunk, offset: int): int =
+proc jumpInstruction(self: Debugger, instruction: OpCode) =
     ## Debugs jumps
-    var jump: int
+    var orig = self.current
-    case instruction:
+    var jump = [self.chunk.code[self.current + 1], self.chunk.code[self.current + 2], self.chunk.code[self.current + 3]].fromTriple().int()
-        of Jump, JumpIfFalse, JumpIfTrue, JumpIfFalsePop, JumpForwards, JumpBackwards:
-            jump = [chunk.code[offset + 1], chunk.code[offset + 2]].fromDouble().int()
-        of LongJump, LongJumpIfFalse, LongJumpIfTrue, LongJumpIfFalsePop,
-                LongJumpForwards, LongJumpBackwards:
-            jump = [chunk.code[offset + 1], chunk.code[offset + 2], chunk.code[
-                    offset + 3]].fromTriple().int()
-        else:
-            discard # Unreachable
     printInstruction(instruction, true)
     printDebug("Jump size: ")
     stdout.styledWrite(fgYellow, $jump)
     nl()
-    return offset + 3
+    self.current += 4
+    for i in countup(orig, self.current + 1):
+        self.checkFrameStart(i)
 
 
-proc disassembleInstruction*(chunk: Chunk, offset: int): int =
+proc disassembleInstruction*(self: Debugger) =
     ## Takes one bytecode instruction and prints it
     printDebug("Offset: ")
-    stdout.styledWriteLine(fgYellow, $offset)
+    stdout.styledWriteLine(fgYellow, $(self.current))
     printDebug("Line: ")
-    stdout.styledWriteLine(fgYellow, &"{chunk.getLine(offset)}")
+    stdout.styledWriteLine(fgYellow, &"{self.chunk.getLine(self.current)}")
-    var opcode = OpCode(chunk.code[offset])
+    var opcode = OpCode(self.chunk.code[self.current])
     case opcode:
         of simpleInstructions:
-            result = simpleInstruction(opcode, offset)
+            self.simpleInstruction(opcode)
         of constantInstructions:
-            result = constantInstruction(opcode, chunk, offset)
+            self.constantInstruction(opcode)
         of stackDoubleInstructions:
-            result = stackDoubleInstruction(opcode, chunk, offset)
+            self.stackDoubleInstruction(opcode)
         of stackTripleInstructions:
-            result = stackTripleInstruction(opcode, chunk, offset)
+            self.stackTripleInstruction(opcode)
         of argumentDoubleInstructions:
-            result = argumentDoubleInstruction(opcode, chunk, offset)
+            self.argumentDoubleInstruction(opcode)
         of argumentTripleInstructions:
-            result = argumentTripleInstruction(opcode, chunk, offset)
+            self.argumentTripleInstruction(opcode)
         of callInstructions:
-            result = callInstruction(opcode, chunk, offset)
+            self.callInstruction(opcode)
         of jumpInstructions:
-            result = jumpInstruction(opcode, chunk, offset)
+            self.jumpInstruction(opcode)
+        of LoadFunction:
+            self.functionInstruction(opcode)
+        of LoadReturnAddress:
+            self.loadAddressInstruction(opcode)
         else:
-            echo &"DEBUG - Unknown opcode {opcode} at index {offset}"
+            echo &"DEBUG - Unknown opcode {opcode} at index {self.current}"
-            result = offset + 1
+            self.current += 1
 
 
-proc disassembleChunk*(chunk: Chunk, name: string) =
+proc parseCFIData(self: Debugger) =
-    ## Takes a chunk of bytecode, and prints it
+    ## Parses CFI information in the chunk
-    echo &"==== Peon Bytecode Debugger - Chunk '{name}' ====\n"
+    var 
-    var index = 0
+        start, stop, argc: int
-    while index < chunk.code.len:
+        name: string
-        index = disassembleInstruction(chunk, index)
+        idx = 0
+        size = 0
+    while idx < len(self.chunk.cfi) - 1:
+        start = int([self.chunk.cfi[idx], self.chunk.cfi[idx + 1], self.chunk.cfi[idx + 2]].fromTriple())
+        idx += 3
+        stop = int([self.chunk.cfi[idx], self.chunk.cfi[idx + 1], self.chunk.cfi[idx + 2]].fromTriple())
+        idx += 3
+        argc = int(self.chunk.cfi[idx])
+        inc(idx)
+        size = int([self.chunk.cfi[idx], self.chunk.cfi[idx + 1]].fromDouble())
+        idx += 2
+        name = self.chunk.cfi[idx..<idx + size].fromBytes()
+        inc(idx, size)
+        self.cfiData.add(CFIElement(start: start, stop: stop,
+                                    argc: argc, name: name))
+
+
+proc disassembleChunk*(self: Debugger, chunk: Chunk, name: string) =
+    ## Takes a chunk of bytecode and prints it
+    self.chunk = chunk
+    styledEcho fgBlue, &"==== Peon Bytecode Debugger: Session starting - Chunk '{name}' ====\n"
+    self.current = 0
+    self.parseCFIData()
+    while self.current < self.chunk.code.len:
+        self.disassembleInstruction()
         echo ""
-    echo &"==== Debug session ended - Chunk '{name}' ===="
+    styledEcho fgBlue, &"==== Peon Bytecode Debugger: Session ended - Chunk '{name}' ===="
 
 

src/util/multibyte.nim

@@ -12,8 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-## Utilities to convert from/to our 16-bit and 24-bit representations
+## Utilities to handle multibyte sequences
-## of numbers
 
 
 proc toDouble*(input: int | uint | uint16): array[2, uint8] =
@@ -58,4 +57,29 @@ proc fromQuad*(input: array[4, uint8]): uint =
 proc fromLong*(input: array[8, uint8]): uint =
     ## Rebuilts the output of toQuad into
     ## an uint
-    copyMem(result.addr, unsafeAddr(input), sizeof(uint64))
+    copyMem(result.addr, unsafeAddr(input), sizeof(uint64))
+
+
+proc toBytes*(s: string): seq[byte] =
+    ## Converts a string into a sequence
+    ## of bytes
+    for c in s:
+        result.add(byte(c))
+
+
+proc toBytes*(s: int): array[8, uint8] =
+    ## Converts
+    result = cast[array[8, uint8]](s)
+
+
+proc fromBytes*(input: seq[byte]): string =
+    ## Converts a sequence of bytes to
+    ## a string
+    for b in input:
+        result.add(char(b))
+
+
+proc extend*[T](s: var seq[T], a: openarray[T]) =
+    ## Extends s with the elements of a
+    for e in a:
+        s.add(e)

src/util/serializer.nim

@@ -13,14 +13,14 @@
 # limitations under the License.
 import ../frontend/meta/errors
 import ../frontend/meta/bytecode
-import ../config
-import multibyte
 import ../frontend/compiler
+import multibyte
+import ../config
 
-import strformat
+
-import strutils
+import std/strformat
-import nimSHA2
+import std/strutils
-import times
+import std/times
 
 
 export ast
@@ -35,16 +35,15 @@ type
         ## the Serializer.read*
         ## procedures to store
         ## metadata
-        fileHash*: string
+        version*: tuple[major, minor, patch: int]
-        peonVer*: tuple[major, minor, patch: int]
+        branch*: string
-        peonBranch*: string
+        commit*: string
-        commitHash*: string
         compileDate*: int
         chunk*: Chunk
 
 
 proc `$`*(self: Serialized): string =
-    result = &"Serialized(fileHash={self.fileHash}, version={self.peonVer.major}.{self.peonVer.minor}.{self.peonVer.patch}, branch={self.peonBranch}), commitHash={self.commitHash}, date={self.compileDate}, chunk={self.chunk[]}"
+    result = &"Serialized(version={self.version.major}.{self.version.minor}.{self.version.patch}, branch={self.branch}), commitHash={self.commit}, date={self.compileDate}, chunk={self.chunk[]}"
 
 
 proc error(self: Serializer, message: string) =
@@ -61,44 +60,16 @@ proc newSerializer*(self: Serializer = nil): Serializer =
     result.chunk = nil
 
 
-## Basic routines and helpers to convert various objects from and to to their byte representation
+proc writeHeaders(self: Serializer, stream: var seq[byte]) =
-
-proc toBytes(self: Serializer, s: string): seq[byte] =
-    for c in s:
-        result.add(byte(c))
-
-
-proc toBytes(self: Serializer, s: int): array[8, uint8] =
-    result = cast[array[8, uint8]](s)
-
-
-proc toBytes(self: Serializer, d: SHA256Digest): seq[byte] =
-    for b in d:
-        result.add(b)
-
-
-proc bytesToString(self: Serializer, input: seq[byte]): string =
-    for b in input:
-        result.add(char(b))
-
-
-proc extend[T](s: var seq[T], a: openarray[T]) =
-    ## Extends s with the elements of a
-    for e in a:
-        s.add(e)
-
-
-proc writeHeaders(self: Serializer, stream: var seq[byte], file: string) =
     ## Writes the Peon bytecode headers in-place into a byte stream
-    stream.extend(self.toBytes(BYTECODE_MARKER))
+    stream.extend(PeonBytecodeMarker.toBytes())
     stream.add(byte(PEON_VERSION.major))
     stream.add(byte(PEON_VERSION.minor))
     stream.add(byte(PEON_VERSION.patch))
     stream.add(byte(len(PEON_BRANCH)))
-    stream.extend(self.toBytes(PEON_BRANCH))
+    stream.extend(PEON_BRANCH.toBytes())
-    stream.extend(self.toBytes(PEON_COMMIT_HASH))
+    stream.extend(PEON_COMMIT_HASH.toBytes())
-    stream.extend(self.toBytes(getTime().toUnixFloat().int()))
+    stream.extend(getTime().toUnixFloat().int().toBytes())
-    stream.extend(self.toBytes(computeSHA256(file)))
 
 
 proc writeLineData(self: Serializer, stream: var seq[byte]) =
@@ -107,7 +78,14 @@ proc writeLineData(self: Serializer, stream: var seq[byte]) =
     stream.extend(len(self.chunk.lines).toQuad())
     for b in self.chunk.lines:
         stream.extend(b.toTriple())
-    
+
+
+proc writeCFIData(self: Serializer, stream: var seq[byte]) =
+    ## Writes Call Frame Information for debugging
+    ## functions
+    stream.extend(len(self.chunk.cfi).toQuad())
+    stream.extend(self.chunk.cfi)
+
 
 proc writeConstants(self: Serializer, stream: var seq[byte]) =
     ## Writes the constants table in-place into the 
@@ -128,28 +106,27 @@ proc readHeaders(self: Serializer, stream: seq[byte], serialized: Serialized): i
     ## Reads the bytecode headers from a given stream
     ## of bytes
     var stream = stream
-    if stream[0..<len(BYTECODE_MARKER)] != self.toBytes(BYTECODE_MARKER):
+    if stream[0..<len(PeonBytecodeMarker)] != PeonBytecodeMarker.toBytes():
         self.error("malformed bytecode marker")
-    result += len(BYTECODE_MARKER)
+    result += len(PeonBytecodeMarker)
-    stream = stream[len(BYTECODE_MARKER)..^1]
+    stream = stream[len(PeonBytecodeMarker)..^1]
-    serialized.peonVer = (major: int(stream[0]), minor: int(stream[1]), patch: int(stream[2]))
+    serialized.version = (major: int(stream[0]), minor: int(stream[1]), patch: int(stream[2]))
     stream = stream[3..^1]
     result += 3
     let branchLength = stream[0]
     stream = stream[1..^1]
     result += 1
-    serialized.peonBranch = self.bytesToString(stream[0..<branchLength])
+    serialized.branch = stream[0..<branchLength].fromBytes()
     stream = stream[branchLength..^1]
     result += int(branchLength)
-    serialized.commitHash = self.bytesToString(stream[0..<40]).toLowerAscii()
+    serialized.commit = stream[0..<40].fromBytes().toLowerAscii()
     stream = stream[40..^1]
     result += 40
     serialized.compileDate = int(fromLong([stream[0], stream[1], stream[2],
             stream[3], stream[4], stream[5], stream[6], stream[7]]))
     stream = stream[8..^1]
     result += 8
-    serialized.fileHash = self.bytesToString(stream[0..<32]).toHex().toLowerAscii()
+
-    result += 32
 
 
 proc readLineData(self: Serializer, stream: seq[byte]): int =
@@ -164,6 +141,17 @@ proc readLineData(self: Serializer, stream: seq[byte]): int =
         stream = stream[3..^1]
 
 
+proc readCFIData(self: Serializer, stream: seq[byte]): int =
+    ## Reads Call Frame Information from a stream
+    ## of bytes
+    let size = [stream[0], stream[1], stream[2], stream[3]].fromQuad()
+    result += 4
+    var stream = stream[4..^1]
+    for i in countup(0, int(size) - 1):
+        self.chunk.cfi.add(stream[i])
+        inc(result)
+
+
 proc readConstants(self: Serializer, stream: seq[byte]): int =
     ## Reads the constant table from the given stream 
     ## of bytes
@@ -186,24 +174,22 @@ proc readCode(self: Serializer, stream: seq[byte]): int =
     return int(size)
 
 
-proc dumpBytes*(self: Serializer, chunk: Chunk, file, filename: string): seq[byte] =
+proc dumpBytes*(self: Serializer, chunk: Chunk, filename: string): seq[byte] =
     ## Dumps the given bytecode and file to a sequence of bytes and returns it.
-    ## The file argument must be the actual file's content and is needed to 
-    ## compute its SHA256 hash.
-    self.file = file
     self.filename = filename
     self.chunk = chunk
-    self.writeHeaders(result, self.file)
+    self.writeHeaders(result)
     self.writeLineData(result)
+    self.writeCFIData(result)
     self.writeConstants(result)
     self.writeCode(result)
 
 
-proc dumpFile*(self: Serializer, chunk: Chunk, file, filename, dest: string) =
+proc dumpFile*(self: Serializer, chunk: Chunk, filename, dest: string) =
     ## Dumps the result of dumpBytes to a file at dest
     var fp = open(dest, fmWrite)
     defer: fp.close()
-    let data = self.dumpBytes(chunk, file, filename)
+    let data = self.dumpBytes(chunk, filename)
     discard fp.writeBytes(data, 0, len(data))
 
 
@@ -218,6 +204,7 @@ proc loadBytes*(self: Serializer, stream: seq[byte]): Serialized =
     try:
         stream = stream[self.readHeaders(stream, result)..^1]
         stream = stream[self.readLineData(stream)..^1]
+        stream = stream[self.readCFIData(stream)..^1]
         stream = stream[self.readConstants(stream)..^1]
         stream = stream[self.readCode(stream)..^1]
     except IndexDefect:

src/util/symbols.nim

@@ -0,0 +1,63 @@
+import ../frontend/lexer
+
+
+proc fillSymbolTable*(tokenizer: Lexer) =
+    ## Initializes the Lexer's symbol
+    ## table with the builtin symbols
+    ## and keywords
+
+    # 1-byte symbols
+    tokenizer.symbols.addSymbol("{", LeftBrace)
+    tokenizer.symbols.addSymbol("}", RightBrace)
+    tokenizer.symbols.addSymbol("(", LeftParen)
+    tokenizer.symbols.addSymbol(")", RightParen)
+    tokenizer.symbols.addSymbol("[", LeftBracket)
+    tokenizer.symbols.addSymbol("]", RightBracket)
+    tokenizer.symbols.addSymbol(".", Dot)
+    tokenizer.symbols.addSymbol(",", Comma)
+    tokenizer.symbols.addSymbol(";", Semicolon)
+    # tokenizer.symbols.addSymbol("\n", Semicolon)  # TODO: Broken
+    # Keywords
+    tokenizer.symbols.addKeyword("type", TokenType.Type)
+    tokenizer.symbols.addKeyword("enum", Enum)
+    tokenizer.symbols.addKeyword("case", Case)
+    tokenizer.symbols.addKeyword("operator", Operator)
+    tokenizer.symbols.addKeyword("generator", Generator)
+    tokenizer.symbols.addKeyword("fn", TokenType.Function)
+    tokenizer.symbols.addKeyword("coroutine", Coroutine)
+    tokenizer.symbols.addKeyword("break", TokenType.Break)
+    tokenizer.symbols.addKeyword("continue", Continue)
+    tokenizer.symbols.addKeyword("while", While)
+    tokenizer.symbols.addKeyword("for", For)
+    tokenizer.symbols.addKeyword("foreach", Foreach)
+    tokenizer.symbols.addKeyword("if", If)
+    tokenizer.symbols.addKeyword("else", Else)
+    tokenizer.symbols.addKeyword("await", TokenType.Await)
+    tokenizer.symbols.addKeyword("defer", Defer)
+    tokenizer.symbols.addKeyword("try", Try)
+    tokenizer.symbols.addKeyword("except", Except)
+    tokenizer.symbols.addKeyword("finally", Finally)
+    tokenizer.symbols.addKeyword("raise", TokenType.Raise)
+    tokenizer.symbols.addKeyword("assert", TokenType.Assert)
+    tokenizer.symbols.addKeyword("const", Const)
+    tokenizer.symbols.addKeyword("let", Let)
+    tokenizer.symbols.addKeyword("var", TokenType.Var)
+    tokenizer.symbols.addKeyword("import", Import)
+    tokenizer.symbols.addKeyword("yield", TokenType.Yield)
+    tokenizer.symbols.addKeyword("return", TokenType.Return)
+    tokenizer.symbols.addKeyword("object", Object)
+    # These are more like expressions with a reserved
+    # name that produce a value of a builtin type,
+    # but we don't need to care about that until
+    # we're in the parsing/ compilation steps so
+    # it's fine
+    tokenizer.symbols.addKeyword("nan", NotANumber)
+    tokenizer.symbols.addKeyword("inf", Infinity)
+    tokenizer.symbols.addKeyword("nil", TokenType.Nil)
+    tokenizer.symbols.addKeyword("true", True)
+    tokenizer.symbols.addKeyword("false", False)
+    tokenizer.symbols.addKeyword("ref", TokenType.Ref)
+    tokenizer.symbols.addKeyword("ptr", TokenType.Ptr)
+    for sym in [">", "<", "=", "~", "/", "+", "-", "_", "*", "?", "@", ":", "==", "!=",
+                ">=", "<=", "+=", "-=", "/=", "*=", "**=", "!"]:
+        tokenizer.symbols.addSymbol(sym, Symbol)

tests/calls.pn

@@ -0,0 +1,19 @@
+# Tests simple calls
+import std;
+
+
+fn noReturn(n: int) {
+    var n = n;
+    var `17` = 17;
+    return;
+}
+
+
+fn fooBar(a, b: int): int {
+    var baz = 38;
+    return a;
+}
+
+
+noReturn(1);
+print(fooBar(1, 3));  # 1

tests/chainedCalls.pn

@@ -0,0 +1,26 @@
+fn first(a, b: int): int {
+    return a;
+}
+
+fn second(a, b: int): int {
+    return b;
+}
+
+
+fn last(a, b, c: int): int {
+       return c;
+}
+
+
+fn middle(a, b, c: int): int {
+       return last(a, c, b);
+}
+
+
+fn first(a, b, c: int): int {
+       return middle(b, a, c);
+}
+
+
+first(1, 2, 3);
+var x = first(second(1, 2), 3);

tests/closures.pn

@@ -0,0 +1,16 @@
+# Tests closures
+import std;
+
+
+fn makeClosure(n: int): fn: int {
+    let n = n;  # Workaround
+    fn inner: int {
+        return n;
+    }
+    return inner;
+}
+
+
+var closure = makeClosure(1)();
+print(closure);   # 1
+print(makeClosure(2)());  # 2

tests/comparisons.pn

@@ -0,0 +1,152 @@
+# Tests that comparisons work
+import std;
+
+
+# int64
+print(3 > 2);  # true
+print(2 < 3);  # true
+print(3 < 2);  # false
+print(2 > 3);  # false
+print(2 != 3); # true
+print(3 != 2); # true
+print(3 == 2); # false
+print(2 == 3); # false
+print(2 <= 3); # true
+print(3 >= 2); # true
+print(2 >= 3); # false
+print(3 <= 2); # false
+# uint64
+var x = 3'u64;
+var y = 2'u64;
+print(x > y);  # true
+print(y < x);  # true
+print(x < y);  # false
+print(y > x);  # false
+print(y != x); # true
+print(x != y); # true
+print(x == y); # false
+print(y == x); # false
+print(y <= x); # true
+print(x >= y); # true
+print(y >= x); # false
+print(x <= y); # false
+# int32
+var x1 = 3'i32;
+var y1 = 2'i32;
+print(x1 > y1);  # true
+print(y1 < x1);  # true
+print(x1 < y1);  # false
+print(y1 > x1);  # false
+print(y1 != x1); # true
+print(x1 != y1); # true
+print(x1 == y1); # false
+print(y1 == x1); # false
+print(y1 <= x1); # true
+print(x1 >= y1); # true
+print(y1 >= x1); # false
+print(x1 <= y1); # false
+# uint32
+var x2 = 3'u32;
+var y2 = 2'u32;
+print(x2 > y2);  # true
+print(y2 < x2);  # true
+print(x2 < y2);  # false
+print(y2 > x2);  # false
+print(y2 != x2); # true
+print(x2 != y2); # true
+print(x2 == y2); # false
+print(y2 == x2); # false
+print(y2 <= x2); # true
+print(x2 >= y2); # true
+print(y2 >= x2); # false
+print(x2 <= y2); # false
+# int16
+var x3 = 3'i16;
+var y3 = 2'i16;
+print(x3 > y3);  # true
+print(y3 < x3);  # true
+print(x3 < y3);  # false
+print(y3 > x3);  # false
+print(y3 != x3); # true
+print(x3 != y3); # true
+print(x3 == y3); # false
+print(y3 == x3); # false
+print(y3 <= x3); # true
+print(x3 >= y3); # true
+print(y3 >= x3); # false
+print(x3 <= y3); # false
+# uint16
+var x4 = 3'u16;
+var y4 = 2'u16;
+print(x4 > y4);  # true
+print(y4 < x4);  # true
+print(x4 < y4);  # false
+print(y4 > x4);  # false
+print(y4 != x4); # true
+print(x4 != y4); # true
+print(x4 == y4); # false
+print(y4 == x4); # false
+print(y4 <= x4); # true
+print(x4 >= y4); # true
+print(y4 >= x4); # false
+print(x4 <= y4); # false
+# int8
+var x5 = 3'i8;
+var y5 = 2'i8;
+print(x5 > y5);  # true
+print(y5 < x5);  # true
+print(x5 < y5);  # false
+print(y5 > x5);  # false
+print(y5 != x5); # true
+print(x5 != y5); # true
+print(x5 == y5); # false
+print(y5 == x5); # false
+print(y5 <= x5); # true
+print(x5 >= y5); # true
+print(y5 >= x5); # false
+print(x5 <= y5); # false
+# uint8
+var x6 = 3'u8;
+var y6 = 2'u8;
+print(x6 > y6);  # true
+print(y6 < x6);  # true
+print(x6 < y6);  # false
+print(y6 > x6);  # false
+print(y6 != x6); # true
+print(x6 != y6); # true
+print(x6 == y6); # false
+print(y6 == x6); # false
+print(y6 <= x6); # true
+print(x6 >= y6); # true
+print(y6 >= x6); # false
+print(x6 <= y6); # false
+# float64
+var x7 = 3.0;
+var y7 = 2.0;
+print(x7 > y7);  # true
+print(y7 < x7);  # true
+print(x7 < y7);  # false
+print(y7 > x7);  # false
+print(y7 != x7); # true
+print(x7 != y7); # true
+print(x7 == y7); # false
+print(y7 == x7); # false
+print(y7 <= x7); # true
+print(x7 >= y7); # true
+print(y7 >= x7); # false
+print(x7 <= y7); # false
+# float32
+var x8 = 3'f32;
+var y8 = 2'f32;
+print(x8 > y8);  # true
+print(y8 < x8);  # true
+print(x8 < y8);  # false
+print(y8 > x8);  # false
+print(y8 != x8); # true
+print(x8 != y8); # true
+print(x8 == y8); # false
+print(y8 == x8); # false
+print(y8 <= x8); # true
+print(x8 >= y8); # true
+print(y8 >= x8); # false
+print(x8 <= y8); # false

tests/dispatch.pn

@@ -1,5 +1,10 @@
+# Tests operator dispatching
+
+
 operator `+`(a: int): int {
     return a;
 }
 
+
 +1;  # Works: defined for int64
++1'i32;  # Will not work 

tests/fib.pn

@@ -0,0 +1,52 @@
+operator `<`(a, b: int): bool {
+    #pragma[magic: "LessThan", pure]
+}
+
+
+operator `+`(a, b: int): int {
+    #pragma[magic: "Add", pure]
+}
+
+
+operator `-`(a, b: int): int {
+    #pragma[magic: "Subtract", pure]
+}
+
+
+operator `-`(a, b: float): float {
+    #pragma[magic: "SubtractFloat64", pure]
+}
+
+fn clock: float {
+    #pragma[magic: "SysClock64"]
+}
+
+
+fn print(x: int) {
+    #pragma[magic: "PrintInt64"]
+}
+
+
+fn print(x: float) {
+    #pragma[magic: "PrintFloat64"]
+}
+
+
+fn print(x: string) {
+    #pragma[magic: "PrintString"]
+}
+
+
+fn fib(n: int): int {
+    if n < 2 {
+        return n;
+    }
+    return fib(n - 2) + fib(n - 1);
+}
+
+
+print("Computing the value of fib(30)");
+var x = clock();
+print(fib(30));
+print(clock() - x);
+print("Done!");

tests/functionObj.pn

@@ -0,0 +1,29 @@
+# Tests first class functions
+
+import std;
+
+
+fn outer: fn (n: int): int {
+    fn inner(n: int): int {
+        return n;
+    }
+    return inner;
+}
+
+
+
+fn getAdder(a, b: int): fn: int64 {
+    var x = a;
+    var y = b;
+    fn adder: int {
+        return x + y;
+    }
+    return adder;
+}
+
+
+print(outer()(1));  # 1
+var a = 1;
+var b = 2;
+var adder = getAdder(a, b);
+print(adder());   # 3

tests/generics.pn

@@ -0,0 +1,16 @@
+operator `+`(a, b: int): int {
+    #pragma[magic: "AddInt64", pure]
+}    
+   
+
+operator `+`(a, b: int32): int32 {
+    #pragma[magic: "AddInt64", pure]
+}
+
+
+fn sum[T](a, b: T): T {
+    return a + b;
+}
+
+sum(1, 2);
+sum(1'i32, 2'i32);

tests/mutable.pn

@@ -0,0 +1,19 @@
+# Tests var parameters
+
+import std;
+
+
+operator `+=`(a: var int, b: int) {
+    a = a + b;
+}
+
+
+fn plusOne(x: var int): int {
+    x += 1;
+    return x;
+}
+
+
+var x = 5;
+print(plusOne(x));
+# plusOne(38);  # If you uncomment this, the compiler errors out!

tests/nestedCalls.pn

@@ -0,0 +1,22 @@
+# Tests nested calls
+import std;
+
+
+fn outer: int {
+    fn inner: int {
+        return 69420;
+    }
+    return inner();
+}
+
+
+fn outerTwo(n: int): int {
+    fn inner(z: int): int {
+        return z;
+    }
+    return inner(n);
+}
+
+
+print(outerTwo(5));
+print(outer());

tests/operators.pn

@@ -0,0 +1,10 @@
+# Tests the creation and use of custom operators
+import std;
+
+
+operator `sum`(a, b: int): int {
+    return a + b;
+}    
+
+
+print(2 sum 2);  # 4

tests/scopes.pn

@@ -0,0 +1,14 @@
+# Tests local scopes
+import std;
+
+
+var x = 5;
+{
+    var x = 55;
+    {
+        var x = 22;
+        print(x);
+    }
+    print(x);
+}
+print(x);

tests/std.pn

@@ -0,0 +1,560 @@
+## The peon standard library
+
+# Builtin arithmetic operators for Peon
+# Note: Most of these do nothing on their own. All they do
+# is serve as placeholders for emitting specific VM
+# instructions. They're implemented this way because:
+# - They tie into the existing type system nicely
+# - It makes the implementation easier and more flexible
+
+
+operator `+`*(a, b: int): int {
+    #pragma[magic: "SignedAdd", pure]
+}
+
+
+operator `+`*(a, b: uint64): uint64 {
+    #pragma[magic: "Add", pure]
+}
+
+
+operator `+`*(a, b: int32): int32 {
+    #pragma[magic: "SignedAdd", pure]
+}
+
+
+operator `+`*(a, b: uint32): uint32 {
+    #pragma[magic: "Add", pure]
+}
+
+
+operator `+`*(a, b: int16): int16 {
+    #pragma[magic: "SignedAdd", pure]
+}
+
+
+operator `+`*(a, b: uint16): uint16 {
+    #pragma[magic: "Add", pure]
+}
+
+
+operator `+`*(a, b: int8): int8 {
+    #pragma[magic: "Add", pure]
+}
+
+
+operator `+`*(a, b: uint8): uint8 {
+    #pragma[magic: "AddUInt8", pure]
+}
+
+
+operator `+`*(a, b: float64): float64 {
+    #pragma[magic: "AddFloat64", pure]
+}
+
+
+operator `+`*(a, b: float32): float32 {
+    #pragma[magic: "AddFloat32", pure]
+}
+
+
+operator `-`*(a, b: int): int {
+    #pragma[magic: "SubInt64", pure]
+}
+
+
+operator `-`*(a, b: uint64): uint64 {
+    #pragma[magic: "SubUInt64", pure]
+}
+
+
+operator `-`*(a, b: int32): int32 {
+    #pragma[magic: "SubInt32", pure]
+}
+
+
+operator `-`*(a, b: uint32): uint32 {
+    #pragma[magic: "SubUInt32", pure]
+}
+
+
+operator `-`*(a, b: int16): int16 {
+    #pragma[magic: "SubInt16", pure]
+}
+
+
+operator `-`*(a, b: uint16): uint16 {
+    #pragma[magic: "SubUInt16", pure]
+}
+
+
+operator `-`*(a, b: int8): int8 {
+    #pragma[magic: "SubInt8", pure]
+}
+
+
+operator `-`*(a, b: uint8): uint8 {
+    #pragma[magic: "SubUInt8", pure]
+}
+
+
+operator `-`*(a, b: float64): float64 {
+    #pragma[magic: "SubFloat64", pure]
+}
+
+
+operator `-`*(a, b: float32): float32 {
+    #pragma[magic: "SubFloat32", pure]
+}
+
+
+operator `*`*(a, b: int): int {
+    #pragma[magic: "SignedMultiply", pure]
+}
+
+
+operator `*`*(a, b: uint64): uint64 {
+    #pragma[magic: "MulUInt64", pure]
+}
+
+
+operator `*`*(a, b: int32): int32 {
+    #pragma[magic: "SignedMultiply", pure]
+}
+
+
+operator `*`*(a, b: uint32): uint32 {
+    #pragma[magic: "MulUInt32", pure]
+}
+
+
+operator `*`*(a, b: int16): int16 {
+    #pragma[magic: "SignedMultiply", pure]
+}
+
+
+operator `*`*(a, b: uint16): uint16 {
+    #pragma[magic: "MulUInt16", pure]
+}
+
+
+operator `*`*(a, b: int8): int8 {
+    #pragma[magic: "SignedMultiply", pure]
+}
+
+
+operator `*`*(a, b: uint8): uint8 {
+    #pragma[magic: "MulUInt8", pure]
+}
+
+
+operator `*`*(a, b: float64): float64 {
+    #pragma[magic: "MulFloat64", pure]
+}
+
+
+operator `*`*(a, b: float32): float32 {
+    #pragma[magic: "MulFloat32", pure]
+}
+
+
+operator `/`*(a, b: int): int {
+    #pragma[magic: "DivInt64", pure]
+}
+
+
+operator `/`*(a, b: uint64): uint64 {
+    #pragma[magic: "DivUInt64", pure]
+}
+
+
+operator `/`*(a, b: int32): int32 {
+    #pragma[magic: "DivInt32", pure]
+}
+
+
+operator `/`*(a, b: uint32): uint32 {
+    #pragma[magic: "DivUInt32", pure]
+}
+
+
+operator `/`*(a, b: int16): int16 {
+    #pragma[magic: "DivInt16", pure]
+}
+
+
+operator `/`*(a, b: uint16): uint16 {
+    #pragma[magic: "DivUInt16", pure]
+}
+
+
+operator `/`*(a, b: int8): int8 {
+    #pragma[magic: "DivInt8", pure]
+}
+
+
+operator `/`*(a, b: uint8): uint8 {
+    #pragma[magic: "DivUInt8", pure]
+}
+
+
+operator `/`*(a, b: float64): float64 {
+    #pragma[magic: "DivFloat64", pure]
+}
+
+
+operator `/`*(a, b: float32): float32 {
+    #pragma[magic: "DivFloat32", pure]
+}
+
+
+operator `**`*(a, b: int64): int64 {
+    #pragma[magic: "PowInt64", pure]
+}
+
+
+# Comparison operators
+
+operator `>`*(a, b: int): bool {
+    #pragma[magic: "GreaterThanInt64", pure]
+}
+
+
+operator `<`*(a, b: int): bool {
+    #pragma[magic: "LessThanInt64", pure]
+}
+
+
+operator `==`*(a, b: int): bool {
+    #pragma[magic: "EqualInt64", pure]
+
+}
+
+operator `!=`*(a, b: int): bool {
+    #pragma[magic: "NotEqualInt64", pure]
+}
+
+
+operator `>`*(a, b: uint64): bool {
+    #pragma[magic: "GreaterThanUInt64", pure]
+}
+
+
+operator `<`*(a, b: uint64): bool {
+    #pragma[magic: "LessThanUInt64", pure]
+}
+
+
+operator `==`*(a, b: uint64): bool {
+    #pragma[magic: "EqualUInt64", pure]
+}
+
+
+operator `!=`*(a, b: uint64): bool {
+    #pragma[magic: "NotEqualUInt64", pure]
+}
+
+
+operator `>`*(a, b: int32): bool {
+    #pragma[magic: "GreaterThanInt32", pure]
+}
+
+
+operator `<`*(a, b: int32): bool {
+    #pragma[magic: "LessThanInt32", pure]
+}
+
+
+operator `==`*(a, b: int32): bool {
+    #pragma[magic: "EqualInt32", pure]
+}
+
+
+operator `!=`*(a, b: int32): bool {
+    #pragma[magic: "NotEqualInt32", pure]
+}
+
+operator `>`*(a, b: uint32): bool {
+    #pragma[magic: "GreaterThanUInt32", pure]
+}
+
+
+operator `<`*(a, b: uint32): bool {
+    #pragma[magic: "LessThanUInt32", pure]
+}
+
+
+operator `==`*(a, b: uint32): bool {
+    #pragma[magic: "EqualUInt32", pure]
+}
+
+
+operator `!=`*(a, b: uint32): bool {
+    #pragma[magic: "NotEqualUInt32", pure]
+}
+
+
+operator `>`*(a, b: int16): bool {
+    #pragma[magic: "GreaterThanInt16", pure]
+}
+
+
+operator `<`*(a, b: int16): bool {
+    #pragma[magic: "LessThanInt16", pure]
+}
+
+
+operator `==`*(a, b: int16): bool {
+    #pragma[magic: "EqualInt16", pure]
+}
+
+
+operator `!=`*(a, b: int16): bool {
+    #pragma[magic: "NotEqualInt16", pure]
+}
+
+
+operator `>`*(a, b: uint16): bool {
+    #pragma[magic: "GreaterThanUInt16", pure]
+}
+
+
+operator `<`*(a, b: uint16): bool {
+    #pragma[magic: "LessThanUInt16", pure]
+}
+
+
+operator `==`*(a, b: uint16): bool {
+    #pragma[magic: "EqualUInt16", pure]
+}
+
+
+operator `!=`*(a, b: uint16): bool {
+    #pragma[magic: "NotEqualUInt16", pure]
+}
+
+
+operator `>`*(a, b: int8): bool {
+    #pragma[magic: "GreaterThanInt8", pure]
+}
+
+
+operator `<`*(a, b: int8): bool {
+    #pragma[magic: "LessThanInt8", pure]
+}
+
+
+operator `==`*(a, b: int8): bool {
+    #pragma[magic: "EqualInt8", pure]
+}
+
+
+operator `!=`*(a, b: int8): bool {
+    #pragma[magic: "NotEqualInt8", pure]
+}
+
+
+operator `>`*(a, b: uint8): bool {
+    #pragma[magic: "GreaterThanUInt8", pure]
+}
+
+
+operator `<`*(a, b: uint8): bool {
+    #pragma[magic: "LessThanUInt8", pure]
+}
+
+
+operator `==`*(a, b: uint8): bool {
+    #pragma[magic: "EqualUInt8", pure]
+}
+
+
+operator `!=`*(a, b: uint8): bool {
+    #pragma[magic: "NotEqualUInt8", pure]
+}
+
+
+operator `>`*(a, b: float): bool {
+    #pragma[magic: "GreaterThanFloat64", pure]
+}
+
+
+operator `<`*(a, b: float): bool {
+    #pragma[magic: "LessThanFloat64", pure]
+}
+
+
+operator `==`*(a, b: float): bool {
+    #pragma[magic: "EqualFloat64", pure]
+}
+
+
+operator `!=`*(a, b: float): bool {
+    #pragma[magic: "NotEqualFloat64", pure]
+}
+
+
+operator `>`*(a, b: float32): bool {
+    #pragma[magic: "GreaterThanFloat32", pure]
+}
+
+
+operator `<`*(a, b: float32): bool {
+    #pragma[magic: "LessThanFloat32", pure]
+}
+
+
+operator `==`*(a, b: float32): bool {
+    #pragma[magic: "EqualFloat32", pure]
+}
+
+
+operator `!=`*(a, b: float32): bool {
+    #pragma[magic: "NotEqualFloat32", pure]
+}
+
+
+operator `>=`*(a, b: int): bool {
+    #pragma[magic: "GreaterOrEqualInt64", pure]
+}
+
+
+operator `<=`*(a, b: int): bool {
+    #pragma[magic: "LessOrEqualInt64", pure]
+}
+
+
+operator `>=`*(a, b: uint64): bool {
+    #pragma[magic: "GreaterOrEqualUInt64", pure]
+}
+
+
+operator `<=`*(a, b: uint64): bool {
+    #pragma[magic: "LessOrEqualUInt64", pure]
+}
+
+
+operator `>=`*(a, b: int32): bool {
+    #pragma[magic: "GreaterOrEqualInt32", pure]
+}
+
+
+operator `<=`*(a, b: int32): bool {
+    #pragma[magic: "LessOrEqualInt32", pure]
+}
+
+
+operator `>=`*(a, b: uint32): bool {
+    #pragma[magic: "GreaterOrEqualUInt32", pure]
+}
+
+
+operator `<=`*(a, b: uint32): bool {
+    #pragma[magic: "LessOrEqualUInt32", pure]
+}
+
+
+operator `>=`*(a, b: int16): bool {
+    #pragma[magic: "GreaterOrEqualInt16", pure]
+}
+
+
+operator `<=`*(a, b: int16): bool {
+    #pragma[magic: "LessOrEqualInt16", pure]
+}
+
+
+operator `>=`*(a, b: uint16): bool {
+    #pragma[magic: "GreaterOrEqualUInt16", pure]
+}
+
+
+operator `<=`*(a, b: uint16): bool {
+    #pragma[magic: "LessOrEqualUInt16", pure]
+}
+
+
+operator `>=`*(a, b: int8): bool {
+    #pragma[magic: "GreaterOrEqualInt8", pure]
+}
+
+
+operator `<=`*(a, b: int8): bool {
+    #pragma[magic: "LessOrEqualInt8", pure]
+}
+
+
+operator `>=`*(a, b: uint8): bool {
+    #pragma[magic: "GreaterOrEqualUInt8", pure]
+}
+
+
+operator `<=`*(a, b: uint8): bool {
+    #pragma[magic: "LessOrEqualUInt8", pure]
+}
+
+
+operator `>=`*(a, b: float): bool {
+    #pragma[magic: "GreaterOrEqualFloat64", pure]
+}
+
+
+operator `<=`*(a, b: float): bool {
+    #pragma[magic: "LessOrEqualFloat64", pure]
+}
+
+
+operator `>=`*(a, b: float32): bool {
+    #pragma[magic: "GreaterOrEqualFloat32", pure]
+}
+
+
+operator `<=`*(a, b: float32): bool {
+    #pragma[magic: "LessOrEqualFloat32", pure]
+}
+
+
+operator `and`*(a, b: bool): bool {
+    #pragma[magic: "LogicalAnd", pure]
+}
+
+
+operator `or`*(a, b: bool): bool {
+    #pragma[magic: "LogicalOr", pure]
+}
+
+
+# Assignment operators
+
+operator `=`[T: Any](a: var T, b: T) {
+    #pragma[magic: "GenericAssign"]
+}
+
+
+# Some useful builtins
+
+fn clock*: float {
+    #pragma[magic: "SysClock64", pure]
+}
+
+
+# TODO: Replace with generics
+
+fn print*(x: float) {
+    #pragma[magic: "GenericPrint"]
+}
+
+
+fn print*(x: int) {
+    #pragma[magic: "GenericPrint"]
+}
+
+
+fn print*(x: string) {
+    #pragma[magic: "GenericPrint"]
+}
+
+
+fn print*(x: bool) {
+    #pragma[magic: "GenericPrint"]
+}

tests/test.pn

@@ -1 +0,0 @@
-# TODO