first version of reference, some new tests

README.md

@@ -4,20 +4,15 @@ Nondescript is a toy programming language. Currently its compiler is being rewri
 
 # Some of nondescript's features
 
+- Dynamic typing, bytecode interpreter
 - Almost everything is an expression
-- block expressions can be labelled (@label) and the break statement takes a label to break out of
-- set block expression results using :label
-- set the return value of functions using :result
-- lists (example: @[1, 4, 5])
-- tables (example: @{ "hello" = "world"})
-- length operator #
-- ampersand operator to chain binary operators to finished expressions
-- just a little syntax sugar (see sugar.nds in tests)
-- and some more (an introduction to nondescript coming soon...)
+- Syntax tries to balance sugar with consistency
+See [[docs/reference.md]] to learn more about the language.
 
 ## Examples
 
-See the following folders in the source tree, look for the extension .nds:
+If the [[docs/]] folder is not enough, the following folders in the source tree, will have more examples:
+
 - benchmarks/
 - examples/
 - tests/

docs/reference.md

@@ -0,0 +1,299 @@
+Note: this is a work in progress.
+
+Sections:
+1. User's manual to nondescript
+2. Design notes
+3. Roadmap
+4. Known bugs
+
+# User's manual to nondescript
+
+## File format
+
+As a convention, nondescript scripts end with `.nds`, and are text files in UTF-8 encoding (without BOM) and LF (unix) line endings. CR is considered whitespace though, so source files with CRLF (windows) line endings should work. CR-only line endings are not supported.
+
+## Overview of syntax
+
+This section lists all operators and keywords and shows example syntax of how they are used.
+
+### Primitives
+
+Below you can see some variables being declared and some values being assigned to them.
+
+```
+var a = nil; //nil
+var b = false; // booleans
+var c = true;
+var d = 4; //numbers (internally all floating point)
+var e = 8.2;
+var f = "abcde"; // strings
+```
+
+### Collections
+
+Below you can see lists and tables being defined and manipulated.
+
+Lists:
+
+```
+// defining a list with 3 elements
+var list = @[12, "fox", false];
+
+// printing the second element (should output fox)
+// indexing starts at 0
+print(list[1]);
+
+// editing the first element, then printing it (should output 36.0)
+list[0] = list[0] * 3;
+print(list[0]);
+
+// adding a fourth element (larger indexes than the one about to be added are not allowed)
+list[3] = 50;
+print(list[3]); //should print 50.0
+
+// Getting the length of the list, should be 4.0:
+print(#list);
+
+```
+
+Tables:
+
+```
+// defining a table
+var table = @{
+  [1] = "one", // keys are normally in brackets []
+  [2]: "two", // both = and : are allowed
+  three = 3, // string keys can be shortened - no brackets or quotes
+  four: 4, // string keys can also use colon
+  [2+3] = "five" // the square brackets can enclose whole expressions, just like the values can be expressions
+};
+
+// printing a member
+print(table[1]); // will print one
+
+// indexing with dot
+print(table.three); // will print 3.0
+
+// adding new elements
+table[6] = "six";
+
+// printing the number of elements
+print(#table); // should print 6
+
+```
+
+Both tables and lists can have any types as values (+ tables can have any type as keys). Internally lists are a resizeable array, while tables are a hashmap.
+
+### Control flow
+
+For control flow, the choice is between `if`, `or`, `and` and `while` expressions.
+
+If expressions have the following basic syntax:
+
+```
+if (condition) expression [else expression]
+```
+
+```
+// example:
+var x = if (true) 5 else 6;
+```
+
+However, since expression statements are possible (discarding the result), and block expressions are possible, the following combination of building blocks might be more familiar:
+
+```
+if (condition) {
+  statements;
+} else {
+  statements;
+};
+```
+
+If the condition is truthy, if expressions return the result of the "then expression". If the condition is falsey, they return the result of the "else expression", or the condition itself if else is absent. The then/else expressions are also only evaluated if their result is returned.
+
+`and` expressions have the syntax:
+
+```
+(left) and (right)
+```
+
+```
+// example:
+var x = true and 5;
+```
+
+The `and` operator evaluates the left hand side, and if it is truthy it evaluates and returns the right hand side. If the left hand side is falsey, it returns the left hand side.
+
+`or` expressions have the syntax:
+
+```
+(left) or (right)
+```
+
+```
+// example
+var x = false or 5;
+```
+
+The `or` operator evaluates the left hand side, and if it is truthy it returns it. If it is falsey, it evaluates and returns the right hand side.
+
+While expressions have the following syntax:
+
+```
+while (condition) expression
+```
+
+```
+// example:
+var res = while (condition) expression;
+```
+
+While expressions continually check the condition, and if it is truthy, they evaluate the expression. Their return value is the return value of the expression (the "body") during the last run. If the expression (the "body") is never evaluated, the whole while expression evaluates to `nil`.
+
+### Other basic operators
+
+There are some other basic operators not covered yet. These are the basic mathematical operators, the logical operator not, equality and comparison operators:
+
+- Equality (`==`, `!=`)
+- Comparison (`<` ,`>`, `<=`, `>=`)
+- Addition/Subtraction (`+`, `-`)
+- Multiplication/Division (`*`, `/`)
+- Negation (`-`)
+- Logical not (`!`)
+- String concatenation (`+`)
+
+### Grouping expressions
+
+If the default precedence (see section Operator precedence) is undesired, parentheses can be used to "group" a part of an expression. This works as you would probably expect. Here is an example:
+
+```
+print(5*(2+4));
+// computes 2+4 first, then multiplies the result of it by 5, then prints it
+```
+
+Another way of grouping expressions is the ampersand operator. The ampersand operator is equivalent of putting the whole expression to the left of it (from the start of the expression, or from the previous ampersand operator if one is already present to the left) until the ampersand symbol. Here is an example:
+
+```
+print(5 + 6 & * 7);
+// calculates 5+6 first, because & groups everything to the left of it, then multiplies the result of the addition by 7, then prints it
+```
+
+The ampersand operator can be useful for example to set multiple indexes of a list/table in one line, since the index set operator []= returns the original list/table whose elements are being set.
+
+```
+var table = @{};
+
+// just like []=, .= also returns the table
+table.one = 1 &.two = 2 &.three = 3 &.four = 4;
+
+print(table.three);
+//should print 3.0
+```
+
+### Procedures
+
+TODO
+
+- two ways of defining them
+- calling them
+
+#### Closures
+
+TODO
+
+#### Alternative calling syntaxes
+
+TODO
+
+- `::` piping syntax
+- `:` lua-like method syntax
+
+### Block expressions
+
+TODO
+
+- block expressions
+- labels
+- break statements
+
+### Variables, scope
+
+TODO
+
+- locals
+- globals
+
+## Operator precedence
+
+The following is the precedence of operators, from least to most:
+- Ampersand (`&`)
+- Assignment (`=`)
+- Or (`or`)
+- And (`and`)
+- Equality (`==`, `!=`)
+- Comparison (`<` ,`>`, `<=`, `>=`)
+- Addition (`+`, `-`)
+- Multiplication (`*`, `/`)
+- Unary (`-`, `!`, `#`)
+- Calls and indexing (`[]`, `.`, `()`, `:`)
+- Parentheses (`()`, `{}`)
+
+Binary operators are left associative, while unary ones are right associative. All unary operators in the unary precedence level are prefixes, however the call () could be viewed as a suffix unary operation, so not all unary operators are prefixes.
+
+Some other things that are internally part of this precedence list are:
+- if, while expressions - unary precedence
+- list, table and procedure declarations and constants - same precedence as parentheses
+
+In these cases, please note that whatever is inside these will look for a whole expression, so things like the following are equivalent:
+
+```
+var x = 5 + if (true) 1 else 3 + 4;
+//equivalent to
+var x = 5 + (if (true) 1 else 3 + 4)
+```
+
+## Standard library
+
+TODO write section
+
+# Design notes
+
+TODO very incomplete section
+
+- calling functions should not result in any new memory allocations - should be fast
+
+# Roadmap
+
+(All the hidden todo.txt's were cleaned up. Here is a public list of everything that was on them that is still relevant.)
+
+First, compiler v2 has to be completed.
+
+The following features are currently missing, and may or may not be added to nds:
+- Garbage collection
+- `for (i in list) expr` type expressions for iterating lists, tables
+- `[for (i in list) expr]` type generators
+- Easy shell integration with `%` to run commands and pipes with `|`
+- `x &{ :it = 6; }` type manipulation "ampersand blocks"
+- tuples `@(1, 2)`
+- exceptions, coroutines
+- goto @label, but with scope safety so goto has no chance of being dangerous
+- OOP with inheritance
+- defer statements
+- better error reporting
+
+The following details could be changed:
+- for safety, check whether the script has the +x permission before running it (also check if it is located on a noexec mount)
+- dedicated hash set for string interning
+
+The following optimizations could be added:
+- opcodes for arithmetic with a small number on a local variable, such as "add 3 to local variable 5"
+- incrementing/decrementing opcodes
+- tail recursion optimization
+- variable argument length for all opcodes (with templates to manage extra complexity in VM without degrading performance)
+- inlining
+
+# Known bugs
+
+(Note: might not be an exhaustive list of bugs I know of, sorry)
+
+- printing self referential tables results in endless loop

tests/ampersand.nds

@@ -8,6 +8,16 @@ x[5] = 1 & [6] = 0 & [7] = 3 & [7] = 2;
 //expect:@[ 0.0, 2.0, 0.0, 4.0, 5.0, 1.0, 0.0, 2.0 ]
 print (x);
 
+
+// for tables with dot syntax
+
+var table = @{};
+
+table.one = 1 &.two = 2 &.three = 3 &.four = 4;
+
+print(table.three);
+//expect:3.0
+
 // not very useful but still must be correct behavior tests:
 
 // change of precedence where interjected

tests/collections.nds

@@ -1,4 +1,4 @@
-// a test about collections, WIP
+// a test about collections
 
 var returnlist = proc()
   @[1, 2, 3, 4]
@@ -13,4 +13,63 @@ print (a[0][0]);
 
 a[0][0] = 3;
 print (a[0][0]);
-//expect:3.0
+//expect:3.0
+
+// adding new values
+
+var letters = @["a", "b", "c"];
+var letterNumbers = @{};
+
+letters[3] = "d";
+print(letters[3]); //expect:d
+
+var i = 0;
+while (i < #letters) {
+  letterNumbers[letters[i]] = i;
+};
+
+print(letterNumbers.d); //expect:3.0
+
+// from reference.md
+
+var list = @[12, "fox", false];
+
+//expect:fox
+print(list[1]);
+
+list[0] = list[0] * 3;
+print(list[0]);
+//expect:36.0
+
+list[3] = 50;
+print(list[3]);
+//expect:50.0
+
+print(#list);
+//expect:4.0
+
+// table example from reference.md
+
+// defining a table
+var table = @{
+  [1] = "one", // keys are normally in brackets []
+  [2]: "two", // both = and : are allowed
+  three = 3, // string keys can be shortened - no brackets or quotes
+  four: 4, // string keys can also use colon
+  [2+3] = "five" // the square brackets can enclose whole expressions, just like the values can be expressions
+};
+
+// printing a member
+print(table[1]); // will print one
+//expect:one
+
+// indexing with dot
+print(table.three); // will print 3.0
+//expect:3.0
+
+// adding new elements
+table[6] = "six";
+
+// printing the number of elements
+print(#table); // should print 6
+//expect:6.0

tests/scanner.nds

@@ -21,7 +21,7 @@ print (áéíóú);
   {
     :å = "result";
     break @å;
-    // this convolution needed because breaks detect code after them and error
+    // this nesting is needed because breaks detect code after them and error
   };
   print ("after");
 } :: print;