japl/tests/testmarkup.md

# JATS test markup specification

This document specifies the format tests are written
in for JATS (Just Another Test Suite).

## Definitions

A **test file** represents a file inside the `~japl/tests/japl/` directory. All of these files are parsed
to find any defined tests.

A test file can contain multiple tests. A **test** is a
piece of JAPL source code, along with its input
(given in stdin) and expected output (in stdout and stderr), or the means to construct these fields. It
may also contain other metadata, such as the name 
of the test, or whether it is to be skipped.

The test files are parsed line by line. The parser
switches **modes** during the parsing. The modes
dictate the state of the parser, and where each line
ends up (which field of which test).

There is a special mode, **root mode**, which describes
the state of the parser when it just enters the file.
It can stay in this state during parsing, can leave
it by entering another mode and can
return to it, by leaving any mode it has entered.

## Test files

Must be utf-8 (for now only ascii was tested).
Must not contain a BOM. Line endings must be a single 
`\n`. Please configure your editor to support this.

## Syntax

### Mode syntax

The modes are constructed from modelines,
which are lines starting with the character '[', or alternatively
they can also start with the sequence "//[".
Modelines also have to be closed by a ']' character
on the end of this line. These lines may not contain
whitespace before the opening '[' or "//[" nor after then ending
']' characters. Inside the brackets, letters (case
insensitive), numbers, underscores and dashes form
a name describing what the modeline does.

```
[ name ]
//[ name ]
```

Optionally, an argument may be passed, which is 
separated by a colon.

```
[ name : detail ]
//[name: detail]
```

Whitespace inside the brackets is ignored (even inside
names). More than one colon, or any character that
is not whitespace, a letter, a digit, a colon, an 
underscore or a dash inside is a syntax error, which
results in a fatal error, causing the whole test
file to be invalid. The user is always warned when
such a fatal syntax error occurs.

It is possible for lines beginning with '[' to not
be modelines. When a line starts with '[[', it escapes
the opening left bracket, as if the line was a regular
one (the '[[' is reduced to '['). When a line starts
with '[;' it is a comment, which is not a modeline nor
a line that shows up in the current mode.

A different mode can be entered and left by the 
following syntax.

```
[Modename]

[end]
```

A modeline that is not defined to be a legal one 
for the current mode is a syntax error, which
invalidates the whole test file. It also raises 
a visible warning to the user.

## Possible modes

### Root mode

Inside the root mode, all lines that are not modelines
are assumed to be comments.

There is one possible mode to enter from the root mode,
a test mode. The test modes are entered when the "test"
mode line is specified. The detail for the modeline
corresponds to the name of the test.

```
[test: testname]

[end]
```

### Test modes

Inside test modes, all lines that are not modelines are
assumed to be comments.

There are different modelines that do actions or
modes that can be entered from tests. They are all
defined below.

#### Skipping a test

The modeline `skip` skips a test. It does not enter
a different mode, so no `end` is neccessary.

```
[skip]
```

#### Adding JAPL source to a test

The modeline `source` enters the mode source, which
is useful for appending to the JAPL source of the
test.
```
[source]
print("Hello from JAPL!");
[end]
```

There are two kinds of source modes, raw and mixed.
Mixed source mode can be entered if the detail `mixed`
is specified. Raw source mode can be entered if the
detail `raw` is specified. When no detail is specified,
raw source mode is assumed.

In raw source mode, all lines in the mode are
appended as they are to the JAPL source. In mixed
mode, comments inside this JAPL source can be
added to add lines to the expected stdout/stderr or
the stdin of the test using the legacy test format.

They are defined by the sequences `//stdout:`, 
`//stderr:`, `//stdin:`, `//matchout:` and 
`//matcherr:`. Every character after the colon and
before the end of the line is appended to the respective
field of the test. `stdout` adds a raw line to be
matched to the expected stdout of the test. `matchout`
adds a regex to match a line of the stdout of the test.
`stderr` and `matcherr` are the stderr equivalents.
`stdin` adds a line to the stdin that the JAPL source
can read from.

```
[source: mixed]
print("Hello from JAPL!");//stdout:Hello from JAPL!
[end]
```

#### Adding expected output to the test

The mode `stdout` can add standard output to expect
from the JAPL source when it is ran.

```
[test: hello]
[source: raw]
print("Banana.");
[end]
[stdout]
Banana.
[end]
[end]
```

The option `re` can be added if every line is to
be a regex matched against a line of stdout.
The option `nw` will strip leading and trailing 
whitespace from every line in the mode before
adding it to the expected lines. The option `nwre`
adds regex based matching lines after stripping
whitespace.

The mode `stderr` does the same as `stdout`, but
for the standard error. It accepts the same options.

#### Adding input to the test

The mode `stdin` can add standard input that the
JAPL source of the test can read from.

```
[test: inputtest]
[source: raw]
print(readLine());
[end]
[stdin]
Hello there
[end]
[stdout]
Hello there
[end]
```

#### Adding python to the tests

Coming soon.

# Best practices

Tests should be written so that they are valid jpl code. The test title
and modes surrounding source code should be prefixed with `//`. Stdin/stdout 
and other raw non-jpl sources should be inside `/* */` blocks. Single line
commands such as skips should be either prefixed with `//` or inside a `/* */`
block.