# Chapter 16: Testing

**What you'll learn:** how to write tests as ordinary Loup functions, check results with the `debug.assert` helpers, run them with `loup test`, read a failure report, skip a test that cannot run, and lay out test files so both of Loup's test runners find them.

A Loup test is a top-level function whose name starts with `test_` and that takes no parameters. It passes if it returns and fails if it throws. The helpers in `debug.assert` throw an `AssertionError` with a message that names the values involved, and `loup test` finds the test functions, runs each one, and reports the results. There is nothing to install and no configuration file to write.

## A First Test

Save this as `wordcount_test.lpscr`:

```loup
@mode(host.scripted)

from debug.assert import (expect, expectEq);

## The number of words in `text`, separated by spaces.
func countWords(text: string): int {
    var count = 0;
    for (word in text.split(" ")) {
        if (word.length() > 0) {
            count = count + 1;
        }
    }
    return count;
}

func test_counts_words(): void {
    expectEq(countWords("the cat sat"), 3, "three words");
}

func test_ignores_extra_spaces(): void {
    expectEq(countWords("  the   cat  "), 2, "extra spaces");
}

func test_empty_text_has_no_words(): void {
    expect(countWords("") == 0, "empty text");
}
#~ test_counts_words();
#~ test_ignores_extra_spaces();
#~ test_empty_text_has_no_words();
```

*Listing 16-1: wordcount_test.lpscr, a function and three tests for it*

- `from debug.assert import (expect, expectEq);` brings in two assertion helpers. `expect(condition, message)` fails when the condition is false. `expectEq(actual, expected, context)` fails when the two values differ.
- Each `test_` function checks one behaviour. Its name is what the report shows, so make it say what should be true.
- The file name ends in `_test.lpscr`. That is how the test runners recognise a test file.

Run the tests in that file:

```sh
loup-bootstrap test --file wordcount_test.lpscr
```

```text
found 3 tests in 1 files (0 compile failures)
  ✓ ./wordcount_test.lpscr::test_counts_words (20ms)
  ✓ ./wordcount_test.lpscr::test_empty_text_has_no_words (19ms)
  ✓ ./wordcount_test.lpscr::test_ignores_extra_spaces (20ms)

Tests: 3 total, 3 passed, 0 failed, 0 skipped (20ms)
```

Each line names the file and the test, with `✓` for a pass, `✗` for a failure and `-` for a skipped test. Tests run in alphabetical order of their names, so a test must not depend on another test having run first. `loup-bootstrap test` exits with status 0 when every test passes and 1 when any test fails, so it can gate a CI job.

> **Which command?** On macOS, Windows and Linux arm64, `loup test` and `loup-bootstrap test` are the same program. On Linux x86-64, `loup test` runs a second, self-hosted runner, described at the end of this chapter. The examples in this chapter use `loup-bootstrap test`, which behaves the same everywhere.

### Mistake: a bare `assert`

If you have written tests in Python, you will want to write `assert`:

```loup,does_not_compile
@mode(host.scripted)

func countWords(text: string): int {
    return text.split(" ").length();
}

func test_counts_words(): void {
    assert(countWords("the cat sat") == 3, "three words");
}
```

```text
main.lpscr:8:5: error[E0001]: undefined identifier `assert`
```

Loup has no built-in `assert`. The checks live in the `debug.assert` module, and a test file imports the ones it uses. `expect` is the general-purpose one:

```loup
@mode(host.scripted)

from debug.assert import (expect);

func countWords(text: string): int {
    return text.split(" ").length();
}

func test_counts_words(): void {
    expect(countWords("the cat sat") == 3, "three words");
}
#~ test_counts_words();
```

(The self-hosted runner on Linux x86-64 supplies a bare `assert(condition, message)` to the files it runs, so a test that uses it passes there and fails to compile everywhere else. Import `expect` and the file works with every runner and with `loup run`.)

### Mistake: leaving out the context

Every `debug.assert` helper takes a final description of what is being checked:

```loup,does_not_compile
@mode(host.scripted)

from debug.assert import (expectEq);

func test_sum(): void {
    expectEq(1 + 1, 2);
}
```

```text
main.lpscr:6:5: error[E0001]: function 'expectEq' expects 3 argument(s), but 2 provided
```

The context is not optional, because it is the first thing the failure report prints. Write what the value means — `expectEq(1 + 1, 2, "one plus one")` — and a failure tells you which check went wrong without opening the file.

## When a Test Fails

Text often contains tabs. Add a test for that to `wordcount_test.lpscr`:

```loup,not_desired_behavior
#~ @mode(host.scripted)
#~
#~ from debug.assert import (expect, expectEq);
#~
#~ func countWords(text: string): int {
#~     var count = 0;
#~     for (word in text.split(" ")) {
#~         if (word.length() > 0) {
#~             count = count + 1;
#~         }
#~     }
#~     return count;
#~ }
#~
func test_tabs_separate_words(): void {
    expectEq(countWords("the\tcat"), 2, "tab-separated words");
}
```

*Listing 16-2: A new test that exposes a bug*

```sh
loup-bootstrap test --file wordcount_test.lpscr
```

```text
found 4 tests in 1 files (0 compile failures)
  ✓ ./wordcount_test.lpscr::test_counts_words (22ms)
  ✓ ./wordcount_test.lpscr::test_empty_text_has_no_words (23ms)
  ✓ ./wordcount_test.lpscr::test_ignores_extra_spaces (22ms)
  ✗ ./wordcount_test.lpscr::test_tabs_separate_words (22ms)
    AssertionError: tab-separated words: expected 2, got 1

Tests: 4 total, 3 passed, 1 failed, 0 skipped (24ms)
```

The failure line is built from the arguments to `expectEq`: the context you wrote, then the expected value, then the actual one. `countWords` splits only on spaces, so `"the\tcat"` is one word. The test is right and the function is wrong, so fix the function:

```loup
#~ @mode(host.scripted)
#~
#~ from debug.assert import (expectEq);
#~
## The number of words in `text`, separated by spaces or tabs.
func countWords(text: string): int {
    var count = 0;
    for (word in text.replace("\t", " ").split(" ")) {
        if (word.length() > 0) {
            count = count + 1;
        }
    }
    return count;
}
#~
#~ func test_tabs_separate_words(): void {
#~     expectEq(countWords("the\tcat"), 2, "tab-separated words");
#~ }
#~ test_tabs_separate_words();
```

*Listing 16-3: countWords treats a tab like a space*

Run the tests again:

```sh
loup-bootstrap test --file wordcount_test.lpscr
```

```text
found 4 tests in 1 files (0 compile failures)
  ✓ ./wordcount_test.lpscr::test_counts_words (24ms)
  ✓ ./wordcount_test.lpscr::test_empty_text_has_no_words (28ms)
  ✓ ./wordcount_test.lpscr::test_ignores_extra_spaces (24ms)
  ✓ ./wordcount_test.lpscr::test_tabs_separate_words (24ms)

Tests: 4 total, 4 passed, 0 failed, 0 skipped (28ms)
```

## The `debug.assert` Helpers

`debug.assert` (Language Reference §21.18) has seven checks. Each throws `AssertionError` when it fails, with a message that names the values:

| Helper | Fails when | Failure message |
|---|---|---|
| `expect(condition, message)` | `condition` is false | `message` |
| `expectEq(actual, expected, context)` | `actual != expected` | `context: expected E, got A` |
| `expectNeq(actual, forbidden, context)` | `actual == forbidden` | `context: value was equal to forbidden F` |
| `expectSome(option, context)` | the `Option` is `None`; otherwise returns the value | `context: expected Some, got None` |
| `expectContains(items, item, context)` | `item` is not in the list | `context: list does not contain I` |
| `expectThrows(action, context)` | calling `action` does not throw; otherwise returns the exception | `context: expected an exception, none thrown` |
| `expectNear(actual, expected, tolerance, context)` | the floats differ by more than `tolerance` | `context: expected E +/- T, got A` |

`expectThrows` and `expectSome` hand back what they found, so a test can check it further:

```loup
@mode(host.scripted)

from debug.assert import (expectEq, expectSome, expectThrows, expectNear, expectContains);

## A TCP port number from text; throws ValueError outside 1..65535.
func parsePort(text: string): int {
    let n = text.toInt();
    if (n < 1 or n > 65535) {
        throw ValueError(i"port out of range: {$ text $}");
    }
    return n;
}

func test_port_out_of_range_throws(): void {
    let e = expectThrows(func(): void { parsePort("70000"); }, "parsePort(\"70000\")");
    expectEq(e.message, "port out of range: 70000", "error message");
}

func test_default_port_is_configured(): void {
    let settings: dict[string, int] = {"port": 8080};
    let port = expectSome(settings.get("port"), "port setting");
    expectEq(port, 8080, "port value");
}

func test_average_load(): void {
    let samples = [0.5, 0.25, 0.75];
    let average = (samples[0] + samples[1] + samples[2]) / 3.0;
    expectNear(average, 0.5, 0.0001, "average load");
}

func test_known_schemes(): void {
    expectContains(["http", "https"], "https", "supported schemes");
}
#~ test_port_out_of_range_throws();
#~ test_default_port_is_configured();
#~ test_average_load();
#~ test_known_schemes();
```

*Listing 16-4: expectThrows, expectSome, expectNear and expectContains*

The action passed to `expectThrows` is a function value with no parameters: `func(): void { … }` (Chapter 3).

### Mistake: comparing two lists with `expectEq`

```loup,not_desired_behavior
@mode(host.scripted)

from debug.assert import (expectEq);

## The words of `text`, lowercased.
func words(text: string): list[string] {
    var out: list[string] = [];
    for (raw in text.split(" ")) {
        if (raw.length() > 0) {
            out.push(raw.toLowerCase());
        }
    }
    return out;
}

func test_words_are_lowercased(): void {
    expectEq(words("The CAT"), ["the", "cat"], "lowercased words");
}
```

```sh
loup-bootstrap test --file words_test.lpscr
```

```text
found 1 tests in 1 files (0 compile failures)
  ✗ ./words_test.lpscr::test_words_are_lowercased (13ms)
    AssertionError: lowercased words: expected ["the", "cat"], got ["the", "cat"]

Tests: 1 total, 0 passed, 1 failed, 0 skipped (13ms)
```

The two lists print the same and the test still fails. The Language Reference says `==` on lists compares element by element (ruling EQ-3), but Loup 1.25 does not implement that yet: two separately built lists are never `==`, so `expectEq` and `expectNeq` cannot compare them. Until it does, compare something that does compare by value — the joined text, or the length and each element:

```loup
#~ @mode(host.scripted)
#~
#~ from debug.assert import (expectEq);
#~
#~ func words(text: string): list[string] {
#~     var out: list[string] = [];
#~     for (raw in text.split(" ")) {
#~         if (raw.length() > 0) {
#~             out.push(raw.toLowerCase());
#~         }
#~     }
#~     return out;
#~ }
#~
func test_words_are_lowercased(): void {
    let found = words("The CAT");
    expectEq(found.length(), 2, "number of words");
    expectEq(found.join(" "), "the cat", "lowercased words");
}
#~ test_words_are_lowercased();
```

*Listing 16-5: Comparing a list through its length and its joined text*

Numbers, strings, booleans, tuples, enums and `Option` values compare by value in `expectEq`. Lists, dicts and sets do not, and neither do structs once they are passed as `Any` (as `expectEq`'s parameters are): compare a struct's fields one at a time.

### Errors that are not assertions

A test also fails when the code under test throws something other than an `AssertionError`:

```loup,not_desired_behavior
@mode(host.scripted)

from debug.assert import (expectEq);

## The average length of `words`, rounded down.
func averageLength(words: list[string]): int {
    var total = 0;
    for (w in words) {
        total = total + w.length();
    }
    return total / words.length();
}

func test_average_of_no_words(): void {
    expectEq(averageLength([]), 0, "average of nothing");
}
```

```sh
loup-bootstrap test --file average_test.lpscr
```

```text
found 1 tests in 1 files (0 compile failures)
  ✗ ./average_test.lpscr::test_average_of_no_words (12ms)
    AssertionError: integer division or modulo by zero

Tests: 1 total, 0 passed, 1 failed, 0 skipped (12ms)
```

The message is the `IntegerDivisionError`'s, but the report labels it `AssertionError`. Loup 1.25's runner keeps a failed test's message and not its exception type (Testing Reference §23.6), so read the message, not the label, to tell a broken assumption from a crash. The fix belongs in `averageLength`: decide what the average of nothing is (here, `if (words.length() == 0) { return 0; }`) and keep the test.

## Skipping a Test

Some tests need something the machine may not have: a large data file, a network service, a tool. `testing.skipTest(reason)` stops the test and records it as skipped rather than failed:

```loup
@mode(host.scripted)

from debug.assert import (expect);
from testing import (skipTest);
#~ from io.fs import (createDir);
#~ createDir("testdata");
#~ writeText("testdata/corpus.txt", "the cat sat on the mat\n");

func test_big_corpus(): void {
    if (not exists("testdata/corpus.txt")) {
        skipTest("testdata/corpus.txt is not checked out");
    }
    expect(readText("testdata/corpus.txt").length() > 0, "corpus is not empty");
}
#~ test_big_corpus();
```

*Listing 16-6: corpus_test.lpscr skips its test when the input file is missing*

On a machine without `testdata/corpus.txt`:

```sh
loup-bootstrap test --file corpus_test.lpscr
```

```text
found 1 tests in 1 files (0 compile failures)
  - ./corpus_test.lpscr::test_big_corpus (12ms)

Tests: 1 total, 0 passed, 0 failed, 1 skipped (12ms)
```

The pretty report does not print the reason; it is a note for the person reading the test. A skip does not make the run fail.

## Organizing Tests

Put each test file next to the module it tests. Here is a project made with `loup-bootstrap init wordstats`, with `wordcount_test.lpscr` from earlier moved into it and a module of text helpers added:

```text
wordstats/
├── wordstats.lppkg
└── src/
    ├── main.loup
    ├── textstats.lpscr
    ├── textstats_test.lpscr
    └── wordcount_test.lpscr
```

The module is an ordinary script module with `pub` functions:

```loup
@mode(host.scripted)

## The words of `text`, lowercased. Spaces and tabs separate words.
pub func words(text: string): list[string] {
    var out: list[string] = [];
    for (raw in text.replace("\t", " ").split(" ")) {
        if (raw.length() > 0) {
            out.push(raw.toLowerCase());
        }
    }
    return out;
}

## How many times each word occurs.
pub func frequencies(text: string): dict[string, int] {
    var counts: dict[string, int] = {};
    for (w in words(text)) {
        counts[w] = (counts.get(w) ?? 0) + 1;
    }
    return counts;
}
```

*Listing 16-7: src/textstats.lpscr*

The test file imports it by name, as any other script would:

```loup,ignore
@mode(host.scripted)

from debug.assert import (expectEq);
from textstats import (words, frequencies);

func test_words_are_lowercased(): void {
    expectEq(words("The CAT").join(" "), "the cat", "lowercased words");
}

func test_frequencies_count_repeats(): void {
    let counts = frequencies("the cat and the hat");
    expectEq(counts["the"], 2, "count of 'the'");
    expectEq(counts["cat"], 1, "count of 'cat'");
}
```

*Listing 16-8: src/textstats_test.lpscr (it imports Listing 16-7, so it runs as a pair)*

In a project, `loup-bootstrap test` with no arguments finds every test file:

```sh
cd wordstats
loup-bootstrap test
```

```text
found 6 tests in 2 files (0 compile failures)
  ✓ src/textstats_test.lpscr::test_frequencies_count_repeats (29ms)
  ✓ src/textstats_test.lpscr::test_words_are_lowercased (28ms)
  ✓ src/wordcount_test.lpscr::test_counts_words (33ms)
  ✓ src/wordcount_test.lpscr::test_empty_text_has_no_words (28ms)
  ✓ src/wordcount_test.lpscr::test_ignores_extra_spaces (29ms)
  ✓ src/wordcount_test.lpscr::test_tabs_separate_words (29ms)

Tests: 6 total, 6 passed, 0 failed, 0 skipped (33ms)
```

(The runner prints each file's full path; the listings in this chapter shorten it to the part inside the project.)

### How test files are found

`loup-bootstrap test` reads the project manifest, `<name>.lppkg`, from the current directory or a parent. Its `[test]` table, as `loup-bootstrap init` writes it, is:

```toml
[test]
pattern = "**/*_test.loup"
parallel = true
timeout = 30
```

The runner treats as test files:

- every file matching `pattern` — a pattern ending in `.loup` also matches the same name ending in `.lpscr`, so the default finds `*_test.loup` and `*_test.lpscr` anywhere in the project;
- every `*.test.loup` and `*.test.lpscr` file;
- every `.loup` and `.lpscr` file under the project's `tests/` directory.

In each test file, a test is a top-level function named `test_…` with no parameters. A `test_` function that takes a parameter is not a test and is silently ignored. Outside a project, use `--file`, as in the first examples.

### Mistake: a test file the runner does not see

Python's convention is `test_textstats.py`. Name a Loup file that way and nothing complains:

```sh
mv src/textstats_test.lpscr src/test_textstats.lpscr
loup-bootstrap test
```

```text
found 4 tests in 1 files (0 compile failures)
  ✓ src/wordcount_test.lpscr::test_counts_words (19ms)
  ✓ src/wordcount_test.lpscr::test_empty_text_has_no_words (19ms)
  ✓ src/wordcount_test.lpscr::test_ignores_extra_spaces (21ms)
  ✓ src/wordcount_test.lpscr::test_tabs_separate_words (20ms)

Tests: 4 total, 4 passed, 0 failed, 0 skipped (21ms)
```

`test_textstats.lpscr` matches no discovery rule, so its two tests silently disappear from the report and the run still succeeds. A project whose only test file is misnamed reports `found 0 tests in 0 files` and exits with status 0. If a new test never shows up in the report, check the file name first: the suffix is `_test`, as in `textstats_test.lpscr`.

### Tests in a `tests/` directory

You can keep tests in `tests/` instead, but they then need a relative import to reach `src/`: `from ..src.textstats import (words);`. `loup-bootstrap test` accepts that; the self-hosted runner described below does not run a file with a relative import and reports it as a failure. Test files beside their modules, importing them by plain name, work with both runners.

## Choosing What to Run and How to Report It

```sh
loup-bootstrap test --filter tabs                # tests whose name contains "tabs"
loup-bootstrap test --file src/textstats_test.lpscr
loup-bootstrap test --format json                # one JSON document
loup-bootstrap test --format junit               # JUnit XML, for CI servers
loup-bootstrap test --no-color                   # plain text for logs
loup-bootstrap test --no-parallel --timeout 120  # sequential, 120 s per test
```

A filter selects tests by their qualified name (`file::test_name`):

```sh
loup-bootstrap test --filter tabs
```

```text
found 6 tests in 2 files (0 compile failures)
  ✓ src/wordcount_test.lpscr::test_tabs_separate_words (11ms)

Tests: 1 total, 1 passed, 0 failed, 0 skipped (11ms)
```

The JSON report has one entry per test, with the failure's message. This is the tab test from Listing 16-2, before the fix, run with `loup-bootstrap test --format json --file wordcount_test.lpscr --filter tabs` (the runner prints its `found …` line first):

```text
{
  "duration_ms": 12,
  "failed": 1,
  "passed": 0,
  "results": [
    {
      "duration_ms": 12,
      "error": {
        "message": "tab-separated words: expected 2, got 1",
        "trace": [],
        "type": "AssertionError"
      },
      "name": "./wordcount_test.lpscr::test_tabs_separate_words",
      "status": "fail"
    }
  ],
  "skipped": 0,
  "total": 1
}
```

`loup-bootstrap test --help` lists the rest, including `--native`, which builds `host.native` tests ahead of time, and `--freestanding`, which runs freestanding tests under QEMU on Linux.

## The Self-Hosted Runner on Linux x86-64

On Linux x86-64, `loup test` hands the run to `loup-test`, the self-hosted runner (Testing Reference §23, §23.9). It is written in Loup and works differently:

- `loup test [DIR]` looks for `*_test.lpscr` and `*_test.loup` files under each directory (default: the current one). It needs no manifest.
- Each file runs in its own `loup run --allow-all` process, through a generated driver that calls every argument-less `test_*` function and reports `PASS`, `FAIL`, `SKIP` or `STALE` (a file that reported nothing) per file.
- Test bodies may call a bare `assert(condition, message)`; the driver supplies it as `debug.assert.expect` unless the file imports its own.
- `--strict` turns a file that reports nothing, a run that finds no test files, and a skipped test into failures. Use it in CI: it catches the misnamed file from the previous section.
- A `loup-test.toml` file at or above a test sets its `timeout`, `max-mem`, `louppath` and `allow-skip`.
- Each test process gets two environment variables: `LOUP_TEST_FILE` (the test file) and `LOUP_TEST_TMP` (a fresh scratch directory that is removed afterwards).

Stage0's `loup-bootstrap test` has no `--strict`. A test file written as in this chapter — named `*_test.lpscr`, importing its `debug.assert` helpers, next to the module it tests — runs unchanged under both runners.

## What Is Not There Yet

> ⚠ **Planned.** Test functions declared `async func` or `gen func` are found but not driven: the runner calls them as plain functions, so an `async` test's `Task` is never awaited (Testing Reference §23.6). Keep test functions synchronous.

> ⚠ **Planned.** The Testing Reference (a Draft) describes a larger framework — test roots marked by `__testroot__.loup`, dependency injection with resource factories, markers and include/exclude expressions, test sequencing and constraints — in its §1–§12, and reserves section numbers for mocking, property-based testing and tandem validators. None of it ships in Loup 1.25. The `testing` module provides `runTest`, `skipTest`, `TestResult`, `TestSummary`, `TestFailure` and `SkipSignal`; `debug.assert` provides the checks.

Until then, the patterns are plain code. To replace a dependency in a test, pass it in: a function that takes a `fn(string): string` for "fetch this URL" can be given a lambda that returns canned text. For setup and cleanup, write a helper that builds the fixture and call it at the start of each test, and put cleanup that must run even when the test fails in a `finally` block (`defer` is accepted but does not run in Loup 1.25; see Chapter 4).

## Summary

- A test is a top-level `func test_…(): void` with no parameters, in a file whose name ends in `_test.lpscr` (or `_test.loup`).
- Tests fail by throwing. Import the checks from `debug.assert`: `expect`, `expectEq`, `expectNeq`, `expectSome`, `expectContains`, `expectThrows`, `expectNear`. Each takes a context string that the failure message starts with.
- `loup-bootstrap test` runs a project's tests (or one file with `--file`), prints `✓`/`✗`/`-` per test, and exits 1 if any test failed.
- `expectEq` cannot compare two lists in Loup 1.25 (`==` on lists is not element-wise yet); compare their length and joined text.
- A non-assertion exception fails the test too, but is labelled `AssertionError` in the report; read the message.
- `testing.skipTest(reason)` skips a test that cannot run here.
- A misnamed test file is silently ignored. Keep tests beside their modules as `*_test.lpscr`, importing them by name, and they run under both runners; on Linux x86-64, `loup test --strict` also fails a run that finds no tests.

## Try It Yourself

1. Write `slug_test.lpscr` with a function `slug(title: string): string` that lowercases a title and replaces spaces with `-`, and three tests for it: a plain title, a title with several spaces in a row, and an empty title.
2. Add a test that `slug` keeps digits. Make it fail first, read the report, then make it pass.
3. Write `parseConfig(text: string): dict[string, string]` for `key=value` lines and use `expectThrows` to test that a line without `=` throws a `ValueError`.
4. Rename `slug_test.lpscr` to `test_slug.lpscr` inside a project and watch the tests disappear from the report.

Next: [Chapter 17](ch17-js-host-interop.md) is a design preview of JavaScript interop; [Chapter 18](ch18-building-script-tools.md) builds a complete command-line tool.