Chapter 16: Testing

What you'll learn: how to write tests as ordinary Loup functions, check results with the debug.assert helpers, run them with loup test, read a failure report, skip a test that cannot run, and lay out test files so both of Loup's test runners find them.

A Loup test is a top-level function whose name starts with test_ and that takes no parameters. It passes if it returns and fails if it throws. The helpers in debug.assert throw an AssertionError with a message that names the values involved, and loup test finds the test functions, runs each one, and reports the results. There is nothing to install and no configuration file to write.

A First Test

Save this as wordcount_test.lpscr:

@mode(host.scripted)

from debug.assert import (expect, expectEq);

## The number of words in `text`, separated by spaces.
func countWords(text: string): int {
    var count = 0;
    for (word in text.split(" ")) {
        if (word.length() > 0) {
            count = count + 1;
        }
    }
    return count;
}

func test_counts_words(): void {
    expectEq(countWords("the cat sat"), 3, "three words");
}

func test_ignores_extra_spaces(): void {
    expectEq(countWords("  the   cat  "), 2, "extra spaces");
}

func test_empty_text_has_no_words(): void {
    expect(countWords("") == 0, "empty text");
}

Listing 16-1: wordcount_test.lpscr, a function and three tests for it

from debug.assert import (expect, expectEq); brings in two assertion helpers. expect(condition, message) fails when the condition is false. expectEq(actual, expected, context) fails when the two values differ.
Each test_ function checks one behaviour. Its name is what the report shows, so make it say what should be true.
The file name ends in _test.lpscr. That is how the test runners recognise a test file.

Run the tests in that file:

loup-bootstrap test --file wordcount_test.lpscr

found 3 tests in 1 files (0 compile failures)
  ✓ ./wordcount_test.lpscr::test_counts_words (20ms)
  ✓ ./wordcount_test.lpscr::test_empty_text_has_no_words (19ms)
  ✓ ./wordcount_test.lpscr::test_ignores_extra_spaces (20ms)

Tests: 3 total, 3 passed, 0 failed, 0 skipped (20ms)

Each line names the file and the test, with ✓ for a pass, ✗ for a failure and - for a skipped test. Tests run in alphabetical order of their names, so a test must not depend on another test having run first. loup-bootstrap test exits with status 0 when every test passes and 1 when any test fails, so it can gate a CI job.

Which command? On macOS, Windows and Linux arm64, loup test and loup-bootstrap test are the same program. On Linux x86-64, loup test runs a second, self-hosted runner, described at the end of this chapter. The examples in this chapter use loup-bootstrap test, which behaves the same everywhere.

Mistake: a bare `assert`

If you have written tests in Python, you will want to write assert:

This code does not compile

@mode(host.scripted)

func countWords(text: string): int {
    return text.split(" ").length();
}

func test_counts_words(): void {
    assert(countWords("the cat sat") == 3, "three words");
}

main.lpscr:8:5: error[E0001]: undefined identifier `assert`

Loup has no built-in assert. The checks live in the debug.assert module, and a test file imports the ones it uses. expect is the general-purpose one:

@mode(host.scripted)

from debug.assert import (expect);

func countWords(text: string): int {
    return text.split(" ").length();
}

func test_counts_words(): void {
    expect(countWords("the cat sat") == 3, "three words");
}

(The self-hosted runner on Linux x86-64 supplies a bare assert(condition, message) to the files it runs, so a test that uses it passes there and fails to compile everywhere else. Import expect and the file works with every runner and with loup run.)

Mistake: leaving out the context

Every debug.assert helper takes a final description of what is being checked:

This code does not compile

@mode(host.scripted)

from debug.assert import (expectEq);

func test_sum(): void {
    expectEq(1 + 1, 2);
}

main.lpscr:6:5: error[E0001]: function 'expectEq' expects 3 argument(s), but 2 provided

The context is not optional, because it is the first thing the failure report prints. Write what the value means — expectEq(1 + 1, 2, "one plus one") — and a failure tells you which check went wrong without opening the file.

When a Test Fails

Text often contains tabs. Add a test for that to wordcount_test.lpscr:

This code does not do what it should

func test_tabs_separate_words(): void {
    expectEq(countWords("the\tcat"), 2, "tab-separated words");
}

Listing 16-2: A new test that exposes a bug

loup-bootstrap test --file wordcount_test.lpscr

found 4 tests in 1 files (0 compile failures)
  ✓ ./wordcount_test.lpscr::test_counts_words (22ms)
  ✓ ./wordcount_test.lpscr::test_empty_text_has_no_words (23ms)
  ✓ ./wordcount_test.lpscr::test_ignores_extra_spaces (22ms)
  ✗ ./wordcount_test.lpscr::test_tabs_separate_words (22ms)
    AssertionError: tab-separated words: expected 2, got 1

Tests: 4 total, 3 passed, 1 failed, 0 skipped (24ms)

The failure line is built from the arguments to expectEq: the context you wrote, then the expected value, then the actual one. countWords splits only on spaces, so "the\tcat" is one word. The test is right and the function is wrong, so fix the function:

## The number of words in `text`, separated by spaces or tabs.
func countWords(text: string): int {
    var count = 0;
    for (word in text.replace("\t", " ").split(" ")) {
        if (word.length() > 0) {
            count = count + 1;
        }
    }
    return count;
}

Listing 16-3: countWords treats a tab like a space

Run the tests again:

loup-bootstrap test --file wordcount_test.lpscr

found 4 tests in 1 files (0 compile failures)
  ✓ ./wordcount_test.lpscr::test_counts_words (24ms)
  ✓ ./wordcount_test.lpscr::test_empty_text_has_no_words (28ms)
  ✓ ./wordcount_test.lpscr::test_ignores_extra_spaces (24ms)
  ✓ ./wordcount_test.lpscr::test_tabs_separate_words (24ms)

Tests: 4 total, 4 passed, 0 failed, 0 skipped (28ms)

The `debug.assert` Helpers

debug.assert (Language Reference §21.18) has seven checks. Each throws AssertionError when it fails, with a message that names the values:

Helper	Fails when	Failure message
`expect(condition, message)`	`condition` is false	`message`
`expectEq(actual, expected, context)`	`actual != expected`	`context: expected E, got A`
`expectNeq(actual, forbidden, context)`	`actual == forbidden`	`context: value was equal to forbidden F`
`expectSome(option, context)`	the `Option` is `None`; otherwise returns the value	`context: expected Some, got None`
`expectContains(items, item, context)`	`item` is not in the list	`context: list does not contain I`
`expectThrows(action, context)`	calling `action` does not throw; otherwise returns the exception	`context: expected an exception, none thrown`
`expectNear(actual, expected, tolerance, context)`	the floats differ by more than `tolerance`	`context: expected E +/- T, got A`

expectThrows and expectSome hand back what they found, so a test can check it further:

@mode(host.scripted)

from debug.assert import (expectEq, expectSome, expectThrows, expectNear, expectContains);

## A TCP port number from text; throws ValueError outside 1..65535.
func parsePort(text: string): int {
    let n = text.toInt();
    if (n < 1 or n > 65535) {
        throw ValueError(i"port out of range: {$ text $}");
    }
    return n;
}

func test_port_out_of_range_throws(): void {
    let e = expectThrows(func(): void { parsePort("70000"); }, "parsePort(\"70000\")");
    expectEq(e.message, "port out of range: 70000", "error message");
}

func test_default_port_is_configured(): void {
    let settings: dict[string, int] = {"port": 8080};
    let port = expectSome(settings.get("port"), "port setting");
    expectEq(port, 8080, "port value");
}

func test_average_load(): void {
    let samples = [0.5, 0.25, 0.75];
    let average = (samples[0] + samples[1] + samples[2]) / 3.0;
    expectNear(average, 0.5, 0.0001, "average load");
}

func test_known_schemes(): void {
    expectContains(["http", "https"], "https", "supported schemes");
}

Listing 16-4: expectThrows, expectSome, expectNear and expectContains

The action passed to expectThrows is a function value with no parameters: func(): void { … } (Chapter 3).

Mistake: comparing two lists with `expectEq`

This code does not do what it should

@mode(host.scripted)

from debug.assert import (expectEq);

## The words of `text`, lowercased.
func words(text: string): list[string] {
    var out: list[string] = [];
    for (raw in text.split(" ")) {
        if (raw.length() > 0) {
            out.push(raw.toLowerCase());
        }
    }
    return out;
}

func test_words_are_lowercased(): void {
    expectEq(words("The CAT"), ["the", "cat"], "lowercased words");
}

loup-bootstrap test --file words_test.lpscr

found 1 tests in 1 files (0 compile failures)
  ✗ ./words_test.lpscr::test_words_are_lowercased (13ms)
    AssertionError: lowercased words: expected ["the", "cat"], got ["the", "cat"]

Tests: 1 total, 0 passed, 1 failed, 0 skipped (13ms)

The two lists print the same and the test still fails. The Language Reference says == on lists compares element by element (ruling EQ-3), but Loup 1.25 does not implement that yet: two separately built lists are never ==, so expectEq and expectNeq cannot compare them. Until it does, compare something that does compare by value — the joined text, or the length and each element:

func test_words_are_lowercased(): void {
    let found = words("The CAT");
    expectEq(found.length(), 2, "number of words");
    expectEq(found.join(" "), "the cat", "lowercased words");
}

Listing 16-5: Comparing a list through its length and its joined text

Numbers, strings, booleans, tuples, enums and Option values compare by value in expectEq. Lists, dicts and sets do not, and neither do structs once they are passed as Any (as expectEq's parameters are): compare a struct's fields one at a time.

Errors that are not assertions

A test also fails when the code under test throws something other than an AssertionError:

This code does not do what it should

@mode(host.scripted)

from debug.assert import (expectEq);

## The average length of `words`, rounded down.
func averageLength(words: list[string]): int {
    var total = 0;
    for (w in words) {
        total = total + w.length();
    }
    return total / words.length();
}

func test_average_of_no_words(): void {
    expectEq(averageLength([]), 0, "average of nothing");
}

loup-bootstrap test --file average_test.lpscr

found 1 tests in 1 files (0 compile failures)
  ✗ ./average_test.lpscr::test_average_of_no_words (12ms)
    AssertionError: integer division or modulo by zero

Tests: 1 total, 0 passed, 1 failed, 0 skipped (12ms)

The message is the IntegerDivisionError's, but the report labels it AssertionError. Loup 1.25's runner keeps a failed test's message and not its exception type (Testing Reference §23.6), so read the message, not the label, to tell a broken assumption from a crash. The fix belongs in averageLength: decide what the average of nothing is (here, if (words.length() == 0) { return 0; }) and keep the test.

Skipping a Test

Some tests need something the machine may not have: a large data file, a network service, a tool. testing.skipTest(reason) stops the test and records it as skipped rather than failed:

@mode(host.scripted)

from debug.assert import (expect);
from testing import (skipTest);

func test_big_corpus(): void {
    if (not exists("testdata/corpus.txt")) {
        skipTest("testdata/corpus.txt is not checked out");
    }
    expect(readText("testdata/corpus.txt").length() > 0, "corpus is not empty");
}

Listing 16-6: corpus_test.lpscr skips its test when the input file is missing

On a machine without testdata/corpus.txt:

loup-bootstrap test --file corpus_test.lpscr

found 1 tests in 1 files (0 compile failures)
  - ./corpus_test.lpscr::test_big_corpus (12ms)

Tests: 1 total, 0 passed, 0 failed, 1 skipped (12ms)

The pretty report does not print the reason; it is a note for the person reading the test. A skip does not make the run fail.

Organizing Tests

Put each test file next to the module it tests. Here is a project made with loup-bootstrap init wordstats, with wordcount_test.lpscr from earlier moved into it and a module of text helpers added:

wordstats/
├── wordstats.lppkg
└── src/
    ├── main.loup
    ├── textstats.lpscr
    ├── textstats_test.lpscr
    └── wordcount_test.lpscr

The module is an ordinary script module with pub functions:

@mode(host.scripted)

## The words of `text`, lowercased. Spaces and tabs separate words.
pub func words(text: string): list[string] {
    var out: list[string] = [];
    for (raw in text.replace("\t", " ").split(" ")) {
        if (raw.length() > 0) {
            out.push(raw.toLowerCase());
        }
    }
    return out;
}

## How many times each word occurs.
pub func frequencies(text: string): dict[string, int] {
    var counts: dict[string, int] = {};
    for (w in words(text)) {
        counts[w] = (counts.get(w) ?? 0) + 1;
    }
    return counts;
}

Listing 16-7: src/textstats.lpscr

The test file imports it by name, as any other script would:

Fragment — not a complete program

@mode(host.scripted)

from debug.assert import (expectEq);
from textstats import (words, frequencies);

func test_words_are_lowercased(): void {
    expectEq(words("The CAT").join(" "), "the cat", "lowercased words");
}

func test_frequencies_count_repeats(): void {
    let counts = frequencies("the cat and the hat");
    expectEq(counts["the"], 2, "count of 'the'");
    expectEq(counts["cat"], 1, "count of 'cat'");
}

Listing 16-8: src/textstats_test.lpscr (it imports Listing 16-7, so it runs as a pair)

In a project, loup-bootstrap test with no arguments finds every test file:

cd wordstats
loup-bootstrap test

found 6 tests in 2 files (0 compile failures)
  ✓ src/textstats_test.lpscr::test_frequencies_count_repeats (29ms)
  ✓ src/textstats_test.lpscr::test_words_are_lowercased (28ms)
  ✓ src/wordcount_test.lpscr::test_counts_words (33ms)
  ✓ src/wordcount_test.lpscr::test_empty_text_has_no_words (28ms)
  ✓ src/wordcount_test.lpscr::test_ignores_extra_spaces (29ms)
  ✓ src/wordcount_test.lpscr::test_tabs_separate_words (29ms)

Tests: 6 total, 6 passed, 0 failed, 0 skipped (33ms)

(The runner prints each file's full path; the listings in this chapter shorten it to the part inside the project.)

How test files are found

loup-bootstrap test reads the project manifest, <name>.lppkg, from the current directory or a parent. Its [test] table, as loup-bootstrap init writes it, is:

[test]
pattern = "**/*_test.loup"
parallel = true
timeout = 30

The runner treats as test files:

every file matching pattern — a pattern ending in .loup also matches the same name ending in .lpscr, so the default finds *_test.loup and *_test.lpscr anywhere in the project;
every *.test.loup and *.test.lpscr file;
every .loup and .lpscr file under the project's tests/ directory.

In each test file, a test is a top-level function named test_… with no parameters. A test_ function that takes a parameter is not a test and is silently ignored. Outside a project, use --file, as in the first examples.

Mistake: a test file the runner does not see

Python's convention is test_textstats.py. Name a Loup file that way and nothing complains:

mv src/textstats_test.lpscr src/test_textstats.lpscr
loup-bootstrap test

found 4 tests in 1 files (0 compile failures)
  ✓ src/wordcount_test.lpscr::test_counts_words (19ms)
  ✓ src/wordcount_test.lpscr::test_empty_text_has_no_words (19ms)
  ✓ src/wordcount_test.lpscr::test_ignores_extra_spaces (21ms)
  ✓ src/wordcount_test.lpscr::test_tabs_separate_words (20ms)

Tests: 4 total, 4 passed, 0 failed, 0 skipped (21ms)

test_textstats.lpscr matches no discovery rule, so its two tests silently disappear from the report and the run still succeeds. A project whose only test file is misnamed reports found 0 tests in 0 files and exits with status 0. If a new test never shows up in the report, check the file name first: the suffix is _test, as in textstats_test.lpscr.

Tests in a `tests/` directory

You can keep tests in tests/ instead, but they then need a relative import to reach src/: from ..src.textstats import (words);. loup-bootstrap test accepts that; the self-hosted runner described below does not run a file with a relative import and reports it as a failure. Test files beside their modules, importing them by plain name, work with both runners.

Choosing What to Run and How to Report It

loup-bootstrap test --filter tabs                # tests whose name contains "tabs"
loup-bootstrap test --file src/textstats_test.lpscr
loup-bootstrap test --format json                # one JSON document
loup-bootstrap test --format junit               # JUnit XML, for CI servers
loup-bootstrap test --no-color                   # plain text for logs
loup-bootstrap test --no-parallel --timeout 120  # sequential, 120 s per test

A filter selects tests by their qualified name (file::test_name):

loup-bootstrap test --filter tabs

found 6 tests in 2 files (0 compile failures)
  ✓ src/wordcount_test.lpscr::test_tabs_separate_words (11ms)

Tests: 1 total, 1 passed, 0 failed, 0 skipped (11ms)

The JSON report has one entry per test, with the failure's message. This is the tab test from Listing 16-2, before the fix, run with loup-bootstrap test --format json --file wordcount_test.lpscr --filter tabs (the runner prints its found … line first):

{
  "duration_ms": 12,
  "failed": 1,
  "passed": 0,
  "results": [
    {
      "duration_ms": 12,
      "error": {
        "message": "tab-separated words: expected 2, got 1",
        "trace": [],
        "type": "AssertionError"
      },
      "name": "./wordcount_test.lpscr::test_tabs_separate_words",
      "status": "fail"
    }
  ],
  "skipped": 0,
  "total": 1
}

loup-bootstrap test --help lists the rest, including --native, which builds host.native tests ahead of time, and --freestanding, which runs freestanding tests under QEMU on Linux.

The Self-Hosted Runner on Linux x86-64

On Linux x86-64, loup test hands the run to loup-test, the self-hosted runner (Testing Reference §23, §23.9). It is written in Loup and works differently:

loup test [DIR] looks for *_test.lpscr and *_test.loup files under each directory (default: the current one). It needs no manifest.
Each file runs in its own loup run --allow-all process, through a generated driver that calls every argument-less test_* function and reports PASS, FAIL, SKIP or STALE (a file that reported nothing) per file.
Test bodies may call a bare assert(condition, message); the driver supplies it as debug.assert.expect unless the file imports its own.
--strict turns a file that reports nothing, a run that finds no test files, and a skipped test into failures. Use it in CI: it catches the misnamed file from the previous section.
A loup-test.toml file at or above a test sets its timeout, max-mem, louppath and allow-skip.
Each test process gets two environment variables: LOUP_TEST_FILE (the test file) and LOUP_TEST_TMP (a fresh scratch directory that is removed afterwards).

Stage0's loup-bootstrap test has no --strict. A test file written as in this chapter — named *_test.lpscr, importing its debug.assert helpers, next to the module it tests — runs unchanged under both runners.

What Is Not There Yet

⚠ Planned. Test functions declared async func or gen func are found but not driven: the runner calls them as plain functions, so an async test's Task is never awaited (Testing Reference §23.6). Keep test functions synchronous.

⚠ Planned. The Testing Reference (a Draft) describes a larger framework — test roots marked by __testroot__.loup, dependency injection with resource factories, markers and include/exclude expressions, test sequencing and constraints — in its §1–§12, and reserves section numbers for mocking, property-based testing and tandem validators. None of it ships in Loup 1.25. The testing module provides runTest, skipTest, TestResult, TestSummary, TestFailure and SkipSignal; debug.assert provides the checks.

Until then, the patterns are plain code. To replace a dependency in a test, pass it in: a function that takes a fn(string): string for "fetch this URL" can be given a lambda that returns canned text. For setup and cleanup, write a helper that builds the fixture and call it at the start of each test, and put cleanup that must run even when the test fails in a finally block (defer is accepted but does not run in Loup 1.25; see Chapter 4).

Summary

A test is a top-level func test_…(): void with no parameters, in a file whose name ends in _test.lpscr (or _test.loup).
Tests fail by throwing. Import the checks from debug.assert: expect, expectEq, expectNeq, expectSome, expectContains, expectThrows, expectNear. Each takes a context string that the failure message starts with.
loup-bootstrap test runs a project's tests (or one file with --file), prints ✓/✗/- per test, and exits 1 if any test failed.
expectEq cannot compare two lists in Loup 1.25 (== on lists is not element-wise yet); compare their length and joined text.
A non-assertion exception fails the test too, but is labelled AssertionError in the report; read the message.
testing.skipTest(reason) skips a test that cannot run here.
A misnamed test file is silently ignored. Keep tests beside their modules as *_test.lpscr, importing them by name, and they run under both runners; on Linux x86-64, loup test --strict also fails a run that finds no tests.

Try It Yourself

Write slug_test.lpscr with a function slug(title: string): string that lowercases a title and replaces spaces with -, and three tests for it: a plain title, a title with several spaces in a row, and an empty title.
Add a test that slug keeps digits. Make it fail first, read the report, then make it pass.
Write parseConfig(text: string): dict[string, string] for key=value lines and use expectThrows to test that a line without = throws a ValueError.
Rename slug_test.lpscr to test_slug.lpscr inside a project and watch the tests disappear from the report.

Next: Chapter 17 is a design preview of JavaScript interop; Chapter 18 builds a complete command-line tool.

Chapter 16: Testing

A First Test

Mistake: a bare assert

Mistake: leaving out the context

When a Test Fails

The debug.assert Helpers

Mistake: comparing two lists with expectEq

Errors that are not assertions

Skipping a Test

Organizing Tests

How test files are found

Mistake: a test file the runner does not see

Tests in a tests/ directory

Choosing What to Run and How to Report It

The Self-Hosted Runner on Linux x86-64

What Is Not There Yet

Summary

Try It Yourself

Mistake: a bare `assert`

The `debug.assert` Helpers

Mistake: comparing two lists with `expectEq`

Tests in a `tests/` directory