Chapter 5 — Streaming JSOS Results

A test framework's results pipeline is the part that survives when everything else doesn't. A run that crashes after three hours is useless if the framework was buffering its output for the final report. This chapter is about why testbase streams, what it streams, and how a consumer reads it.

codecraftai-testbase writes test results in a streaming format called JSOS — JSON Separated Object Stream. The format is straightforward: each event is a self-contained JSON object, prefixed by an ASCII record-separator character (Unicode code point U+001E) and a newline. A consumer reads the file by scanning for the separator, capturing the JSON between separators, and parsing.

The format exists because the alternative — buffering events until the run ends and writing one consolidated JSON document — loses everything when a run crashes. Streaming means a partial run is still a useful run.

This chapter walks the format, the recorders that produce it, the result files testbase emits, and the consumer patterns that read the stream. A note up front on division of labour: plain codecraftai-testbase defaults to a stdout recorder and runs standalone — it writes the JSOS/JSON artefacts only when a run is given an output directory, and it never renders the HTML summary. The full HTML-rendering recorder and the presence-folder results-directory layout are the responsibility of IntegratedTestJob in codecraftai-integration-runtime. The format is identical in both cases; this chapter is about the format and the standalone recorder, and flags where the integration tier takes over.

What "streaming" means in this context

A streaming results file is one where:

1. Each record is appended to the file as soon as the event happens.
2. The file is parseable at any moment, including immediately after a crash.
3. A consumer can tail -f the file and process records in near-real-time.

JSOS satisfies all three. Each record is a complete JSON object; appending one record does not invalidate any previously written records; a tail consumer can see a record as soon as the recorder writes it (the recorder flushes after each event — see the recorder lifecycle below).

Compared to alternatives:

• JSON Lines (newline-delimited JSON) is the closest cousin. JSOS uses the ASCII record-separator character (\x1e) instead of \n because some result fields legitimately contain newlines (multi-line stack traces, multi-line log excerpts). A newline-delimited format requires escaping every newline in the payload; the record-separator character doesn't appear in normal text, so no escaping is needed.
• JUnit XML is the format CI dashboards historically read. It is not streamable: the document is closed at the end of the run. Testbase produces a JSON-shaped consolidated file for batch consumers, but JSOS is the live stream.
• Protobuf streams would be more compact but harder to inspect by hand. The portfolio's debuggability principle from Volume 0 chapter 1 prefers human-readable text.

The record separator

The single line that defines the format is in the recorder import block:

# codecraftai-testbase/source/packages/ccai/testbase/results/recorders/jsonresultrecorder.py:24

from ccai.xmodules.jsos import CHAR_RECORD_SEPERATOR

CHAR_RECORD_SEPERATOR is the two-byte sequence "\x1E\n" — the ASCII record-separator character followed by a newline. (The newline is a readability convenience: it keeps the raw file greppable line-by-line in a pager. The structural delimiter is the \x1E; the newline is not load-bearing.) Every JSOS file is a sequence of (separator, json_object) pairs. The first bytes of the file are a separator; from there each separator marks the start of the next record.

A reader does not need testbase to parse the file. Any program that can split on \x1e and parse JSON can consume the stream:

# A minimal reader.

def read_jsos(path: Path) -> Iterator[dict]:
    """
        Read a JSOS file and yield each record as a dict.
    """
    with path.open("rb") as handle:
        raw = handle.read()
    parts = raw.split(b"\x1e")
    for part in parts:
        text = part.strip()
        if len(text) == 0:
            continue
        record = json.loads(text)
        yield record
    return

The chapter does not claim this is the consumer the framework ships — Chapter 6 walks the agents CLI's parser (jsosparser.py). But the format is open: any reader that can find \x1e and parse JSON between occurrences is a conforming consumer.

The three result files

When a run is given an output directory, the recorder produces three files in the run's results directory. The filenames are chosen by TestJob.begin() (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:148–150) and the serialisation is performed by JsonResultRecorder (codecraftai-testbase/source/packages/ccai/testbase/results/recorders/jsonresultrecorder.py:90), the JSOS/JSON serialisation core both the standalone and integration recorders inherit from:

testrun_summary.json

Written at finalize. The high-level summary of the run:

{
  "run_id": "20260430-153012-abc123",
  "started_at": "2026-04-30T15:30:12.123Z",
  "finished_at": "2026-04-30T16:19:43.456Z",
  "duration_seconds": 2971,
  "total": 3712,
  "passed": 3680,
  "failed": 18,
  "errored": 4,
  "skipped": 10,
  "host": {
    "hostname": "ci-runner-04",
    "os": "Linux 6.5.0",
    "python": "3.12.5"
  }
}

This is the artefact a CI dashboard polls for status. It does not include per-test detail; for that, the consumer reads the stream or the consolidated file.

testrun_results_stream.jsos

Written as tests finish. Each record is one of a small set of event kinds:

• test_started
• test_finished (with outcome ∈ {passed, failed, errored, skipped})
• property_iteration (one per iteration of a @given test)
• phase_started (one per sequencer phase)
• phase_finished
• diagnostic_capture (host details, package versions, etc.)
• coupling_event (integration coupling lifecycle)

A representative test_finished record:

{
  "kind": "test_finished",
  "test_name": "ccai.agentkit.tests.test_costtracker.test_addition_is_commutative",
  "started_at": "2026-04-30T15:30:14.001Z",
  "finished_at": "2026-04-30T15:30:14.018Z",
  "duration_ms": 17,
  "outcome": "passed",
  "monikers": ["unit"],
  "pivots": null
}

A representative test_finished for a failure:

{
  "kind": "test_finished",
  "test_name": "ccai.agentkit.tests.test_invoice.test_invoice_total_includes_tax",
  "started_at": "2026-04-30T15:35:21.501Z",
  "finished_at": "2026-04-30T15:35:21.612Z",
  "duration_ms": 111,
  "outcome": "failed",
  "monikers": ["billing"],
  "exception": {
    "type": "AssertionError",
    "message": "expected Decimal('108.75'), got Decimal('100.00')",
    "traceback": [
      "  File \".../test_invoice.py\", line 14, in test_invoice_total_includes_tax",
      "    assert rtnval"
    ]
  }
}

The stream is the rich source of truth. Anything you can ask about a test run can be answered by parsing the stream.

testrun_results.json

Written at finalize. The consolidated final document. Same information as the stream, reorganised into a tree by package → module → test for batch consumers (post-run analysis tools, human readers). Producing both is cheap because the recorder already has every record in memory by finalize time.

The consolidated file is the one a build engineer commits to a results-archive bucket; the stream file is what they keep locally for live monitoring.

The recorder lifecycle

TestJob.begin() (Chapter 2) resolves the results directory and the artefact filenames (codecraftai-testbase/source/packages/ccai/testbase/testjob.py:148–150); the recorder itself is constructed later, in TestJob.execute(), through the _create_recorder(...) provider hook (.../testbase/testjob.py:564) and entered as a context manager (.../testbase/testjob.py:256–257):

# .../testbase/testjob.py — recorder construction in execute()

with self._create_recorder(runid=runid, start=start, render_info=render_info,
                           build_info=build_info, pipeline_info=pipeline_info,
                           job_info=job_info) as recorder:
    ...
    tseq.execute_tests(runid, recorder)

The default recorder is StdoutResultRecorder, not JsonResultRecorder. Plain testbase runs standalone — no codecraftai-this-app, no RuntimeVariables, no HTML summary template — so TestJob._create_recorder() returns a StdoutResultRecorder (.../testbase/results/recorders/stdoutresultrecorder.py:36). That recorder subclasses JsonResultRecorder to reuse the JSOS-stream and JSON-summary serialisation, but it makes initialize_report() a no-op (skipping the HTML template and static-resource publication) and, on finalize(), prints the human-readable run summary to stdout. The integration tier overrides this hook to use the full JsonResultRecorder instead — see "Where the integration tier takes over" below.

Recorder setup, on context entry, does the following:

1. Initialises the output environment (for the stdout recorder, report initialisation is a no-op; the JSOS/JSON files are still written when an output directory is configured).
2. Opens testrun_results_stream.jsos for writing.
3. Writes the run-level preview / initial summary.

Every test result the sequencer produces is then forwarded to the recorder, which (.../jsonresultrecorder.py:268–341):

1. Serialises the ResultNode to JSON via to_json(...).
2. Writes the record-separator sequence (\x1E\n).
3. Writes the JSON.
4. Calls flush() on the underlying stream handle.
5. Updates the pass/fail/error/skip counters under a lock.

The flush is what makes the stream tail-able. Without it Python's I/O buffering would batch writes and a tail consumer would see records in lumps. The flush has a measurable cost on fast suites — but on automation suites the per-test cost is dwarfed by the test itself, and the flush is worth its weight.

On context exit / finalize():

1. The recorder finalises counts and result determination.
2. It builds the consolidated JSON tree from its in-memory result log and writes testrun_results.json.
3. It builds the summary aggregates and writes testrun_summary.json.
4. The StdoutResultRecorder additionally prints the summary to stdout (exactly once — finalize guards against the double-invocation that the context manager's __exit__ would otherwise cause).

If the run crashes between setup and finalize(), the stream file is partial but valid: every record between the opening separator and the last full record is parseable. A consumer reading the partial stream can compute its own summary from what's there.

Where the integration tier takes over

Everything above is the standalone recorder. Three things that this chapter used to attribute to testbase are now the responsibility of IntegratedTestJob in codecraftai-integration-runtime (codecraftai-integration-runtime/source/packages/ccai/integrationruntime/integratedtestjob.py:51). IntegratedTestJob extends TestJob and overrides exactly three provider hooks:

1. The full HTML-rendering recorder. IntegratedTestJob._create_recorder() (.../integratedtestjob.py:79) returns a JsonResultRecorder in place of the StdoutResultRecorder — the JSOS stream plus the consolidated/summary JSON and the rendered HTML summary (the template copy and static-resource publication that StdoutResultRecorder.initialize_report() no-ops out).

2. The presence-folder results-directory layout. IntegratedTestJob._create_output_context() returns a RuntimeOutputContext (.../integrationruntime/runtimeoutputcontext.py:40), which delegates to ccai.thisapp.paths so that results, diagnostics, and per-testcase by-products land under the activated application's well-known presence folder. Those path helpers are activation-gated — they raise SemanticError unless the process was activated as a job — which is exactly why plain testbase (which never activates) cannot use them and roots its artefacts under AM_OUTPUT_DIRECTORY or <cwd>/.output via the standalone SimpleOutputContext instead.

3. Real CI/build metadata. IntegratedTestJob._create_run_metadata() returns a RuntimeRunMetadata (.../integrationruntime/runtimerunmetadata.py) sourced from the resolved RuntimeVariables, where the standalone StandaloneRunMetadata supplies an auto-generated run id and (unknown) placeholders for build/pipeline/job. This is the metadata that populates the build, pipeline, and job blocks of testrun_summary.json.

The intended inheritance chain is TestJob -> IntegratedTestJob -> InteropLibraryTestJob -> DistributedTestJob. As of this writing IntegratedTestJob is built and wired, but the interop tier has not yet been re-parented onto it — in the interop source, InteropLibraryTestJob still extends TestJob directly. The JSOS format is unaffected either way; the only thing in flight is which class supplies the recorder/context/metadata.

The point for this chapter: the format is identical across both tiers. Whether your records come from a standalone stdout run or an activated integration run, a \x1E\n-delimited stream of JSON objects is what lands on disk, and the same testbase agents consumers read it.

Streaming to external services

For runs that need to report to an external dashboard while running, testbase ships a streaming encoder and result streamer alongside the recorder (codecraftai-testbase/source/packages/ccai/testbase/results/streaming/resultstreamer.py:44–78).

The classes:

• StreamingJsonEncoder (line 44) — a json.JSONEncoder subclass that handles datetime, Decimal, and other types the standard encoder doesn't. Used by the recorder to serialise events.
• ResultStreamer (line 64) — a worker that runs in a background thread, reads events from a queue, and POSTs them to an external URL. Used when the test job is configured with a streaming_url.

The streamer is opt-in: when the job's constructor receives a streaming_url, the recorder also pushes events into the streamer's queue. The streamer does best-effort delivery — failures are logged but don't fail the run — because the local JSOS stream is the source of truth and the remote stream is a convenience.

This is the Aspects pattern from Volume 0 chapter 4 once more: the recorder's behaviour (write locally vs. write locally and push remotely) is parameterised, not decorated. A test author writes the same test; the build engineer picks the recorder's behaviour.

Consumer patterns

Three consumer shapes show up across the portfolio:

Live monitoring

A developer running a long suite watches the stream:

tail -F results/testrun_results_stream.jsos | testbase agents testresult list --stream

The agents CLI's testresult list --stream command (Chapter 6) parses the JSOS as it arrives and prints a one-line summary per test. A failing test is colour-coded. When the run ends the stream tail closes cleanly.

Failure extraction

After a run, an engineer wants the failures only:

testbase agents testresult failures --results results/

The failures subcommand reads the stream file, filters to outcome != passed, and emits a structured JSON document an engineer can grep through or feed to an agent. The codecraftai-testbase/source/packages/ccai/testbase/cli/cmdtree/agents/testresult/failures.py file is the implementation.

Dashboard ingestion

A CI dashboard polls the summary file every minute, then on completion ingests the consolidated JSON for per-test detail. The dashboard does not normally consume the JSOS stream — it is the live tool, not the historical record.

For dashboards that do want live updates, the ResultStreamer's remote-streaming mode pushes events to a URL the dashboard owns. The dashboard receives JSON-encoded events in real time without having to poll a file.

Replayability

A point worth making explicit: the stream is the source of truth. The summary and consolidated files are both derivable from the stream. If a build engineer keeps only the stream, they can recompute the other two at any time.

This matters for compliance and audit cases where every test event must be archived but storage is at a premium. The stream is the smallest representation that loses nothing. The consolidated file is a convenience.

It also matters for replay: a run's stream can be fed into a replayer that reproduces the run's timing on a different host, for human or agent viewing. Volume 8 (Time-Travel Debugging in Production) builds on the same pattern at a different layer of the stack — the principles are the same.

What stays out of the stream

Some things deliberately do not become JSOS records:

• Per-test stdout / stderr. These are written to a per-testcase by-products directory resolved through the output context (get_testcase_byproducts_dir(test_id) — rooted under .output for standalone runs, or under the presence folder for integration runs). The stream record carries a reference to the directory but not the contents — log volume can be large, and dragging it into the stream would balloon record sizes.
• Per-test screenshots / artefacts. Same — a reference, not the bytes.
• Coverage data. Coverage is written as a separate file by the coverage plugin, not as a JSOS record.

The principle: the stream carries the structured data of the run; the directory carries the blob data. A consumer that needs the blobs walks the directory; a consumer that needs the structure parses the stream.

The agents-readable contract

A note of forward reference: the exact same JSOS records the recorder produces are what the agents CLI subcommands (Chapter 6) parse. There is no separate "agent format" — the contract between the framework and any consumer (human or AI) is the JSOS stream. This is why Chapter 6 treats the agents CLI as a thin parser-and-formatter on top of the stream rather than a separate subsystem.

What's next

• Chapter 6 — The testbase agents CLI. The thin layer that parses the JSOS stream and presents it to humans and AI agents alike.
• Chapter 7 — Mocking. Tests that don't touch real systems still produce JSOS records — the recorder treats mocks identically to real resources.