# Code Craft AI books > The Code Craft AI books published at https://code-craft.ai/references/codecraftai. # The Code Craft AI Books A series of nine volumes and an appendix on building durable automation software in Python, drawn from the `codecraftai-*` packages. The series teaches a discipline (debuggability, explicitness, contracts that cut across packages) that works whether or not you keep using these packages, and every claim in it is anchored to a file and line in the public source. The text is kept in the public [codecraftai-books repository](https://github.com/code-craft-ai-company/codecraftai-books) and published at . The two are the same text. ## The volumes The recommended reading order is Volume 0 through Volume 8, then the appendix. Each title opens that volume's contents page. | Vol | Title | Subtitle | Chapters | |----:|-------|----------|--------:| | **0** | [**Foundations of Automation Engineering with Python**](volumes/00-foundations/README.md) | The CodeCraftX Discipline and the Foundation Tier | 10 | | **1** | [**Building a Test Framework**](volumes/01-test-framework/README.md) | testbase, arcli, and Single-Host Test Infrastructure | 10 | | **2** | [**Distributed Automation**](volumes/02-distributed-automation/README.md) | The TestJob Inheritance Chain and Multi-Host Test Execution | 16 | | **3** | [**Integration Automation and CI Orchestration**](volumes/03-integration-and-orchestration/README.md) | Typed Configurations as the Contract Between Orchestrations and Jobs | 28 | | **4** | [**Cross-Platform System Management**](volumes/04-cross-platform-sysman/README.md) | 52 Cross-OS Contracts and 5 Reference Implementations | 10 | | **5** | [**Interop and Protocols**](volumes/05-interop-and-protocols/README.md) | The Aspects Pattern, the Wire, and the Extension Model | 10 | | **6** | [**Topology, Provisioning, and UI Automation**](volumes/06-topology-provisioning-ui/README.md) | Modelling the Environment Under Automation | 15 | | **7** | [**Building an AI Agent SDK**](volumes/07-ai-agent-sdk/README.md) | Lifecycle, Tools, Memory, and Observability for Production Agents | 9 | | **8** | [**Time-Travel Debugging in Production**](volumes/08-timetravel-debugging/README.md) | Pod Instrumentation, Server Ingestion, and the Replay-Driven Debugging Workflow | 8 | | App. | Operating a Code Craft AI Subscription Platform | From Git-SHA Pinning to a Published-Package Ecosystem | planned | ## Who should read which volume | Profile | Volumes most relevant | |---|---| | Junior Python engineer entering automation work | 0, 1, 3 | | SDET / test-infra engineer | 0, 1, 2, 3 | | Platform / DevOps engineer building CI pipelines | 0, 3, 4, 6 | | SRE running cross-OS automation at scale | 0, 3, 4, 6 | | Integration engineer building protocol implementations | 0, 5 | | AI / agent developer shipping production agents | 0, 5, 7 | | Senior engineer / architect needing observability for long-lived services | 0, 8 | | Internal-tool maintainer running a package distribution | Appendix | ## How the books are written - The text is Markdown. Code is in fenced blocks with a language tag. - Every claim is anchored to a concrete file and line in one of the `codecraftai-*` repositories. Line numbers move as the code does, so treat them as a pointer to where to look. - The series is narrated in the voice of CodeCraftX, the engineering persona used across the portfolio. - Each volume lives in `volumes//`: a `README.md` that is its contents page, and one file per chapter under `chapters/`. ## Status Volumes 0 to 8 are drafted in full. The appendix is planned. These are working drafts, and a draft has errors: where a chapter is wrong, unclear or out of date with the code, we want to know. ## Errata, questions and suggestions Open an issue in the [codecraftai-books repository](https://github.com/code-craft-ai-company/codecraftai-books/issues), and say which chapter and passage you mean. Issues are the only contribution we accept; see [CONTRIBUTING.md](https://github.com/code-craft-ai-company/codecraftai-books/blob/main/CONTRIBUTING.md). ## Copyright Copyright © 2026 Code Craft AI LLC. All rights reserved. The software the books describe is licensed separately, in each `codecraftai-*` repository. # Volume 0 — Foundations of Automation Engineering with Python *The CodeCraftX Discipline and the Foundation Tier* ## About this volume Volume 0 is **prerequisite reading** for the rest of the series. Every later volume references the CodeCraftX persona, the error vocabulary, the Aspects pattern, the explicit-`return` convention, the configuration model, and the application-identity machinery introduced here. Without Volume 0, every later volume would have to re-explain the foundations. ## Who it is for Any Python engineer building a serious automation platform — SDETs, platform engineers, SREs, integration engineers. This is the book a senior engineer hands a new hire on day one. ## Source material The volume draws from the foundation tier of the `codecraftai-*` portfolio: - `codecraftai-foundations` — one distribution holding the contract layer (`ccai.interfaces`), the exception vocabulary and enhanced tracebacks (`ccai.errors`), the waiting primitives (`ccai.waiting`), the minimal `Aspects` carrier (`ccai.aspects`) and base classes such as `Freezable` (`ccai.bases`). Until 2026-09-02 these were the separate `codecraftai-interfaces`, `codecraftai-errors` and `codecraftai-waiting` packages; import paths did not change, and the chapters below still describe them by their subpackage names. - `codecraftai-this-app` — application identity + persona variables - `codecraftai-extension` — factory + plug-in machinery - `codecraftai-runtime` — runtime variables + well-known paths - `codecraftai-configuration` — config loading + multi-source - `codecraftai-credentials` — typed credential vocabulary - `codecraftai-dataprofiles` — connection profile classes - `codecraftai-storage-database` — DB scaffolding - `codecraftai-xmodules` — xformatting, ximport, xcollections, etc. ## Contents - [Chapter 1 — The CodeCraftX Philosophy](chapters/01-the-codecraftai-x-philosophy.md) - [Chapter 2 — Coding Standards as a Tool, Not a Style Guide](chapters/02-coding-standards-as-a-tool.md) - [Chapter 3 — The Error Vocabulary](chapters/03-the-error-vocabulary.md) - [Chapter 4 — Interfaces and Contracts](chapters/04-interfaces-and-contracts.md) - [Chapter 5 — Waiting with Rich Diagnostics](chapters/05-waiting-with-rich-diagnostics.md) - [Chapter 6 — Application Identity and Presence](chapters/06-application-identity-and-presence.md) - [Chapter 7 — Extension and Factory Machinery](chapters/07-extension-and-factory-machinery.md) - [Chapter 8 — Runtime Variables and Well-Known Paths](chapters/08-runtime-variables-and-well-known-paths.md) - [Chapter 9 — Configuration Loading](chapters/09-configuration-loading.md) - [Chapter 10 — Credentials, Data Profiles, Storage Scaffolding](chapters/10-credentials-data-profiles-storage.md) --- Part of [The Code Craft AI Books](../../README.md). # Chapter 1 — The CodeCraftX Philosophy > *Software for automation is software that runs unattended, in > environments you can't directly observe, against systems whose > failures you can't always reproduce. The discipline that makes > automation software durable is not the same as the discipline > that makes application software ship. This chapter is about > what's different.* This is the first chapter of the first volume in the series. Everything that follows — across this volume and the eight volumes after it — rests on the philosophy established here. The philosophy is concrete, not abstract: a small set of engineering principles that have specific consequences in every line of code the `codecraftai-*` portfolio ships. The principles are summarised in a single block of text that opens every repository's `AGENTS.md` file across the portfolio: ``` # Core Philosophy - Readability is important - Debuggability is paramount - Scalability and maintainability are critical - Flat is better than nested - Sparse is better than dense - Explicit is better than implicit - Simple is better than complex - Stability is more important than performance - Errors should never pass silently unless purposefully handled and silenced - If the implementation is hard to explain, follow, or maintain, try another way ``` Ten lines. Each one is the result of a tradeoff someone made deliberately, with consequences across thousands of lines of code. This chapter unpacks what each line *means in code*, so that the rest of the series can use them as shorthand. ## Who is CodeCraftX? Throughout this series, the narrator is **CodeCraftX** — a Principal Software Engineer and Architect at Code Craft AI LLC. Every repository in the portfolio's `AGENTS.md` opens with a description of this persona: > *CodeCraftX is a very creative, but very disciplined software > engineer that values code stability, reusability and scalability > over quick fixes and fast delivery. CodeCraftX always keeps > in mind the company's goal of building a comprehensive suite of > foundation Python packages that are well designed, cohesive, and > easy to use and that provide rapid but stable results.* The persona is not a marketing voice. It is a *constraint satisfier* — a fictional engineer who consistently applies the philosophy above and reaches the same conclusions every time. When two engineers on the same team would have disagreed about a design choice, CodeCraftX is the imagined third engineer who applies the philosophy, lands the choice, and moves on. The persona shows up in the portfolio's `AGENTS.md` files because the philosophy needs a *consistent* voice to apply it; the same discipline ten people apply produces ten subtly different codebases unless an authority figure resolves the disagreements. The series's prose is in the voice of CodeCraftX as well, for the same reason. An automation platform built by ten teams applying their own preferences produces a platform that's hard to learn. An automation platform built by ten teams applying *the same* philosophy produces a platform whose conventions transfer. The persona is the carrier of the conventions. ## Debuggability is paramount Of the ten principles, this is the one with the most concrete consequences. *Debuggability is paramount* means: when the code is failing in production, on a machine you can't SSH into, on a day when you have an hour to find the bug — every other consideration loses to the consideration of getting useful information out of the failure. Debuggability has *cost*. Every choice the philosophy makes for debuggability is a choice that's slower, more verbose, less "clever" than the alternative. A short example. The non-debuggable form: ```python def compute_total(items): return sum(item.price * (1 - item.discount_rate) for item in items if item.in_stock) ``` The debuggable form: ```python def compute_total(items): in_stock_items = [item for item in items if item.in_stock] item_totals = [] for item in in_stock_items: discounted_price = item.price * (1 - item.discount_rate) item_totals.append(discounted_price) total = sum(item_totals) return total ``` Three times as many lines. Three intermediate variables. An explicit `return` even though the value could be returned inline. All of this is *cost*. The benefit shows up at debug time. When the production failure is "the total is wrong for customer A," the debugger or the log can show the values of `in_stock_items`, of `item_totals`, of the final `total` — each at a known location in the function. The non-debuggable form requires re-running with print statements inserted, or carefully stepping through a generator expression in a way most debuggers handle poorly. The cost of the debuggable form is paid *once* per function written. The benefit is paid every time the function is debugged. Across a codebase that runs unattended for months and is debugged when something breaks, the math always favours debuggability. This is the philosophy's most important principle, and it shapes the others. When two principles seem to conflict, debuggability wins. ## Explicit is better than implicit Python's *Zen* names this; the philosophy turns it into specific rules. Three of those rules show up across the portfolio. ### No mutable defaults ```python # Forbidden def add_item(item, items=[]): items.append(item) return items ``` The `items=[]` default is a *single list shared across every invocation* — the canonical Python footgun. Every codebase in the foundation tier checks for this; the CodeCraftX persona catches it during code review every time. The right form: ```python def add_item(item, items=None): if items is None: items = [] items.append(item) return items ``` More verbose. Explicit about the default. No shared mutable state across calls. The discipline appears in `CLAUDE.md` under the "Default Values" heading and is enforced in code review. ### Named parameter passing ```python # Discouraged result = compute_total(catalog, region, currency) ``` Three positional arguments; the reader has to remember the order. The discipline: ```python # Preferred result = compute_total(catalog=catalog, region=region, currency=currency) ``` Every parameter named at the call site. When the function adds a fourth parameter — say, `tax_jurisdiction` — the existing call sites are unambiguous about which arguments they're passing. When parameters are reordered, no call site is affected. The cost: a few characters per call. The benefit: refactor safety and readability. This is the sort of small, recurring trade-off the philosophy is consistent about. ### Explicit `return` statements Volume 3's chapter on the CodeCraftX coding-standards discussion (chapter 2 of this volume) develops this in detail. The short version: every function ends in an explicit `return`, even when the function's "natural" return is `None`. ```python def configure_logger(logger): logger.setLevel(logging.INFO) formatter = logging.Formatter("%(asctime)s %(levelname)s %(message)s") handler = logging.StreamHandler() handler.setFormatter(formatter) logger.addHandler(handler) return # <-- explicit, even when None ``` The trailing `return` exists because the Python interpreter *generates* a return-bytecode whether or not the source has the keyword. When the keyword is present, a debugger can break at that line and inspect every local variable in the function before exit. When the keyword is absent, the breakpoint has to attach to the closing brace's invisible bytecode location, which most debuggers handle awkwardly. There are three further reasons covered in chapter 2: refactor safety (orphaned code becomes unreachable rather than blending into the next function), function-end visibility (the closing `return` is a clear delimiter), and consistency (every function ends the same way). None of these is the *biggest* reason on its own; together they justify the verbosity. ## Errors should never pass silently unless purposefully handled and silenced The exception-handling discipline in the foundation tier has six specific rules, all derived from this principle. Chapter 3 of this volume develops them in full. The summary version: 1. **Do not handle exceptions unless you can recover and continue the intended purpose.** A bare `except:` that swallows the exception and returns is a bug. 2. **Catch and reclassify** to add specific contextual information when you have it. 3. **Catch and reclassify** to improve code context in failure reports. 4. **Catch and add extra contextual information** about unique circumstances. 5. **`SemanticError` must NEVER be caught** — always let it bubble. This is the API-misuse signal; catching it would mask the misuse. 6. **In all other cases, let exceptions bubble.** The third special exception type — `ConfigurationError` — has similar must-bubble semantics in most contexts; chapter 3 catalogues all three. The principle is *not* "exceptions are good." The principle is *errors that fail loudly are easier to debug than errors that fail silently*. When the code can't recover from an exception, silently absorbing it just makes the failure harder to find later. When the code *can* recover, it's the code's responsibility to do so explicitly — and to log what it recovered from. ## Stability is more important than performance Most automation code is not in a hot loop. Most automation code is glue: it runs a step, waits for the step to finish, processes the result, runs the next step. The CPU is bored; the network or the subprocess is the bottleneck. When automation code is the hot loop — when a property test runs a thousand cases, when a packet inspector evaluates millions of rules, when a deserialiser parses a multi-gigabyte JSOS stream — *then* performance matters. But for most code in the portfolio, performance is *not* the constraint. Stability is. Stability has specific concrete consequences: ### No clever one-liners A list comprehension that produces three values in three nested generators saves a few lines of code. The cost shows up the first time someone has to debug why a particular item didn't make it through the chain. Expanding to intermediate variables costs nothing at runtime; it costs a small amount of readability *gain* over the lifetime of the codebase. ### No micro-optimisation that costs explicitness Premature optimisation is the canonical example. In the portfolio, the principle goes further: *even when the optimisation is correct*, if it costs explicitness, the philosophy chooses explicitness. A `dict[key]` that could be a `getattr(obj, key)` to save a hashtable lookup is not the right trade. ### Stability across versions When a function's signature changes between releases, every caller has to update. When the change is a *parameter rename* or a *parameter reorder*, the named-parameter-passing discipline above turns the change into a one-line update at each call site plus a clear deprecation warning. When the change is the function's *return type*, the explicit return-and-assign discipline (return values to local variables before returning) makes the change visible in code review. The portfolio's git history shows this discipline at work. When `codecraftai-errors` renamed `NotOverloadedError` to `AbstractMethodError`, the impact across the portfolio was mechanical: every import was updated, every raise site was updated, no semantic change in any caller. The named-import discipline kept the change small. ## If the implementation is hard to explain, follow, or maintain, try another way The last principle is the *meta-principle*: when applying the others doesn't produce code that's clear, the answer is to back up and try a different design rather than to push through. This is the principle that catches *clever architecture*. A design with five layers of indirection where two would do. A factory that produces factories that produce factory-protocol adapters. An API that's "elegant" by some abstract metric but that nobody can use without reading the source. Each of these is the symptom of an engineering team falling in love with a pattern at the expense of its users. The philosophy catches the pattern in code review, *before* it ships, by asking the simple question: *can a new engineer read this and understand what it does in five minutes?* If the answer is no, the design is wrong. Not partially wrong; not "elegant but hard to learn"; not "powerful with a steep curve" — *wrong*, in the sense of failing to serve the people who will maintain it. The right response is to back up and try again. ## What this philosophy is *not* Three things the philosophy explicitly does *not* say. ### Not "Python's PEP 8 verbatim" PEP 8 is a style guide. The philosophy is a *design discipline*. PEP 8's rules about line length, naming, and whitespace are enforced through linting; the discipline above is enforced through code review. The two layers complement each other; neither replaces the other. The portfolio uses standard Python tooling for PEP 8 — no special formatter, no custom linter for whitespace. The discipline above is what's *additional*. ### Not "the Zen of Python verbatim" The Zen of Python is a poem. Several of its lines appear (in spirit) in the philosophy above — *flat is better than nested*, *sparse is better than dense*, *explicit is better than implicit*. But the philosophy adds rules the Zen doesn't have (debuggability paramount, errors-never-silent, stability-over-performance) and omits Zen lines that don't apply to automation code (*beautiful is better than ugly* — automation code is rarely beautiful, and trying to make it so is usually a mistake). The Zen is foundational; the philosophy is *operational* — the day-to-day rules that apply when writing automation code. ### Not "agree with everything or you can't contribute" The philosophy is opinionated. Reasonable engineers disagree with parts of it — the explicit-`return` rule and the named- parameter-passing rule both have detractors. The philosophy is not a litmus test; it's a *consistent set of choices that produce consistent outcomes*. An engineer who disagrees with one rule but accepts the rest can absolutely contribute; the project's code review process resolves the local disagreement against the project's chosen direction. What the philosophy *won't* tolerate is *inconsistent application*. A codebase where some functions use named parameters and others don't, where some functions end with `return` and others don't, where some exceptions are swallowed and others aren't — that's a codebase where the discipline has been broken without being replaced. Either follow the philosophy or replace it; don't half- follow it. ## The meta-thesis: framework as skeleton and patterns Beyond the ten engineering principles there is a single load-bearing design choice that shapes the entire portfolio. Every later volume references it and every chapter that says "this is left to the user" is an instance of it: > *The test framework just sets up the skeleton and patterns; > the organization customises and refines them to meet the needs > of the automation scenarios.* This is the *meta-thesis* of the `codecraftai-*` portfolio. It explains why so much of the framework's surface is "left to the organisation": - Topology schemas — the framework supplies the loader and one canonical shape (`clusters` with `name` / `group` / `nodes` / `spares`); the organisation supplies the rest by subclassing `Topology`. - Feature handling on devices — the framework dispatches a small set of built-in Features; the organisation adds the rest by subclassing `Landscape`. - Verification logic — the framework supplies hooks (`validate_topology` and friends); the organisation implements the project-specific verification. - Runtime profiles — the framework supplies the composition machinery (the `MergeMap` overlay covered in chapter 9); the organisation defines the actual profiles (`Normal`, `Debug`, `UPNP Diagnostics`, `Bluetooth Diagnostics`, etc.). - Coordinators / Agents / ProtocolExtensions — the framework supplies built-ins for common protocols (SSH, UPnP, Serial, Power) and the *pattern* for adding more; the organisation adds the protocols their automation actually needs. - Tandem validators — the framework supplies the `Validator` base class (in `codecraftai-testbase/source/packages/ccai/testbase/tandem/`) and the attachment mechanism; the organisation writes the validators that watch the system states their tests care about. What the framework *is* opinionated about (the skeleton): - The `TestJob` lifecycle and its five `fire_*` hooks. - The single-actor + multi-observer determinism model (Volume 1 chapter 1; Volume 2 chapter 2). - The Pod-as-unit-of-parallelism principle (Volume 2 chapter 3). - The six configuration categories (Specifications, Landscapes, Topologies, Runtimes, Credentials, Data Profiles), all uniformly composable via `MergeMap` (chapter 9 of this volume, Volume 3 part II). - The CI hand-off as a *verified contract* — Landscape reachability + Topology configured-state checked before the first test runs (Volume 3 chapter 21). - The JSOS streaming results format and the `agents` CLI shape (Volume 1 chapters 5 and 6). - The `arcli` decorator surface for command-line tooling (Volume 1 chapter 8). - The Aspects pattern for behaviour parameterisation (Volume 5 chapter 1). What the framework is *not* opinionated about (left to the organisation): - The shape of any individual Topology beyond the canonical `clusters` form. - Which Features any given device has. - What the organisation's validators check for. - What the organisation's runtime profiles do. - What protocols the organisation needs Agents for. - What the organisation's Specifications declare to provision. ### How the customisation actually plugs in The concrete mechanism: the user subclasses `TestJob` (or one of its descendants — `InteropLibraryTestJob`, `DistributedTestJob`). The subclass overrides the template methods that return the user's `Landscape` subclass, the user's `Topology` subclass, the user's sequencer, and so on. The framework's entry points (`generic_test_entrypoint(job_type=...)` and `distributed_test_entrypoint(job_type=...)`) accept the user's job class via the `job_type=` parameter. That single chain — `TestJob` subclass → custom `Landscape` / `Topology` returned by template methods → custom Features and schemas dispatched inside those subclasses — is the entire customisation contract. Volume 2 chapter 4 walks the inheritance hierarchy in detail; Volume 6 walks Landscape and Topology subclassing as it applies to the environment model. ### Why the meta-thesis matters Without this framing, every "left to the user" note in every chapter reads as *gap*. With it, every "left to the user" note reads as *intentional design*. The framework is *opinionated about patterns and unopinionated about domain*, and that distinction is the design choice that makes the portfolio coherent across automation scenarios as different as a single-host unit suite, a 12-host distributed integration suite, a cloud-provisioned reference platform, and an embedded firmware test rig. This series treats every "left to the user" as a feature. When a chapter says "users implement this by subclassing X," the implication is *this is exactly where the framework expects you to specialise*, not *this is where the framework stops short*. ## How the philosophy shapes the rest of the series Every later chapter of this volume — and every later volume — applies the philosophy's principles to a specific area of code. A few preview connections: - **Chapter 3 (Error vocabulary)** is the philosophy's *errors- never-silent* principle made operational with specific exception classes. - **Chapter 4 (Interfaces and contracts)** is the philosophy's *explicit-is-better-than-implicit* principle made operational through abstract base classes that name what implementers must provide. - **Chapter 6 (Application identity and presence)** is the philosophy's *debuggability-paramount* principle made operational through the `ThisApplication` machinery that gives every running process a structured identity. - **Volume 3 chapter 7 (Runtime files vs environment variables)** is the philosophy's *explicit-is-better-than-implicit* principle applied at the orchestration-to-job seam. - **Volume 5 chapter 1 (The Aspects pattern)** is the philosophy's *stability-over-performance* and *if-it's-hard-to- follow-try-another-way* principles applied to behaviour-knob parameterisation. The principles are not used as decoration. They drive the specific design decisions in every chapter that follows. When you find yourself reading a later chapter and wondering "why is the code shaped this way?" — the answer is in this chapter. ## Summary - The CodeCraftX philosophy is **ten engineering principles** that govern the `codecraftai-*` portfolio. Each principle has concrete code-level consequences enforced through code review. - **Debuggability is paramount** is the most important principle. When two principles conflict, debuggability wins. Every choice the philosophy makes for debuggability has a cost paid in verbosity; the cost is paid once per function written, and the benefit is paid every time the function is debugged. - **Explicit is better than implicit** turns into specific rules: no mutable defaults, named parameter passing, explicit `return` statements. Each rule trades a few characters of verbosity for refactor safety and readability. - **Errors should never pass silently** turns into the six-rule exception-handling discipline of chapter 3. The principle is not "exceptions are good"; it's "errors that fail loudly are easier to debug than errors that fail silently." - **Stability is more important than performance** for code that isn't in a hot loop. Most automation code isn't. When the code is the hot loop, the principle inverts; the philosophy acknowledges this. - **The meta-principle**: if applying the others doesn't produce clear code, back up and try a different design. This catches clever architecture before it ships. - The philosophy is not PEP 8, not the Zen of Python verbatim, and not a litmus test. It is a *consistent set of opinionated choices*; the consistency is what produces a portfolio whose conventions transfer across packages. - The narrator of the series — **CodeCraftX** — is the persona that consistently applies the philosophy. The persona's voice carries the conventions; the conventions carry the platform. The next chapter takes the philosophy's principles and turns them into the *specific coding standards* every package in the portfolio enforces. Where this chapter said *what the philosophy is*, the next chapter says *what the philosophy looks like in code that ships*. # Chapter 2 — Coding Standards as a Tool, Not a Style Guide > *A style guide tells you how to format your code. A coding > standard tells you how to engineer it. This chapter is about > the second.* The previous chapter named ten engineering principles. This chapter takes those principles and turns them into the specific *rules* every package in the foundation tier follows. Each rule has a *reason*. The reason is the chapter's central concern; the rules without the reasons are arbitrary, and arbitrary rules get worked around. The canonical source of these rules is the `CLAUDE.md` file kept at the root of the engineering workspace, outside any one repository. The file's job is to tell automated coding assistants what the project's conventions are; the same file serves equally well as the human-readable reference for the team. This chapter unpacks each rule with the rationale and a worked example. ## Formatting rules with engineering reasons The first family of rules looks like style preferences but isn't. Each one buys something specific. ### No semicolons to terminate or join lines Python permits semicolons; the convention forbids them. The reason is debuggability: a line that uses semicolons to chain statements (`x = 1; y = 2; z = x + y`) is one source-line that hits multiple bytecode instructions. When the debugger breaks at that line, it stops at the first instruction; stepping doesn't take you to the next statement on the same line — it takes you to *the next line*. Splitting `x = 1; y = 2; z = x + y` into three separate lines gives the debugger three breakpoint anchors. This is a small example of debuggability-paramount; the cost is one newline per statement, the benefit is one breakpoint per statement. ### `snake_case` for functions and methods The convention is consistent with PEP 8. The reason is *machine readability* — when the codebase is grepped for symbol names across packages, the convention tells the reader whether a name is a function (`run_command`), a class (`RunCommand`), or a module (`run_command.py`). No string can be ambiguous about its kind. ### Type hints on every parameter and return ```python # Required def parse_workflow(workflow_stream: TextIOWrapper) -> Workflow: ... # Forbidden def parse_workflow(workflow_stream): ... ``` The discipline is universal: every parameter and every return, including for private helpers. The reason is twofold: - **Static analysis** catches type mismatches before they reach runtime. The IDE flags the call site that passes the wrong type; the linter flags the function that returns the wrong shape. - **Documentation** comes for free. A reader of an unfamiliar function knows what it expects and what it produces from the signature alone, without having to read the body. ### `self` and `cls` discipline Instance methods take `self` as the first argument; class methods take `cls`. The convention is consistent with Python's standard naming. The reason is the *constraint satisfier* role from chapter 1 — when the codebase is uniform on this point, a method's binding is unambiguous from the first parameter. ## Return statement rules The `return` discipline is one of the most distinctive parts of the convention. It has four specific rules. ### Every function and method ends with `return` ```python def configure_logger(logger: Logger) -> None: logger.setLevel(logging.INFO) logger.addHandler(StreamHandler()) return ``` The `return` is required even when the function returns `None`. Three reasons (the chapter-1 preview develops these in detail): 1. **Debugger breakpoint anchor.** The Python interpreter generates a return-bytecode whether or not the source has the keyword. An explicit `return` lets the debugger attach a breakpoint at a *named line*, which inspects every local variable in the function before exit. Without the keyword, the breakpoint lands on the closing brace's invisible bytecode location, which most debuggers handle awkwardly. 2. **Refactor safety.** When a function is deleted (or a `def` line is removed during refactoring), its trailing `return` becomes *unreachable code* in the next function down. Linters flag unreachable code; the bug is caught. Without the trailing `return`, the orphaned function body silently merges with the next function's body, and the bug ships. 3. **Function-end visibility.** `return` at column zero (after dedent) is a clear delimiter — the reader knows the function ends here. ### Assign return values to a local variable before returning ```python # Discouraged def get_user_count(query: Query) -> int: return query.execute().scalar() + 1 # Preferred def get_user_count(query: Query) -> int: raw_count = query.execute().scalar() user_count = raw_count + 1 return user_count ``` The local variable serves two purposes. At debug time, the debugger can break on the `return` and inspect `user_count` to see what's about to be returned. At maintenance time, the intermediate name documents the calculation — `raw_count + 1` is opaque without the variable name; `user_count = raw_count + 1` explains the offset's meaning. ### No compound `return` statements ```python # Forbidden def find_or_default(key: str, registry: Dict[str, Any]) -> Any: return registry.get(key, default_for_key(key)) # Required def find_or_default(key: str, registry: Dict[str, Any]) -> Any: if key in registry: return registry[key] default_value = default_for_key(key) return default_value ``` The forbidden form has *two* function calls and a conditional all on one line. Setting a breakpoint on it gives no fine-grained control. The required form gives the engineer three breakpoint anchors — the `if`, the first `return`, and the second `return` — and makes the conditional logic explicit. ### Standalone `return` for void functions ```python def install_logger(logger: Logger) -> None: logger.addHandler(StreamHandler()) return # not `return None`; just `return` ``` When the function returns nothing meaningful, the `return` statement is bare. This is a convention rather than a correctness issue (the two forms are equivalent), but the bare form distinguishes void functions visually from functions that happen to return `None` as a sentinel. ## No mutable defaults ```python # Forbidden — every call shares the same list def add_item(item: str, items: List[str] = []) -> List[str]: items.append(item) return items # Required def add_item(item: str, items: Optional[List[str]] = None) -> List[str]: if items is None: items = [] items.append(item) return items ``` The forbidden form has a default argument that is *evaluated once at function definition time*. Every call that doesn't override `items` shares the same list — an action-at-a-distance bug that's easy to introduce and hard to catch. The required form takes `None` as the default and creates the mutable inside the body. Every call gets a fresh list when no list is passed. The verbosity is two extra lines; the safety is permanent. ## Class and file naming The portfolio's class-and-file-naming convention is opinionated: ``` # File: someclass.py class SomeClass: pass ``` The class name is `PascalCase` with no underscores. The file name is the class name lowercased, also without underscores. One class per file, where the class is the file's reason for existing. The reason: navigation. When an engineer reads `from ccai.taskplus.tasks.runcommand import RunCommand`, the path `ccai/taskplus/tasks/runcommand.py` is the file's location mechanically. No mental translation, no searching, no case-folding rules to remember. The convention has consequences. When a module needs five small classes that belong together, they don't all live in one file — they live in five files in the same package, with `__init__.py` re-exporting them as needed. This is *more* files than a less disciplined codebase, by design; the navigation gain is worth the file count. ## Class-variable initialisation in `__init__` ```python class Workflow: def __init__(self, environment: Dict[str, str], parameters: Dict[str, Any], flow: List[FlowNode]): self._environment = environment self._parameters = parameters self._flow = flow return ``` Every instance variable is initialised in `__init__`, and *only* in `__init__`. Adding a `self.foo = bar` in any other method violates the rule. The reason: when the constructor lists every instance variable in one place, a reader can see the class's full state in one view. When variables are introduced in random methods, the reader has to walk the whole class to know what state exists. The first pattern is *predictable*; the second is *archaeology*. ## Decision expressions Two specific rules govern conditional expressions. ### No shortcut truthy/falsy checks ```python # Forbidden if some_list: # ... # Required if some_list is not None and len(some_list) > 0: # ... ``` The forbidden form mashes two conditions ("not None" and "not empty") into one. When the variable is `None`, the test is False; when the variable is `[]`, the test is also False; when the variable is `[0]`, the test is True. These three cases have different semantics and probably want different handling — but the shortcut hides them. The required form makes the two checks explicit. When the variable is `None`, the first clause short-circuits. When the variable is empty, the second clause does. When the variable has content, both pass. Each branch can have its own error message, its own logging, its own recovery path. ### Separate error conditions, separate handling ```python # Forbidden if name is None or len(name) == 0 or name in BLACKLIST: raise ConfigurationError("Name is invalid.") # Required if name is None: raise ConfigurationError("The 'name' parameter is required.") if len(name) == 0: raise ConfigurationError("The 'name' parameter cannot be empty.") if name in BLACKLIST: raise ConfigurationError("The name '%s' is blacklisted." % name) ``` The forbidden form gives the same error message regardless of which condition triggered. When the user receives "Name is invalid," they don't know whether to fix a missing parameter, an empty string, or a blacklist violation. The required form gives a *specific* error per condition. The caller's mental model snaps to the right fix immediately. This is the *errors-never-silent* principle from chapter 1 made operational at the validation layer. ## Decorator usage Decorators are *only* allowed for two purposes: 1. **Adding metadata.** `@priority(1)`, `@categorize("integration")`, `@deprecated(since="2.0")`. The decorator attaches a value or tag to the function; the function's behaviour is unchanged. 2. **Registering with a mapping.** `@app.route("/")`, `@click.command()`, `@register_handler("user_login")`. The decorator adds the function to a registry; the function's body is otherwise untouched. Decorators are *not* allowed for: - **Behaviour modification.** No `@retry`, no `@timeout`, no `@cache`, no `@log_calls`. Behaviour modification belongs in parameters or in wrapping objects (see Volume 5's Aspects pattern), not in decorators. - **Cross-cutting concerns.** No `@requires_auth`, no `@validate_input`, no `@audit_log`. These belong in explicit middleware or in the function's body. The reason: behaviour-modifying decorators *hide* control flow. A function decorated with `@retry(3)` doesn't run once when the caller invokes it; it runs up to three times, with delays in between, with exception swallowing built in. The caller reading the call site has no visual cue that any of this is happening. The same behaviour expressed as an explicit `aspects=Aspects(retry=3)` parameter is *visible* at the call site; the reader knows what they're getting. This is the *explicit-is-better-than-implicit* principle from chapter 1 made operational at the decorator boundary. Volume 5 develops the Aspects pattern as the affirmative answer for behaviour-knob parameterisation. ## Documentation comments The convention is Sphinx-style triple-quoted docstrings: ```python def some_function(a: int, b: str) -> str: """ This is a test function. :param a: The value for 'a'. :param b: The value for 'b'. :returns: The word 'blah' in a string. :raises: ValueError """ if a <= 0: raise ValueError("'a' must be greater than 0") rtnval = "blah" return rtnval ``` The shape: - Multi-line triple-quote, even for single-line descriptions. - Description starts on the *second* line. The first line is whitespace. - Body indented four spaces inside the opening triple quote. - `:param name:` for parameters. No type in the docstring (the type hint carries it). - `:returns:` for the return. No type in the docstring. - `:raises:` for the exception types the function can raise. The shape produces parseable docstrings — Sphinx's `autodoc` and the IDE's hover-help both render these correctly. The type-information-in-hints discipline avoids duplicating the type in the docstring (where it would drift from the signature). ## Exception handling The exception-handling discipline has six rules covered briefly in chapter 1; chapter 3 develops them in full. As a preview: 1. Don't handle exceptions you can't recover from. 2. Catch and reclassify to add specific contextual information. 3. Catch and reclassify to improve code-context in failure reports. 4. Catch and add extra contextual information about unique circumstances. 5. `SemanticError` must NEVER be caught. Same for `ConfigurationError` in most contexts. 6. In all other cases, let exceptions bubble. The discipline is *not* "exceptions are good" or "exceptions are bad." It is *errors that fail loudly are easier to debug than errors that fail silently*, applied at the catch boundary. ## Multi-line error messages Long error messages are constructed as a list of lines, then joined: ```python err_msg_lines = ["Failed to find expected devices."] err_msg_lines.append("EXPECTED: ({})".format(len(expected))) for key in expected: err_msg_lines.append(" {}:".format(key)) err_msg = os.linesep.join(err_msg_lines) raise TimeoutError(err_msg) from None ``` Three reasons: - **Cross-platform line endings.** `os.linesep` is `\r\n` on Windows and `\n` on Unix. Joining with the constant produces correct line endings on every platform without the author having to think about it. - **Composability.** Building the list with conditionals — appending different sections based on what context is available — is straightforward. Building the same string with `+=` and embedded newlines is brittle. - **Readability.** The list-of-lines form makes the error message's *structure* visible in code: a header line, a count, an indented per-item block, a footer. The string- concatenation form hides the structure. ## InterOp API design — the Aspects pattern For APIs that interact with remote systems (SSH, REST, filesystem operations), the discipline forbids behaviour-modifying decorators (rule above) and forbids hardcoded retry/timeout behaviour. Instead, the API takes an `Aspects` object as a parameter: ```python def run_command(command: str, *, aspects: Aspects = None) -> CommandResult: if aspects is None: aspects = DEFAULT_ASPECTS # ... use aspects.timeout, aspects.retry_count, aspects.retry_backoff_ms ... ``` The `Aspects` object packages every behaviour parameter the API might want — timeout, retry count, retry backoff, logging level, sync vs async — into one scalable parameter. Adding a new behaviour knob is one field in the `Aspects` class; existing callers don't need to know about it. Volume 5 chapter 1 develops the Aspects pattern in full, including the read-only-vs-state-modifying API design heuristic (read-only APIs typically don't need Aspects; state-modifying APIs typically do). Chapter 1 of this volume previewed the pattern; chapter 23 of Volume 3 used it for the Retry wrapper task. The discipline is the same across volumes. ## Properties as simple data accessors ```python # Required @property def host(self) -> str: return self._host # Forbidden @property def host(self) -> str: if self._host is None: self._host = self._connection.lookup_host() return self._host ``` A property is a typed read of a stored value. It is *not* a hook for lazy initialisation, side-effecting computation, or any operation that might block. The reason: properties look like attribute access at the call site. When `obj.host` triggers a 30-second SSH connection, the caller has no visual cue that any work is happening. Worse, when `obj.host` is read inside an `__init__` of a related class, the lazy initialisation can produce circular dependencies that are nearly impossible to debug. When lazy initialisation is genuinely needed, the right shape is an explicit method (`get_host()`) with the work named. When caching is needed, a method called `lookup_host()` that caches internally is fine — the call site sees a method invocation and knows that work might be happening. ## Coding standards across volumes The standards in this chapter apply to every volume in the series. Specific applications: - **Volume 1 (Testing)** uses the `assert_*` family of verification functions instead of bare Python `assert` statements; the explicit-is-better discipline shows up in test assertions. - **Volume 3 (Orchestration)** uses the no-mutable-defaults rule in workflow YAML loading; the `parse` classmethod doesn't have `Workflow=Workflow()` defaults. - **Volume 5 (Interop)** is the affirmative-Aspects volume; the decorator discipline of this chapter justifies the Aspects pattern's existence. When you read code in any volume of the series and find yourself asking "why does it look like this?" — chapter 1 is the philosophy; this chapter is the rules; the next chapter is the error vocabulary that makes the *errors-never-silent* principle operational. ## Summary - The standards in this chapter are **rules with reasons**. Each rule has a debuggability, refactor-safety, or explicit-is-better justification. The reasons matter; rules without reasons get worked around. - **Formatting rules**: no semicolons, snake_case functions, type hints everywhere, `self`/`cls` discipline. Each one buys something specific. - **Return statement rules**: every function ends with `return`, return values assigned to local variables before returning, no compound returns, bare `return` for void functions. The rationale is debuggability + refactor safety + function-end visibility. - **No mutable defaults**: prevents action-at-a-distance bugs from shared mutable state across calls. Two extra lines per function with a default; permanent safety. - **Class-and-file naming**: PascalCase classes; lowercase-no- underscores file names matching the class. One class per file. More files than a less-disciplined codebase, by design. - **No shortcut truthy/falsy checks**. `if some_list is not None and len(some_list) > 0` makes both conditions explicit; each can have its own error message and recovery path. - **Decorators only for metadata or registration**. Behaviour- modifying decorators are forbidden; behaviour parameters live in `Aspects` objects. - **Sphinx-style docstrings** on every public function, with `:param:` / `:returns:` / `:raises:` and no type duplication. - **Properties as simple data accessors only**. Lazy init, side-effecting computation, and blocking operations belong in explicit methods, not properties. - **Multi-line error messages** built from a list joined with `os.linesep`. Cross-platform, composable, readable. The next chapter develops the error vocabulary that makes the *errors-never-silent* principle operational. Three special exception types — `SemanticError`, `ConfigurationError`, `AssertionError` — have specific must-bubble or must-catch semantics that the rest of the portfolio depends on. # Chapter 3 — The Error Vocabulary > *Errors that fail loudly are easier to debug than errors that > fail silently. An error vocabulary turns that principle into a > menu of typed exceptions, each with a specific meaning and a > specific must-bubble or must-handle status.* The previous chapter established the coding standards. This chapter develops one of them in depth: the exception-handling discipline. The discipline rests on [`ccai.errors`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/errors/exceptions.py) — a small subpackage of `codecraftai-foundations` whose job is to provide the **typed exception vocabulary** every other ccai package uses. The package's contents are simple in shape but consequential in meaning. Eighty exception classes; most are RuntimeError subclasses; three are deliberately *not* — `SemanticError` and `ConfigurationError` inherit from `BaseException` directly, and `AbstractMethodError` from `RuntimeError`. The inheritance choices encode the must-bubble and must-handle decisions that the rest of the portfolio relies on. ## The three special types Three exception types in the vocabulary have semantics that matter universally across the portfolio. Every package documents them; every package's exception handling respects them. This section is the canonical statement of what each means. ### `SemanticError(BaseException)` — must always bubble ```python class SemanticError(BaseException): """ The base error object for errors that indicate that there is an issue with a piece of automation code and with the way the Automation Kit code is being utilized. """ ``` `SemanticError` signals **API misuse**. When code raises it, the caller has invoked the API in a way the API can't make sense of — passed contradictory parameters, called a method out of order, constructed an object with mutually-exclusive options. These are *the caller's bugs*, and they need to surface to the caller. The inheritance is deliberate: `BaseException`, not `Exception`. This means `except Exception:` *does not catch* `SemanticError`. A bare `except Exception:` block — even a broad recovery handler in well-meaning library code — does not silently absorb a SemanticError. The error bubbles up through every Exception-targeted handler and reaches the top of the stack, where the developer who made the API-misuse mistake sees the failure with full context. The discipline: - **Code may not catch `SemanticError`.** Ever. The only legitimate handler is the *outermost* layer of the application, where the misuse is logged and the process exits. - **Code may raise `SemanticError`** when the caller has violated the API's contract in a way the API can detect. Concrete example. Consider a credential class that requires *either* a password or a keyfile but not both: ```python class SshCredential(BaseCredential): def __init__(self, *, identifier: str, password: Optional[str] = None, keyfile: Optional[str] = None): super().__init__(identifier=identifier) if password is not None and keyfile is not None: raise SemanticError( "SshCredential accepts password OR keyfile, not both." ) if password is None and keyfile is None: raise SemanticError( "SshCredential requires one of: password, keyfile." ) self._password = password self._keyfile = keyfile return ``` A caller who constructs `SshCredential(identifier="x", password="pw", keyfile="/path")` has misused the API. The SemanticError fires; the constructor's caller sees the failure; the caller fixes the misuse. No layer of recovery code in between gets the chance to swallow it. ### `ConfigurationError(BaseException)` — must bubble in most contexts ```python class ConfigurationError(BaseException): """ The base error object for errors that indicate that there is an issue related to improper configuration. """ ``` `ConfigurationError` signals **configuration problems** — a landscape file that's missing a required field, a credential identifier that doesn't resolve, a runtime YAML with a malformed entry. Like `SemanticError`, it inherits from `BaseException` so generic exception handlers don't silently absorb it. The semantics are slightly more nuanced than SemanticError's: - **Library code may catch `ConfigurationError`** to *augment* the message and re-raise. When a configuration problem surfaces three layers deep in a loader, the layer that knows about the source URI can catch the error, append its context, and re-raise: ```python try: payload = self._parse_yaml(content) except ConfigurationError as err: errmsg = "Error parsing %s: %s" % (self._source_uri, err) raise ConfigurationError(errmsg) from err ``` The original exception is preserved as the cause (`from err`); the new message adds the source URI; the caller receives a more informative error. - **Library code may not catch `ConfigurationError` to absorb it.** When the library can't recover from the configuration problem, the error bubbles. The application's top-level handler reports the configuration issue and exits cleanly. The two specific subclasses in the package make finer distinctions: - `MissingConfigurationError` — a required field or file is missing. - `InvalidConfigurationError` — a field is present but holds an unusable value. Both inherit from `ConfigurationError`, so the must-bubble semantics apply to both. Library code that needs to react to "missing vs invalid" specifically can catch the subclass; library code that just wants to know "configuration is wrong" catches the base. ### `AssertionError` — built-in, but with discipline `AssertionError` is Python's built-in. The convention treats it as the *condition or state failure in the target under test* — the check that fires when the system isn't behaving as expected. The discipline: - **Test code raises `AssertionError`** through the `verification.assert_*` family or through bare `assert` statements. The error propagates to the test runner, which reports the failure. - **Library code does *not* raise `AssertionError`** for invariant violations. Use `SemanticError` for "the caller used the API wrong" or `RuntimeError` (or a more specific subclass) for "something unexpected happened." Reserving `AssertionError` for tests keeps the test framework's reporting clean. ## The other categories The package's remaining ~80 exception classes fall into seven operational categories. Each is a `RuntimeError` subclass (or inherits from a more specific built-in like `TimeoutError`), which means generic `except Exception:` handlers *do* catch them when appropriate. ### Command and process - **`CommandError`** — a wrapped subprocess command returned a non-zero exit code. The exception carries the `status`, `stdout`, and `stderr` of the failed command, so the handler has the operational context to diagnose. - **`UnExpectedSessionClosureError`** — an SSH or similar session closed without explicit close. ### Resource lifecycle - **`CheckinError`** — a resource checkin failed. - **`CheckoutError`** — a resource checkout failed (chapter 6 of Volume 6 develops this in the landscape context). - **`PublishError`** — publishing an artifact failed. ### Flow control - **`CancelledError`** — an operation was cancelled. Three subclasses with finer meaning: `TaskingCancelled` for task-level cancellation, `TaskingGroupCancelled` for group-level. - **`HaltError`** — a child flow-control context asks the parent to halt. - **`SkipError`** — a child flow-control context (typically a test case) asks to be skipped. ### Tasking and looping - **`LooperError`** — generic looper-pool error. - **`LooperQueueShutdownError`** — work queued on a shut-down queue, or a worker waiting on an empty shut-down queue. - **`TaskingGroupRuntimeError`** — runtime error inside a task-group execution. - **`TaskingGroupAssertionError`** — an assertion failed inside a task group. Inherits from `AssertionError` (deliberate exception to the "library code doesn't raise AssertionError" rule because tasking groups *are* test-running code). ### Override / abstract method - **`AbstractMethodError`** — an abstract method was called without being overridden. Used by abstract base classes when the base provides a "you must override" stub: ```python class Task(FlowNode): @abstractmethod def perform_task(self, environment, parameters, flowstate) -> int: raise AbstractMethodError( "Task->perform_task must be overloaded by derived classes." ) from None ``` - **`NotSupportedError`** — a method that exists is not supported in the current context (different from "must be overridden"). - **`UnSupportedPlatformError`** — the current platform doesn't support the operation (e.g., a macOS-specific call invoked on Linux). - **`UnknownParameterError`** — an unknown keyword argument was passed. ### Timeout Three subclasses of Python's built-in `TimeoutError`, each naming a specific timeout shape: - **`CompletionTimeout`** — an operation didn't complete in the allotted time. - **`ConnectionTimeout`** — a connection attempt didn't succeed in the allotted time. - **`InActivityTimeout`** — an operation produced no progress for too long (different from "didn't complete" — the operation may still be making progress, but in a different dimension). ### Command-line errors A family of `CommandlineError` subclasses for argparse-style parsing problems: `BadOptionUsageError`, `BadParameterError`, `MissingArgumentError`, `MissingOptionError`, `NoSuchOptionError`, `TooManyArgumentsError`. ### HTTP errors The largest sub-family — roughly 50 classes — mirroring HTTP status codes. Three top-level categories (`HttpResponse3xxError`, `HttpResponse4xxError`, `HttpResponse5xxError`) each have specific subclasses for the common status codes (`HttpNotFoundError`, `HttpUnAuthorizedError`, `HttpInternalServerError`, etc.). The discipline: - **HTTP-aware code may catch the specific subclass** when it has a recovery path (retry on a 503, return cached result on a 404, refresh token on a 401). - **HTTP-aware code may catch the family base** (e.g., `HttpResponse5xxError`) when it wants to handle all server-side errors uniformly. - **HTTP-unaware code lets them bubble.** They are `HTTPException` subclasses; generic exception handlers see them. The fine-grained vocabulary is what makes per-status-code recovery practical without a regex over the message text. ## The six exception-handling rules Chapter 1 previewed the six rules; chapter 2 mentioned them; this chapter is the canonical statement. ### Rule 1: Don't handle exceptions you can't recover from ```python # Forbidden try: result = fetch_remote_value() except Exception: pass # silently absorbed — the result is now undefined ``` The pattern silently produces an undefined value. When the caller eventually uses it, the failure is far from the cause, and the engineer debugging has no breadcrumb. The discipline: *don't catch what you can't handle*. Let the exception bubble; the caller gets to decide. ### Rule 2: Catch and reclassify to add specific contextual information ```python # Required, when the catching layer has context the inner layer doesn't try: payload = self._parse_yaml(content) except yaml.YAMLError as err: errmsg = "Failed to parse landscape file %s." % self._source_path raise InvalidConfigurationError(errmsg) from err ``` The catching layer knows the *source path*; the inner layer doesn't. The catch reclassifies the generic `yaml.YAMLError` into a specific `InvalidConfigurationError` with the path embedded. The original exception is preserved as `__cause__` through `from err`. ### Rule 3: Catch and reclassify to improve code-context in failure reports ```python # Required, when the catching layer can name *what it was trying to do* try: cluster = self._build_cluster(name=name, members=members) except KeyError as err: errmsg = "Building cluster %r failed: missing landscape member %s." % (name, err) raise ConfigurationError(errmsg) from err ``` A `KeyError` from deep inside the dispatch tree is hard to attribute. Reclassifying it at the layer that *named the operation* turns the failure into a clear "I was building cluster X and member Y was missing." ### Rule 4: Catch and add extra contextual information The variant where the original exception type is preserved but the *message* is enriched: ```python try: self._engine.connect() except ConnectionTimeout as err: err.add_note("Last successful connection: %s" % self._last_success) raise ``` (`add_note` is Python 3.11+; for earlier Pythons, the same pattern uses `from` and a wrapping exception.) The original exception type is preserved, so callers' type-based handlers still work; the message gains operational context. ### Rule 5: SemanticError must NEVER be caught The discipline applies to library code, application code, and test code uniformly. A `try: ... except SemanticError:` block *anywhere* is a violation. The rationale (above): SemanticError signals API misuse, and absorbing it lets the misuse continue silently. The same rule applies to `ConfigurationError` in most contexts — the exception should reach the layer that *can act on* a configuration problem (typically the application's top-level handler that emits a clear error message and exits). Library layers may *augment* the error and re-raise (rule 2/3) but may not absorb it. ### Rule 6: In all other cases, let exceptions bubble The default is *no catching*. When a function doesn't have a specific recovery path, it doesn't have a `try`/`except`. The exception travels up the stack; some higher layer either has a recovery or surfaces the failure to the user. This is the *errors-never-silent* principle in action. A codebase whose default is "catch and recover" accumulates recovery points that don't actually recover; the failures get absorbed; the bugs ship. A codebase whose default is "let it bubble" produces clear failures that the engineer can diagnose. ## Why `BaseException` for SemanticError and ConfigurationError Python's exception hierarchy: ``` BaseException ├── SystemExit ├── KeyboardInterrupt ├── GeneratorExit └── Exception ├── ArithmeticError, LookupError, OSError, RuntimeError, ValueError, ... └── (most user-defined exceptions) ``` The convention is that `Exception` is for application errors that *can* be caught; `BaseException` is for system-level events (`SystemExit`, `KeyboardInterrupt`) that *should not* be caught by generic `except Exception:` handlers. By inheriting `SemanticError` and `ConfigurationError` from `BaseException` directly, the package opts them into the same "don't catch generically" treatment. A library wrapper with `except Exception:` for legitimate recovery does *not* swallow a SemanticError — the caller's API misuse still surfaces. This is one of the portfolio's most consequential design choices. Most third-party libraries put their custom exceptions in the Exception subtree; the ccai portfolio deliberately lifts the must-bubble ones out. It's a small choice in the class definition, with a large consequence at every catch site across every package. ## The vocabulary's role across the portfolio Every other package in the foundation tier uses `ccai.errors` for its exception vocabulary. Specific uses: - **Volume 0 chapter 4 (Interfaces)** uses `AbstractMethodError` to signal that an abstract method's stub was hit because no subclass overrode it. - **Volume 0 chapter 5 (Waiting)** uses `CompletionTimeout`, `ConnectionTimeout`, `InActivityTimeout` for the three timeout shapes. - **Volume 0 chapter 9 (Configuration loading)** uses `ConfigurationError`, `MissingConfigurationError`, `InvalidConfigurationError` for the load-time validation. - **Volume 3 (every chapter)** uses the vocabulary uniformly for both library-side and application-side error reporting. The vocabulary is *what makes the portfolio interoperate*. When two packages from different tiers want to talk about errors, they talk in the shared language. A landscape error in Volume 6 that surfaces during a Volume 3 orchestration run is the same exception type both packages understand. ## Summary - **`ccai.errors`** is a small subpackage of `codecraftai-foundations` with ~80 exception classes organised into seven operational categories plus three *special* types whose semantics matter universally. - **`SemanticError(BaseException)`** signals API misuse. Must always bubble. Library code may not catch. Inherits from `BaseException` so generic `except Exception:` does not swallow it. - **`ConfigurationError(BaseException)`** signals configuration problems. Library code may catch only to *augment* the message and re-raise. Same `BaseException` inheritance for the same reason. - **`AssertionError`** is Python's built-in. The convention treats it as the test-framework's signal; library code does not raise it. - The other 80 exceptions are organised by operational category: command/process, resource lifecycle, flow control, tasking, override/abstract, timeout, command-line, HTTP. Each is a `RuntimeError` (or specific-built-in) subclass that *can* be caught when appropriate. - The **six exception-handling rules** govern when to catch and when to bubble: 1. Don't catch what you can't handle. 2. Catch to add contextual information. 3. Catch to improve code-context in reports. 4. Catch to add operational context to existing errors. 5. Never catch `SemanticError`. 6. Default to letting exceptions bubble. - The **`BaseException` choice** for `SemanticError` and `ConfigurationError` is the portfolio's most consequential exception-handling decision. It's small in the class definition; large at every catch site. - The vocabulary is what makes packages **interoperate**. Two packages from different tiers talk about errors in the shared language. The next chapter introduces interfaces and contracts — the abstract-base-class machinery that lets packages declare what implementers must provide without specifying how. # Chapter 4 — Interfaces and Contracts > *An interface is a promise about behaviour without a commitment > to implementation. The promise is the contract; the contract > is what makes packages substitutable.* The previous chapter named the typed exception vocabulary that makes packages talk about errors in the same language. This chapter names the typed *interface* vocabulary that makes packages substitute for each other without their callers caring which implementation is on the other end. The code is the [`ccai.interfaces`](https://github.com/code-craft-ai-company/codecraftai-foundations/tree/main/source/packages/ccai/interfaces) subpackage of `codecraftai-foundations`. Eight files; eight protocols; one purpose: **shared contract definitions that downstream packages implement and other downstream packages consume.** The package itself contains no behaviour — only the typed shape of the behaviour. ## Why protocols, not abstract base classes Python has two main mechanisms for declaring interfaces: - **Abstract base classes** (`abc.ABC` with `@abstractmethod`) — the implementer *inherits* from the ABC. Type checkers verify the inheritance. Runtime instance checks (`isinstance(x, MyABC)`) are accurate. - **Protocols** (`typing.Protocol`) — structural typing. The implementer matches the protocol if it provides the required methods, regardless of inheritance. Type checkers verify the shape; runtime instance checks require `runtime_checkable`. The portfolio's interfaces use **`Protocol`** uniformly. Two reasons: 1. **Decoupling.** An implementer that already inherits from another base (or that lives in a package that doesn't depend on `codecraftai-foundations` at the inheritance level) can still *be* an `ICommandRunner` if it has the right method shape. No forced inheritance; no diamond-inheritance problems. 2. **Substitutability.** Test mocks and fakes don't need to inherit from the interface to type-check as it. A test builds a small class with the right methods and passes it wherever `ICommandRunner` is expected. The cost of protocols: less rich runtime introspection. When the application needs *runtime* enforcement of an interface (rare in the portfolio), abstract base classes are the right choice. When the application needs *structural* checks at the type-checker level, protocols are. The portfolio uses protocols because the structural-check use case dominates. ## The eight protocols The shipped package has eight protocol files. This section walks each, anchored on the actual code. ### `ICommandRunner` — running commands across system boundaries [`icommandrunner.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/icommandrunner.py) defines the most-used protocol in the portfolio. Its job: provide a uniform shape for *running a command*, regardless of whether the command runs locally, over SSH, in a container, or via some other transport. ```python @dataclass class RunResult: exit_code: Optional[int] stdout: str stderr: str timed_out: bool = False class ICommandRunner(Protocol): def run( self, args: List[str], *, elevated: bool = False, cwd: Optional[str] = None, env: Optional[Dict[str, str]] = None, stdin: Optional[str] = None, timeout: Optional[float] = None, on_output: Optional[Callable[[str], None]] = None, on_error: Optional[Callable[[str], None]] = None, capture_output: bool = True, ) -> RunResult: ... ``` Eight keyword-only parameters. The *positional* parameter is the command (`args` as a list, not a string — explicit-is-better and shell-injection-safer). The keyword-only parameters are *aspects* of execution: elevated privilege, working directory, environment, stdin, timeout, output streaming callbacks, capture flag. The shape is consistent across implementations: - A *local* `LocalCommandRunner` calls `subprocess.run` with the args. - A *remote* `SshCommandRunner` packages the args into an SSH command and runs them on the remote host. - A *containerised* `DockerCommandRunner` calls `docker exec`. - A *test mock* `FakeCommandRunner` returns canned `RunResult` values without running anything. Code that needs to "run a command" takes an `ICommandRunner` parameter and gets to work without knowing which runner it received. ### `IFileOperator` — filesystem operations across boundaries [`ifileoperator.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/ifileoperator.py) is `ICommandRunner`'s sibling for filesystem work. Five methods: `write_text`, `delete`, `read_text`, `list_dir`, `exists`. Each takes an `elevated: bool` parameter, mirroring the privilege discipline from the runner. ```python class IFileOperator(Protocol): def write_text(self, path: str, content: str, elevated: bool = False) -> None: ... def delete(self, path: str, elevated: bool = False) -> None: ... def read_text(self, path: str, elevated: bool = False) -> str: ... def list_dir(self, path: str, elevated: bool = False) -> List[str]: ... def exists(self, path: str, elevated: bool = False) -> bool: ... ``` Same substitutability: a `LocalFileOperator` works against the local filesystem; a `SshFileOperator` works against a remote filesystem over SSH; a `S3FileOperator` works against S3-compatible object storage. The caller doesn't care. The minimalism is deliberate. Files have many possible operations (set permissions, change owner, create symlinks, copy ranges, append, truncate); the interface picks the **five most universally needed** and stops there. Implementers can offer *more* on their concrete classes; the interface only promises these five. ### `ISerializable` — dict round-trip [`iserializable.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/iserializable.py) is the smallest protocol in the package. Two methods: ```python class ISerializable(Protocol): @classmethod def from_dict(cls, data: Dict[str, Any]): ... def as_dict(self) -> Dict[str, Any]: ... ``` `as_dict` produces a dictionary representation; `from_dict` constructs an instance from one. The contract is round-trip: `obj == obj.from_dict(obj.as_dict())` should hold for any serialisable object. Code that needs to serialise heterogeneous objects (write to JSON, write to YAML, send over the wire to another process) takes `ISerializable` instances and gets to work. Code that needs to *consume* a configuration document constructs the right concrete class through `from_dict`. The chapter-9 configuration loader uses this pattern: every typed configuration class implements `ISerializable`; the loader calls `from_dict` to construct instances from parsed YAML/JSON. ### `IBasicCredential` — username + password [`ibasiccredential.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/ibasiccredential.py) is the smallest credential interface. Two read-only properties: ```python class IBasicCredential(Protocol): @property def username(self) -> str: ... @property def password(self) -> str: ... ``` Used by code that consumes basic credentials without caring *which* credential class produced them. A [`CouchDbProfile.connection_string(credobj: IBasicCredential)`](https://github.com/code-craft-ai-company/codecraftai-dataprofiles/blob/main/source/packages/ccai/dataprofiles/couchdbprofile.py) takes any object with `.username` and `.password` properties — a real `BasicCredential` from `codecraftai-credentials`, a test `FakeBasicCredential`, an inherited `SshCredential` that has both fields, etc. The credential package's `BasicCredential` *implements* this protocol; `CouchDbProfile` *consumes* it. The implementing package doesn't depend on the consuming package; the consuming package doesn't depend on the implementing package; both depend on the interfaces package. This is the dependency-inversion discipline in action. ### `IIncludeFilter` and `IExcludeFilter` — list filtering [`iincludefilter.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/iincludefilter.py) and [`iexcludefilter.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/iexcludefilter.py) provide a uniform interface for filtering lists of items. They are siblings: ```python class IIncludeFilter(Protocol): def should_include(self, check_object: Any) -> bool: ... def filter(self, input_list: List[Any]) -> List[Any]: ... class IExcludeFilter(Protocol): def should_exclude(self, check_object: Any) -> bool: ... def filter(self, input_list: List[Any]) -> List[Any]: ... ``` The pair separates *include* and *exclude* semantics deliberately. An include filter says "keep these"; an exclude filter says "drop these." Composing the two — running both against the same list — gives a uniform "include unless excluded" pattern that works without conditional flags. Both protocols ship a `filter(input_list)` *default* implementation that walks the list and applies `should_include` / `should_exclude` to each item. Implementers only need to override `should_include` (or `should_exclude`); the list walk is provided. This is one of the few places where the protocol carries a default implementation. It works because Python's `Protocol` is permissive about whether members are defined; the structural check passes if the implementer either overrides or inherits the default. ### `IHint` — friendly identification [`ihint.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/ihint.py) is a small protocol for objects that can produce a "hint" string matching them against another object: ```python class IHint(Protocol): def get_hint(self) -> str: ... def matches(self, obj: Any) -> bool: ... ``` Used by `codecraftai-landscaping`'s [`FriendlyIdentifier`](https://github.com/code-craft-ai-company/codecraftai-landscaping/blob/main/source/packages/ccai/landscaping/friendlyidentifier.py) class. A landscape's apod entries carry friendly names ("dev01", "runner-east-1") that match the actual host name through hint strings. The protocol formalises the relationship. ### `ITestJob` — the test-job contract [`itestjob.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/interfaces/itestjob.py) is the largest of the eight protocols. It defines the contract for a test job — the unit of work `codecraftai-testbase` and its extensions execute: ```python class ITestJob(Protocol): @property def importerrors(self): ... def begin(self) -> None: ... def execute(self) -> int: ... def finalize(self) -> None: ... def query(self, query: str, format: int=0) -> List[Any]: ... ``` Five methods plus one property. The lifecycle is `begin` → `execute` → `finalize`; the property surfaces import errors that occurred during the job's load; the `query` method exposes introspection for tooling. The protocol is the contract Volume 2's distributed test jobs implement (`TestJob` → `InteropLibraryTestJob` → `DistributedTestJob`). Each layer of the inheritance chain satisfies the protocol; consumers (test runners, reporters, schedulers) talk to whichever layer of the chain they have. ## How the interfaces compose The eight protocols are not independent — they compose into the larger contracts the higher tiers of the portfolio rely on. ### Composition example: a sysman feature A Volume 4 sysman feature operates on a per-host basis. Each host is represented by a `HostContext` that carries (among other things): - An `ICommandRunner` — to run commands on the host. - An `IFileOperator` — to read and write files on the host. - An `IBasicCredential` — to authenticate to the host. The sysman feature's code takes a `HostContext` and uses the three interfaces. The same feature works against: - A *local* host (LocalCommandRunner + LocalFileOperator + a null credential). - A *remote* host over SSH (SshCommandRunner + SshFileOperator + SshCredential). - A *containerised* host (DockerCommandRunner + DockerFileOperator + a basic credential). - A *mocked* host in unit tests (FakeCommandRunner + InMemoryFileOperator + FakeCredential). The feature's source code is identical in every case. The substitutability is what makes the cross-OS contract layer in Volume 4 work — the same feature implementation runs against every host because the per-host *machinery* is interface-shaped. ### Composition example: a test runner A test runner consumes: - An `ITestJob` — the job to run. - An `IIncludeFilter` and `IExcludeFilter` — to scope which tests within the job to execute. - An `ISerializable` — to serialise results. The runner's code takes the four interfaces and runs. The same runner works against single-host jobs (`TestJob`), landscape-aware jobs (`InteropLibraryTestJob`), and distributed jobs (`DistributedTestJob`) — because all three implement `ITestJob`. ## What the interfaces do *not* do Three deliberate non-features. ### Not runtime-enforced (typically) The protocols are *type-checker enforced*, not runtime-enforced. A Python program can pass a non-implementer where an `ICommandRunner` is expected; the type checker complains; the runtime doesn't. When *runtime* enforcement is needed, the consuming code does an explicit `isinstance` check (with the protocol marked `@runtime_checkable`) or — more often — does duck-typing checks (`hasattr(runner, 'run')`). This is consistent with Python's structural-typing tradition and with the philosophy's *explicit-is-better-than-implicit* principle. Runtime enforcement of an interface is *additional* rigour that costs runtime cycles; the type checker catches the common case at zero runtime cost. ### Not behaviour, just shape The protocols describe *what methods exist with what signatures*; they do not describe *what those methods do*. A malicious `ICommandRunner` implementation that returns random data instead of running the command satisfies the type contract; the runtime check would still pass; the bug is in the implementer's behaviour. The portfolio mitigates this through testing — each concrete implementation has its own test suite that verifies the *behaviour* matches the documented contract. The interface package doesn't do that work; it shouldn't. ### Not capability discovery The interfaces don't declare what *capabilities* an implementation has beyond the methods it implements. A filesystem implementation that supports symlinks doesn't have a "supports_symlinks" flag on the interface; a runner that streams output doesn't have a "supports_streaming" flag. When a consumer needs capability discovery, the answer is *a separate capability protocol*, not feature flags on a base protocol. This keeps the base protocols small and the discovery logic explicit. ## How interfaces drive the dependency graph The `ccai.interfaces` subpackage, shipped by `codecraftai-foundations`, sits at the bottom of the portfolio's dependency graph. Specifically: ``` codecraftai-foundations (ccai.interfaces) ▲ │ imported by everyone │ ┌────────────────┼─────────────────┐ │ │ │ codecraftai-credentials codecraftai- codecraftai-landscaping │ dataprofiles │ │ │ │ └────────────────┼─────────────────────┘ │ ▼ codecraftai-interop, sysman, ... ``` Two important properties of this graph: - **`ccai.interfaces` depends on nothing.** Not even on `ccai.errors`, its sibling inside the same distribution. `codecraftai-foundations` is the leaf; it can be installed standalone. - **Every package that wants to *interoperate* depends on `codecraftai-foundations`.** The dependency is small (eight small files); the value is large (every package speaks the shared contract language). When a package introduces a new interface, the right place is *here*, not in the introducing package's own surface. This keeps the interfaces *centralised*; consumers only need to import from one place. ## When to add a new interface Three signals that a new interface is the right move: ### Signal 1: two implementations, one consumer When a consuming package has *one shape it wants to consume* and two or more implementing packages produce that shape, the shape is an interface. The interface goes in `ccai.interfaces`; both implementers depend on the interfaces package; the consumer depends only on the interfaces package. ### Signal 2: testing requires a substitute When unit testing a package would benefit from a fake of some external dependency, the dependency's shape is a candidate interface. Testing the sysman feature without spawning real SSH sessions requires a fake `ICommandRunner`; making the fake easy to write requires `ICommandRunner` to be a protocol. ### Signal 3: cross-tier substitutability When a higher-tier package wants to plug a different implementation in for a lower-tier dependency (chapter-7's extension and factory machinery is the substantive form of this pattern), the seam is an interface. The `ccai.interfaces` subpackage deliberately stays *small*. Eight files today; growth is deliberate. Each new interface goes through the question: *do at least two existing or planned implementations want to satisfy this shape?* If yes, add it; if no, don't add the interface yet. ## Summary - **`ccai.interfaces`**, inside `codecraftai-foundations`, is the leaf of the portfolio's dependency graph. Eight protocol files; one purpose: shared contract definitions for substitutable behaviour. - The protocols are **`typing.Protocol`** (structural typing), not `abc.ABC` (nominal typing). Implementers don't inherit; they match by shape. The cost is less runtime introspection; the benefit is decoupling and easy mocking. - The eight protocols: - **`ICommandRunner`** — running commands across local, remote, containerised boundaries. - **`IFileOperator`** — filesystem operations with the same cross-boundary uniformity. - **`ISerializable`** — dict round-trip via `from_dict` and `as_dict`. - **`IBasicCredential`** — username + password read-only properties. - **`IIncludeFilter`** / **`IExcludeFilter`** — uniform list filtering with default implementations of `filter()`. - **`IHint`** — friendly identification. - **`ITestJob`** — the test-job lifecycle contract used by Volume 1 and Volume 2. - The interfaces **compose** into larger contracts. A sysman feature consumes `ICommandRunner` + `IFileOperator` + `IBasicCredential` to operate on a host — local, remote, containerised, or mocked, all with the same code. - The protocols describe **shape, not behaviour**. Behaviour conformance is verified through each concrete implementation's test suite; the interface package itself contains no tests of behaviour. - New interfaces earn their place by satisfying **at least two existing or planned implementations**. The package stays small deliberately; growth is gated. The next chapter introduces the waiting primitives — small helpers for the recurring problem of "wait for a condition" that turn the trivial loop into a debugged, observable, timeout-aware operation. # Chapter 5 — Waiting with Rich Diagnostics > *Every codebase that interacts with anything outside its > process has a `wait_for(...)` function in it somewhere. Most > of them are buggy. This chapter is about the small > investment that makes the function not buggy.* The previous chapter introduced the interfaces. This chapter introduces the smallest substantial subpackage in the foundation tier: [`ccai.waiting`](https://github.com/code-craft-ai-company/codecraftai-foundations/tree/main/source/packages/ccai/waiting), shipped by `codecraftai-foundations`. Three files, two classes, one function. The subpackage's job is to give the rest of the portfolio a *correct* answer for "wait for a condition with a timeout, and tell me what happened if it didn't work." The package's interface is small enough to fit in this paragraph: the `wait_for_it(looper, *args, timeout=N, interval=I, ...)` function calls `looper(wait_context)` repeatedly until either the looper returns `True` (condition met, done) or the timeout elapses (raise `TimeoutError` with rich context). The function is half a page; the *correctness* of the function is the rest of this chapter. ## Why a package for this A naive `wait_for` is half a dozen lines: ```python def wait_for(condition, timeout=60, interval=5): deadline = time.time() + timeout while time.time() < deadline: if condition(): return True time.sleep(interval) return False ``` Every Python codebase has some version of this function. Most of them ship subtle bugs: - The `True/False` return loses every bit of *why* it didn't succeed. The caller has to wrap the call with their own diagnostic. - The final iteration is the *last* one before the deadline; the caller wants to know whether the deadline cut off a *near-miss* or a *not-even-close* — there's no way to tell. - The first iteration runs *immediately*; sometimes the condition needs a brief warm-up, and the first check is wasted. - The interval is constant; sometimes it should be variable (back off on long-running waits). - The condition is called on every loop; logging that the wait is still in progress requires an out-of-band hack. - The timeout's *what for* — what was the wait actually waiting for? — never makes it into the resulting error. Each bug is small. Cumulatively, they make every team's `wait_for` slightly different and slightly worse than it could be. The [`ccai.waiting`](https://github.com/code-craft-ai-company/codecraftai-foundations/tree/main/source/packages/ccai/waiting) subpackage's job is to fix all of them in one place so that every caller benefits. ## The four shipped types The package defines exactly four types in [`waitmodel.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/waitmodel.py): - **`TimeoutContext`** — the state object that tracks a single wait operation. - **`WaitCallback`** — a `Protocol` defining the looper-function signature. - **`WaitContext`** — currently a thin alias for `TimeoutContext`, with the name reserved for future divergence. - **`WaitGate`** — a wrapper around `threading.Event` that adds message and timeout metadata. Plus one context manager: - **`WaitingScope`** — clears a list of gates on entry, waits for them on exit. And one function in [`waitfunction.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/waitfunction.py): - **`wait_for_it(looper, *args, ..., wctx=None)`** — the public entry point. This is the entire shipped API. Five concepts, one function. The smallness is intentional; chapter 1's *if-it's-hard-to- explain-try-another-way* principle would have rejected a larger interface. ## `TimeoutContext` — the state object The [`TimeoutContext`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/waitmodel.py) is the chapter's substantive contribution. It tracks a single wait operation and exposes the operation's progress to both the caller and the looper function. The constructor takes four parameters: ```python def __init__(self, timeout: float, interval: float=0, delay: float=0, doevery: int=-1, what_for: Optional[str]=None): self._timeout = timeout self._interval = interval self._delay = delay self._doevery = doevery self._what_for = what_for self._loop_counter = 0 self._now_time = None self._start_time = None self._end_time = None self._final_attempt = False self._wait_state = TimeoutState.NotStarted return ``` Five timing fields plus internal state. Walk each one. ### `timeout` — the deadline budget The maximum time in seconds the operation may run. After the context's `mark_begin()` call, the deadline is fixed at `start_time + timedelta(seconds=timeout)`. ### `interval` — the gap between checks The seconds to sleep between consecutive looper invocations. Default zero (back-to-back checks). Typical value 1–5 seconds for most operations. ### `delay` — the warm-up The initial seconds to sleep *before* the first looper invocation. Default zero (start immediately). Useful when the system being waited on needs a moment to settle (a service that just started, a database that's still doing initial loads). ### `doevery` — the heartbeat interval The loop-counter interval at which the caller wants a "still waiting" event. Default `-1` (no heartbeat). When set to a positive integer N, the property `is_do_every_interval` returns `True` on every N-th loop iteration; the looper can use this to log a heartbeat without spamming the log on every check: ```python def looper(wctx: TimeoutContext) -> bool: if wctx.is_do_every_interval: logger.info("Still waiting for service to be healthy...") return service.is_healthy() ``` This addresses the "logging on every loop is too noisy" failure mode of the naive `wait_for`. ### `what_for` — the diagnostic string The string description of *what is being waited for*. Carried into any timeout error the context produces, so the failure report names the wait's purpose without the caller having to add it. ```python wctx = TimeoutContext( timeout=60, interval=2, what_for="staging-db to accept connections", ) ``` A timeout from this context produces: ``` Timeout waiting for staging-db to accept connections: timeout=60 start_time=2026-04-30 14:32:11 end_time=2026-04-30 14:33:11 now_time=2026-04-30 14:33:13 time_diff=0:01:02 ``` The error message is *self-documenting*. No log archaeology required. ### State tracking Internal fields track the operation's state through five states (`TimeoutState` enum from [`constants.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/constants.py)): - `NotStarted` — the context exists but `mark_begin()` hasn't been called. - `Tracking` — reserved for future use. - `Running` — the wait is in progress. - `Completed` — the looper returned `True`. - `TimedOut` — the deadline was reached. The state machine is internal but observable through the `completed`, `has_timed_out`, and `wait_state` properties. ## `wait_for_it` — the function The public entry point in [`waitfunction.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/waitfunction.py) is *one function*. Annotated with the chapter's discipline: ```python def wait_for_it(looper: WaitCallback, *largs, what_for: Optional[str]=None, delay: float=DEFAULT_WAIT_DELAY, interval: float=DEFAULT_WAIT_INTERVAL, timeout: float=DEFAULT_WAIT_TIMEOUT, lkwargs: Optional[Dict[Any, Any]]=None, wctx: Optional[WaitContext]=None): if lkwargs is None: lkwargs = {} if what_for is None: what_for = "'{}'".format(looper.__name__) if wctx is None: wctx = WaitContext(timeout, interval=interval, delay=delay, what_for=what_for) if wctx.delay > 0: time.sleep(DEFAULT_WAIT_DELAY) wctx.mark_begin() condition_met = False while True: condition_met = looper(wctx, *largs, **lkwargs) if condition_met: wctx.mark_complete() break if not wctx.should_continue(): break if wctx.interval > 0: time.sleep(wctx.interval) if not condition_met: # Mark the time we are performing the final attempt wctx.mark_time() wctx.mark_final_attempt() condition_met = looper(wctx, *largs, **lkwargs) if not condition_met: toerr = wctx.create_timeout(what_for) raise toerr wctx.mark_complete() return ``` Five things worth naming. ### Defaults are package-level constants `DEFAULT_WAIT_DELAY = 0`, `DEFAULT_WAIT_INTERVAL = 5`, and `DEFAULT_WAIT_TIMEOUT = 60` are defined in [`constants.py`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/constants.py). Keeping the defaults as named constants makes them greppable across the codebase and changeable in one place. ### `what_for` defaults to the looper's `__name__` When the caller doesn't provide a `what_for`, the function uses the looper function's name. This is the *graceful fallback* discipline — when context isn't provided, infer something reasonable rather than leave the failure report empty. ### The looper's first invocation precedes `should_continue` The function's loop runs the looper *first*, then checks whether to continue. This means: the first iteration always runs, even if `timeout=0` was passed. A caller who genuinely wants "check once and timeout immediately" gets the check. ### The "final attempt" pattern This is the chapter's most subtle correctness contribution. When the loop's main pass exhausts the timeout, the function does *one more invocation* of the looper with `wctx.final_attempt = True`. The looper can detect the final attempt and produce a *final diagnostic snapshot* — log the system state, dump relevant data, capture a screenshot — even though the wait is about to fail. Without this pattern, the timeout error fires at the end of the last sleep, with no information about *what state the system was actually in* at the moment of failure. With the pattern, the final invocation is a deliberate "last chance to capture the failure context" call. ```python def wait_for_service_healthy(wctx: TimeoutContext, service) -> bool: is_healthy = service.is_healthy() if not is_healthy and wctx.final_attempt: logger.error("Final state: %s", service.dump_diagnostic_state()) return is_healthy ``` The looper logs the diagnostic state *only* on the final attempt — every intermediate failure is silent (correct for "still waiting"), and the final failure is loud (correct for "timed out, here's why"). ### `create_timeout` produces a rich `TimeoutError` When the function gives up, it asks the context to construct the error: ```python toerr = wctx.create_timeout(what_for) raise toerr ``` The `create_timeout` method (in `TimeoutContext`) builds the diagnostic message from the timing fields plus the `what_for` description. The chapter-3 timeout-error vocabulary (`CompletionTimeout`, `ConnectionTimeout`, `InActivityTimeout`) applies — callers who want a *specific* timeout type wrap the call: ```python try: wait_for_it(check_db_ready, timeout=60, what_for="staging-db ready") except TimeoutError as err: raise ConnectionTimeout(str(err)) from err ``` The `TimeoutError` from the package becomes a `ConnectionTimeout` at the wrapping layer. The original message is preserved; the type changes for the caller's discrimination. ## `WaitGate` and `WaitingScope` — the threading primitives Two additional types address the *cross-thread waiting* case. ### `WaitGate` — a threading.Event with metadata [`WaitGate`](https://github.com/code-craft-ai-company/codecraftai-foundations/blob/main/source/packages/ccai/waiting/waitmodel.py) wraps a `threading.Event` and adds three pieces of metadata: ```python class WaitGate: def __init__(self, gate: threading.Event, message: Optional[str]=None, timeout: Optional[float]=None, timeout_args: Optional[list]=None): self._gate = gate self._message = message self._timeout = timeout self._timeout_args = timeout_args ``` The `message` is the timeout diagnostic the gate carries; the `timeout` is the default wait duration; `timeout_args` are formatting arguments for the message. The gate exposes the underlying event's `set()`, `clear()`, `is_set()`, and `wait()` methods, plus the `raise_timeout` flag that converts a wait failure into a `TimeoutError` directly. ### `WaitingScope` — coordinated multi-gate waits ```python class WaitingScope: def __init__(self, gates: List[WaitGate]): self._gates = gates def __enter__(self): for gate in self._gates: gate.clear() def __exit__(self, ex_type, ex_inst, ex_tb): return def wait(self): for gate in self._gates: gate.wait() ``` The pattern: enter the scope to *clear* all gates, do work that's expected to set them, then call `wait()` to *wait for all of them*. This is the cross-thread version of `wait_for_it` — the looper polls a single condition; the scope waits on multiple events that are set by other threads. A typical usage: ```python service_ready = WaitGate(threading.Event(), timeout=30, message="service ready") config_loaded = WaitGate(threading.Event(), timeout=10, message="config loaded") with WaitingScope([service_ready, config_loaded]): start_service() # this thread sets service_ready load_config() # this thread sets config_loaded # Outside the scope, both gates are set ``` The `WaitingScope` is a small abstraction. Most callers don't need it. When the orchestration is genuinely multi-threaded and the synchronisation is *gate-based*, the scope is the right primitive. ## Three patterns the package enables Three usage patterns recur across the portfolio. ### Pattern 1: simple wait for a condition ```python from ccai.waiting.waitfunction import wait_for_it def is_db_accepting_connections(wctx, host, port): try: with socket.create_connection((host, port), timeout=2): return True except (ConnectionRefusedError, socket.timeout): return False wait_for_it( is_db_accepting_connections, "staging-db.internal", 5432, what_for="staging-db accepting connections", timeout=60, interval=2, ) ``` Six lines. The looper takes the wait context plus any positional args and returns True/False. The function handles the loop, the timeout, the diagnostic. ### Pattern 2: wait with heartbeat logging ```python def is_long_job_done(wctx, job): if wctx.is_do_every_interval: logger.info("Job %s still running (%d/%d sec elapsed)", job.id, wctx.timeout - (wctx.end_time - datetime.now()).seconds, wctx.timeout) return job.is_done() wait_for_it( is_long_job_done, job=long_running_job, timeout=3600, interval=10, ) ``` The looper checks `is_do_every_interval` to throttle log output. With `interval=10`, the heartbeat fires every 60 seconds (every sixth check); without this pattern, every check would log, producing noise. To set up `doevery=6`, the caller passes a custom `TimeoutContext`: ```python wctx = TimeoutContext(timeout=3600, interval=10, doevery=6, what_for="long job") wait_for_it(is_long_job_done, wctx=wctx, lkwargs={"job": long_running_job}) ``` ### Pattern 3: wait with final-attempt diagnostics ```python def is_test_environment_healthy(wctx, env): healthy = env.health_check() if not healthy and wctx.final_attempt: # Last chance — capture full diagnostic for the failure env.dump_state(path="./failure-diag.txt") env.capture_logs(path="./failure-logs/") return healthy wait_for_it( is_test_environment_healthy, env=staging_environment, what_for="staging environment healthy", timeout=300, ) ``` The diagnostic dump runs *only* on the final attempt — once, right at the moment of failure, with the full system state available. Without this pattern, the engineer debugging the timeout has to *re-run* the test to capture the state, which often produces a successful run because the flake doesn't reproduce. ## What the package does *not* do Three deliberate non-features. ### Not async/await The shipped `wait_for_it` is a synchronous function. It uses `time.sleep` between checks and blocks the calling thread. When the calling code is `async`, a different primitive is needed — typically `asyncio.wait_for` or a custom looper — and the shipped package doesn't provide one. This is a deliberate scoping choice. An async equivalent could be added; the discipline is to wait until two callers actually need it before adding the parallel implementation. Adding it preemptively would double the surface area for hypothetical benefit. ### Not retries with backoff The `interval` parameter is constant. When the caller wants exponential backoff (interval doubles after each failed check), the looper itself implements it — typically by tracking attempts in a closure or in `wctx`'s loop counter. The shipped package doesn't bake in a backoff curve because no single curve fits every use case. Linear, exponential, capped- exponential, jittered — each is a few lines of looper code. The discipline keeps the chapter-2 *no-clever-defaults* rule. ### Not progress estimation The `wait_for_it` function knows when the deadline is, but it doesn't know the *progress* of the underlying operation. When the caller wants "60% done" rather than "44 of 60 seconds elapsed," that's a property of the operation, not of the wait. The package leaves progress estimation to the operation — the looper can call into the operation and compute progress, then include it in the heartbeat log. The wait function stays neutral about progress; it only knows about time. ## Cross-references The waiting primitives are used across every higher tier of the portfolio: - **Volume 0 chapter 9 (Configuration loading)** uses `wait_for_it` for the HTTP source's connection retries. - **Volume 3** uses the package implicitly through tasks that block on subprocess completion; the chapter-19 `RunCommand` task wraps `subprocess.communicate()` which is its own wait primitive. - **Volume 5 (Interop)** uses `wait_for_it` extensively for SSH session establishment, DNS lookups, and similar network-bound waits. - **Volume 6 (Landscape)** uses `wait_for_it` for "wait for the device to come online" patterns during landscape activation. The `Aspects` pattern from Volume 5 chapter 1 is the affirmative form of *behaviour parameterisation* the chapter-2 decorator discipline pointed at. Aspects' `timeout`, `interval`, and `retry_count` fields are exactly the fields the wait function expects — the two layers compose cleanly. ## Summary - **`ccai.waiting`** is a small subpackage of `codecraftai-foundations` with one job: give the rest of the portfolio a correct, observable, timeout- aware *wait for a condition* primitive. - The shipped API is **five concepts and one function**: `TimeoutContext`, `WaitCallback`, `WaitContext`, `WaitGate`, `WaitingScope`, plus `wait_for_it`. - `TimeoutContext` carries five timing fields (`timeout`, `interval`, `delay`, `doevery`, `what_for`) plus internal state. The `what_for` field is what makes timeout errors self-documenting. - The **final-attempt pattern** is the chapter's most subtle correctness contribution. After the timeout exhausts, one more looper invocation with `wctx.final_attempt = True` lets the looper capture diagnostic state at the moment of failure. - `wait_for_it` is the public entry point. Defaults to `DEFAULT_WAIT_TIMEOUT=60`, `DEFAULT_WAIT_INTERVAL=5`, `DEFAULT_WAIT_DELAY=0`. Returns nothing on success; raises `TimeoutError` with rich context on failure. - `WaitGate` and `WaitingScope` provide the cross-thread variant — gates are `threading.Event` wrappers with timeout metadata; the scope clears gates on entry and waits for them on exit. - Three patterns recur: **simple condition wait**, **wait with heartbeat logging** (via `doevery` + `is_do_every_interval`), **wait with final-attempt diagnostics** (via `wctx.final_attempt`). - The package is *not* async/await aware, *not* a retry-with- backoff library, and *not* a progress estimator. Each scoping choice is deliberate. - Higher tiers (orchestration, interop, landscape, sysman) all use the wait function for time-bounded operations. The next chapter introduces application identity and presence — the `ThisApplication` machinery that gives every running process in the portfolio a structured identity, with the persona-variable descriptors that make configuration overrides traceable. # Chapter 6 — Application Identity and Presence > *Every running process in the portfolio answers two questions > the same way: "Who am I?" and "Where do my files live?" This > chapter is about the small machinery that makes the answers > consistent.* The previous chapter introduced the waiting primitives. This chapter introduces the *identity* primitives — [`codecraftai-this-app`](https://github.com/code-craft-ai-company/codecraftai-this-app/tree/main/source/packages/ccai/thisapp), the package whose `ThisApplication` singleton represents the running process to itself. Every higher tier of the portfolio calls into `ThisApplication` to resolve configuration variables, locate well-known directories, and emit identity metadata into logs and traces. The package began small but has grown consequential. At its core is one singleton; one function (`establish_app_persona`) that *every* application calls early in startup; one global context that carries identity values across the process. As of the runtime-foundation consolidation it *also* owns the machinery that used to live in the former `codecraftai-runtime` package: the `RuntimeVariables` descriptor set, the well-known-paths helpers, the profile/job enumerations, and the activation sequence. The old `codecraftai-runtime` is being retired — it now holds only thin re-export shims, and import paths shift from `ccai.runtime.X` to `ccai.thisapp.X`. This consolidation is what lets `codecraftai-testbase` sit *above* the identity-and-presence tier without depending on it: testbase reads the handful of core environment knobs it needs directly, and the richer runtime machinery described later in this chapter is what an *integrated* run (Volume 1) layers back on top via `codecraftai-integration-runtime`. ## The two-phase pattern The identity machinery has a deliberate two-phase startup: 1. **`establish_app_persona(...)`** — called once, before any other ccai machinery. This sets the application's *persona*: name, logger name, log file basename, presence directory (where the app's files live on disk), settings file path. After this call, identity is fixed for the process's lifetime. 2. **`ThisApplicationSingleton()`** — called any number of times afterward. Returns the singleton `ThisApplication` instance. The first call constructs the instance from the persona set in phase 1. The split is intentional. Phase 1 is *configuration*; phase 2 is *use*. The singleton refuses to construct itself if phase 1 hasn't run — it raises `SemanticError` (chapter 3's must-bubble type) so the failure is loud and the caller is told exactly what's wrong: ```python errmsg = "You must call 'establish_app_persona' early in your application startup to define " \ "your application's presence settings before attempting to use the `ThisApplication` " \ "object or its corresponding singleton." raise SemanticError(errmsg) ``` This is the *errors-never-silent* discipline applied to initialisation order. An application that forgets to establish its persona doesn't get a `None` back — it gets a clear `SemanticError` telling it to fix the startup order. ## `establish_app_persona` — the canonical entry point The function lives in [`persona.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/persona.py). The signature: ```python def establish_app_persona(*, name: str, logger_name: str, log_basename: str, presence_dir: Optional[str]=None, log_directory: Optional[str]=None, settings_file: Optional[str]=None, extensions: Optional[str]=None, service_name: Optional[str]=None, **other): ``` Three required parameters, four optional. Every parameter is keyword-only (the leading `*`); this is the chapter-2 named- parameter discipline. Walk each parameter. ### `name` — the application's identity The application's name in machine-readable form. Lowercase with hyphens for spaces. Examples: `"codecraftai-testbase"`, `"my-deployment-tool"`, `"nightly-runner"`. Used to derive default paths and to identify the process in logs. ### `logger_name` — the root logger name The Python logger name the application uses as its root. When the application emits logs through `logging.getLogger("my-app")`, `logger_name="my-app"`. Subsystems use child loggers (`logging.getLogger("my-app.subsystem")`); the root name lets the logging configuration apply formatters and handlers hierarchically. ### `log_basename` — the log file's base name The base name of the application's log files. When the app writes `my-app-2026-04-30.log` and `my-app-2026-04-30-debug.log`, `log_basename="my-app"`. The chapter-9 configuration loader uses this to wire up the log file paths. ### `presence_dir` — where the app's files live The directory where the application's files live on disk. When unspecified, defaults to `os.path.join("~", name.lower())` — i.e., `~/my-deployment-tool/` for an app named `"my-deployment-tool"`. The presence directory is the *home base* for the application's runtime artifacts: settings files, log files, captured profiles, output directories, lock files. Code that needs a writable location asks `ThisApplication.presence_dir` and gets a predictable path. ### `log_directory` — where logs go Defaults to `os.path.join(presence_dir, "logs")` when unspecified. Override when logs need to live somewhere other than the presence directory (a system log path, a separate mounted volume). ### `settings_file` — the canonical settings file Defaults to `os.path.join(presence_dir, "settings.config")`. The function has a small convenience: if `settings_file` is a bare filename (no path component), it's resolved relative to the presence directory. When a full path is given, it's expanded through `os.path.expandvars` and `os.path.expanduser`. ### `extensions` — extension module list A semicolon-separated string of extension module names. Used by the chapter-7 extension/factory machinery to discover plug-ins. ### `service_name` — the service identity When the application runs as a system service, the service name. Used by logging and observability to tag events with the service identity. ## What `establish_app_persona` does at runtime The function does three things: 1. **Validates** — confirms the singleton hasn't already been constructed. If it has, raises `SemanticError` (you can't re-establish a persona). 2. **Stores the persona** — copies the parameters onto module- level `AppPersona` class attributes. 3. **Returns** — the function returns `None` (per chapter-2 discipline, with explicit `return`). The actual `ThisApplication` instance isn't constructed yet — that happens lazily on the first `ThisApplicationSingleton()` call. The two-phase pattern keeps startup responsive: an application that calls `establish_app_persona` early but doesn't actually need the singleton until later doesn't pay the construction cost upfront. ## `ThisApplication` — the singleton's surface The [`ThisApplication`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/thisapplication.py) class exposes seven properties (read-only views of the persona fields) and four methods (the *active* surface for variable resolution and overrides): ```python class ThisApplication: # Properties (read-only) @property def name(self) -> str: ... @property def logger_name(self) -> str: ... @property def log_basename(self) -> str: ... @property def log_directory(self) -> str: ... @property def presence_dir(self) -> str: ... @property def settings_file(self) -> str: ... @property def service_name(self) -> str: ... # Methods def context_insert(self, path: str, obj: Any) -> None: ... def context_lookup(self, path: str, default=None, raise_error=False) -> Any: ... def get_variable_value(self, variable_desc: AppVarDesc, ...) -> Union[str, Any]: ... def override_variable_value(self, variable_desc: AppVarDesc, new_value: Any) -> None: ... def resolve_variable_set(self, variable_set: AppVarSet) -> None: ... ``` Each property is a *simple data accessor* — chapter 2's property-as-pure-data discipline. Each method does *one thing* clearly named. ## The variable-descriptor model The richest subsystem in `codecraftai-this-app` is the *variable-descriptor* model. It's the formal mechanism for declaring which configuration values a subsystem cares about, where each value can come from, and how it's converted to a typed value. Three classes participate: ### `AppVarDesc` — the descriptor A descriptor declares *one variable*: ```python AM_LOG_BASENAME = AppVarDesc( source=VariableSource.NONE, ) AM_PRESENCE_DIR = AppVarDesc( source=VariableSource.NONE, context_path=ContextPaths.PRESENCE_HOME_DIRECTORY, ) AM_INTERACTIVE_CONSOLE = AppVarDesc( default=False, converter=parse_bool, ) ``` Each descriptor carries: - **`source`** — where the variable's value originates. Common values: `VariableSource.ENVIRONMENT` (read from `os.environ`), `VariableSource.SETTINGS` (read from the app's settings file), `VariableSource.NONE` (resolved programmatically). - **`default`** — the value to use when the source doesn't provide one. - **`converter`** — a function that takes the raw string and returns the typed value. The chapter-9 loader's `parse_bool`, `split_and_normalize_name_list`, etc., are the shipped converters. - **`context_path`** — when the variable is also published to the shared application context, the well-known path it publishes to. This is the `ContextPaths` enum we'll meet shortly. The descriptor is *just data* — no behaviour, no resolution logic. The behaviour lives on `ThisApplication`. ### `AppVarSet` — a group of descriptors A set of related descriptors live in a class that subclasses `AppVarSet`: ```python class PersonaVariablesDesc(AppVarSet): AM_NAME = AppVarDesc(source=VariableSource.NONE, context_path=ContextPaths.PRESENCE_NAME) AM_LOGGER_NAME = AppVarDesc(source=VariableSource.NONE) AM_LOG_BASENAME = AppVarDesc(source=VariableSource.NONE) AM_PRESENCE_DIR = AppVarDesc(source=VariableSource.NONE, context_path=ContextPaths.PRESENCE_HOME_DIRECTORY) AM_LOG_DIRECTORY = AppVarDesc(source=VariableSource.NONE) AM_SETTINGS_FILENAME = AppVarDesc(source=VariableSource.NONE) AM_EXTENSIONS = AppVarDesc(source=VariableSource.NONE, context_path=ContextPaths.PRESENCE_EXTENSIONS) AM_SERVICE_NAME = AppVarDesc(source=VariableSource.NONE) ``` Eight related descriptors, one class. The class is *iterable* — `AppVarSet.items()` yields the descriptors in declaration order — so resolution code can walk the whole set without naming each descriptor individually. The volume's other variable sets follow the same pattern. `ConfigVariablesDesc` (in `codecraftai-configuration`) declares the configuration-related descriptors. `RuntimeVariablesDesc` — formerly in `codecraftai-runtime`, now in [`ccai.thisapp.runtimevariables`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/runtimevariables.py) — declares the runtime-related descriptors and itself subclasses `PersonaVariablesDesc`. Each subsystem owns its own variable set; the sets compose at the `ThisApplication` level. ### `AppVarAccessor` — typed read-through For convenient access at the call site, a parallel class provides typed read-through accessors: ```python class PersonaVariables: AM_NAME = AppVarAccessor(PersonaVariablesDesc.AM_NAME) AM_LOGGER_NAME = AppVarAccessor(PersonaVariablesDesc.AM_LOGGER_NAME) AM_LOG_BASENAME = AppVarAccessor(PersonaVariablesDesc.AM_LOG_BASENAME) # ... ``` A consumer calls `PersonaVariables.AM_NAME` and gets the resolved value. The accessor object internally calls `ThisApplicationSingleton().get_variable_value(descriptor)`. The two-class pattern (Desc + Accessor) is the chapter's ergonomic concession: `Desc.AM_NAME` is the *descriptor* (used when calling `get_variable_value`); `Accessor.AM_NAME` is the *resolved value* (used at the call site). ## Variable resolution The active surface is `ThisApplication.get_variable_value(desc)`. The resolution order: 1. **Override** — has the variable been overridden via `override_variable_value`? If yes, return the override. 2. **Environment** — does the variable's source include environment variables, and is one set? If yes, return the typed-converted environment value. 3. **Settings** — does the variable's source include the settings file, and is the variable present in it? If yes, return the typed-converted settings value. 4. **Default** — is there an explicit default on the descriptor? If yes, return it. 5. **Else** — return `None` (or the `default` parameter if the caller provided one). The order is deterministic and inspectable — given a descriptor, you can predict what value it'll resolve to without running the code. This is the chapter-1 *explicit-is-better-than-implicit* discipline applied to configuration resolution. ## Override semantics `override_variable_value(desc, new_value)` is the runtime override mechanism. It writes the new value into an internal override map; the next `get_variable_value(desc)` returns it ahead of every other source. Concretely, the implementation does *more* than just store the value: - For descriptors with `VariableSource.ENVIRONMENT` source, the override *also* writes to `os.environ` so child processes inherit the new value. This is the chapter-21 (Volume 3) bootstrap-name-flow mechanism in action — the orchestration's override of `AM_LANDSCAPE_NAMES` reaches subprocess jobs through the environment variable. - For *some* descriptors, the override *also* hot-swaps a matching field on `ThisApplication`. Specifically: `AM_SERVICE_NAME` updates `_service_name`; `AM_LOG_BASENAME` updates `_log_basename`; `AM_LOG_DIRECTORY` updates `_log_directory`; `AM_LOGGER_NAME` updates `_logger_name`. This was the in-flight refactor flagged during the portfolio hygiene sweep in the prior session. The pattern: the override is *immediate* in this process and *propagated* to subprocess processes through `os.environ`. Two levels; same value; one entry point. ## The shared context Beyond variable resolution, `ThisApplication` carries a *context* — a hierarchical key-value store accessible by *path strings*: ```python ctx = ThisApplicationSingleton().context ctx.insert("/environment/landscape/names", ["staging-east"]) landscape_names = ctx.lookup("/environment/landscape/names") ``` The context is shared across the process; it's the *serializable* identity of the running application. Subsystems write configuration data into it; consumers read it out. The well-known paths live in [`ContextPaths`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/contextpaths.py) — a string enum: ```python class ContextPaths(str, Enum): BUILD_RELEASE = "/environment/build/release" BUILD_BRANCH = "/environment/build/branch" BUILD_NAME = "/environment/build/name" CONFIG_LANDSCAPE_NAMES = "/environment/landscape/names" CONFIG_LANDSCAPE_URIS = "/environment/landscape/uris" CONFIG_LANDSCAPE_FILES = "/environment/landscape/files" CONFIG_RUNTIME_NAMES = "/environment/runtime/names" CONFIG_RUNTIME_URIS = "/environment/runtime/uris" CONFIG_TOPOLOGY_NAMES = "/environment/topology/names" # ... JOB_ID = "/environment/job/id" JOB_NAME = "/environment/job/name" JOB_LABEL = "/environment/job/label" JOB_SEED = "/environment/job/seed" # ... ``` Each enum entry is *both a name and a path string*. Code that needs to publish or consume a value uses the enum: ```python ctx.insert(ContextPaths.JOB_ID, run_uuid) job_id = ctx.lookup(ContextPaths.JOB_ID) ``` The string-enum dual-citizenship makes the path *greppable* and *rename-safe*. When a path moves (rare), the enum value changes in one place and every consumer that uses the enum is automatically updated. ## The variable-descriptor + context-path composition A variable descriptor with a `context_path` field automatically publishes its resolved value to the context at that path: ```python AM_PRESENCE_DIR = AppVarDesc( source=VariableSource.NONE, context_path=ContextPaths.PRESENCE_HOME_DIRECTORY, ) ``` When the variable is resolved (at startup, through `resolve_variable_set`), its value is *also* written to `/presence/home-directory` in the context. The descriptor is the *source of truth*; the context path is the *publication endpoint*. This composition is what enables Volume 3's bootstrap-name flow to work cleanly. The orchestration sets `AM_LANDSCAPE_NAMES`; the variable resolution writes the value into both `os.environ` and the context at `/environment/landscape/names`; subsystems that need to *find* the active landscape names read the context path; subsystems that need to *override* call `override_variable_value` and trust the propagation. ## The absorbed runtime foundation Everything above describes the *persona* machinery — the half of `codecraftai-this-app` that was always here. The other half arrived with the runtime-foundation consolidation: the modules that used to constitute the standalone `codecraftai-runtime` package now live under `ccai.thisapp`, and the application-identity-and-presence tier is their home. They build directly on the descriptor model this chapter just introduced. > **As-built note.** The old `codecraftai-runtime` package still > exists, but only as a set of thin re-export shims so that > in-flight callers importing `ccai.runtime.X` keep working > while they migrate to `ccai.thisapp.X`. New code imports from > `ccai.thisapp` directly. ### `RuntimeVariablesDesc` / `RuntimeVariables` — the runtime descriptor set [`runtimevariables.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/runtimevariables.py) declares `RuntimeVariablesDesc`, which *subclasses* `PersonaVariablesDesc` — so a single descriptor set carries both the persona fields and the runtime fields. It is a large set: build metadata (`AM_BUILD_RELEASE`, `AM_BUILD_BRANCH`, `AM_BUILD_NAME`, `AM_BUILD_URL`), pipeline identity (`AM_PIPELINE_ID`, `AM_PIPELINE_NAME`, `AM_PIPELINE_INSTANCE`), job identity (`AM_JOB_ID`, `AM_JOB_NAME`, `AM_JOB_TYPE`, `AM_JOB_SEED`, …), diagnostics knobs, logging levels, the output directory, the test-result path templates, and more. Each descriptor follows the same `AppVarDesc` contract — `default`, `converter`, `context_path` — so every runtime value resolves and publishes through the exact pipeline the persona variables use. The companion accessor class `RuntimeVariables` (subclass of `PersonaVariables`) exposes the typed read-through views. Call-site code reads `RuntimeVariables.AM_JOB_TYPE` and gets the resolved, typed value; resolution code passes `RuntimeVariablesDesc.AM_JOB_TYPE` to `get_variable_value` / `override_variable_value`. A small free function, `resolve_runtime_variables()`, performs the one-time resolution of the whole runtime set: it coerces the start-time and shared-output-directory values, then calls `this_app.resolve_variable_set(RuntimeVariablesDesc)` so every descriptor with a `context_path` publishes into the shared context. It guards against double resolution with a module-level `RUNTIME_VARIABLES_RESOLVED` flag. ### Well-known paths [`paths.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/paths.py) turns the resolved runtime variables into concrete on-disk locations. The shipped helpers include: - `get_presence_directory()` — the application's home base, derived from `RuntimeVariables.AM_PRESENCE_DIR`. - `get_path_for_output()` — the timestamped directory where results and artifacts are deposited. - `get_path_for_testresults()` — the testresults directory. - `get_path_for_testcase_by_products(test_id)` — a per-test-case by-products folder under the testresults directory. - `get_path_for_diagnostics(label)` and `get_path_for_artifacts(label)` — labelled subfolders of the output directory. - `get_summary_html_template_source()`, `get_summary_static_resource_dest_dir()`, `get_summary_static_resource_src_dir()` — the HTML-summary template and static-resource locations. Several of these are *activation-gated*: helpers like `get_path_for_diagnostics`, `get_path_for_testresults`, and `get_path_for_testcase_by_products` raise `SemanticError` when `RuntimeVariables.AM_JOB_TYPE == JobType.Unknown` — i.e. when the process was *not* activated as a job. This is the same errors-never-silent discipline applied to the directory layout: a plain, non-activated process is told plainly that the presence-folder results tree does not exist for it. > **Consequence for testbase.** Plain `codecraftai-testbase` runs > *without* activation — it records to stdout, has no presence > folder, and never touches `RuntimeVariables` or this-app. The > presence-folder results layout, JSOS-to-disk streaming, HTML > summary, and CI/build metadata are all the responsibility of > `IntegratedTestJob` in `codecraftai-integration-runtime`, which > activates the process as a job and *then* delegates to these > path helpers. ### Profile and job enumerations [`enumerations.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/enumerations.py) defines two string enums that govern *how* a process is activated: ```python class ActivationProfile(str, Enum): Command = "command" Console = "console" Job = "job" Service = "service" class JobType(str, Enum): Unknown = "Unknown" Build = "Build" Infrastructure = "Infrastructure" Orchestration = "Orchestration" TestRun = "TestRun" ``` `ActivationProfile` selects the *shape* of the process — a one-shot command, an interactive console, a long-running service, or a job that collects by-products into an output folder. `JobType` sub-classifies a job and drives which results subtree the activation sequence builds (`Build`, `Infrastructure`, `Orchestration`, `TestRun`). ### The activation sequence [`activation.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/activation.py) ties the enumerations, the runtime variables, and the well-known paths together. It exposes one function per profile — `activate_profile_command()`, `activate_profile_console()`, `activate_profile_service(...)`, and `activate_profile_job(job_type)` — each of which: 1. **Guards against double activation** — if `RuntimeVariables.AM_ACTIVATION_PROFILE` is already set, it refuses (a `SemanticError` for most profiles), because a process has exactly one persona-shape. 2. **Records the profile** — overrides `RuntimeVariablesDesc.AM_ACTIVATION_PROFILE` with the chosen `ActivationProfile`. 3. **Runs the common activation** — `_activate_profile_common` resolves the start-time-stamped output directory appropriate to the `JobType`, assigns a job id if one was not supplied, and overrides `AM_OUTPUT_DIRECTORY` accordingly. 4. **Activates logging** — wires up file, console, or rotating service logging through the `ccai.xmodules.xlogging` helpers. `activate_profile_job(JobType.TestRun)` is the entry point an integrated test run uses. In `codecraftai-integration-runtime`, the activation init (`establish_app_persona` → `resolve_runtime_variables` → `activate_profile_job(JobType.TestRun)`) is what populates the presence-folder output context and the `RuntimeVariables`-sourced run metadata that `IntegratedTestJob` records against. This chapter's tier supplies the mechanism; that package supplies the policy of *when* to invoke it. ## Cross-references The identity machinery is used by every higher tier: - **Volume 0 chapter 9 (Configuration loading)** uses `AppVarDesc` and `AppVarAccessor` as the typed descriptor for every configuration variable; `ConfigVariablesDesc` extends `PersonaVariablesDesc`. - **Volume 0 chapter 8 (Runtime variables and well-known paths)** goes deeper on the runtime-related variables (`AM_RUN_ID`, `AM_INTERACTIVE_CONSOLE`, etc.) and the activation sequence — all of which, per the consolidation, now live in `ccai.thisapp` alongside the persona machinery rather than in a separate `codecraftai-runtime` package. - **Volume 3 chapter 14 (Configuration profiles as a discipline)** proposes the `ContextPaths.ACTIVE_PROFILE` addition. The chapter is using the same machinery this chapter introduces. - **Volume 3 chapter 16 (Reproducing CI failures locally)** captures the resolved variables — the snapshot is a JSON serialisation of `AppVarSet.items()` results across all variable sets. The pattern is consistent throughout the portfolio: identity is managed by `ThisApplication`; values are typed through descriptors; publication is through well-known context paths; overrides are explicit and traceable. ## What this machinery is *not* Three deliberate non-features. ### Not a generic dependency-injection container `ThisApplication` carries identity and configuration variables. It does *not* carry arbitrary application objects. When the application has a database connection, a cache client, an HTTP session — those live in their own singletons or are passed explicitly to functions. The identity machinery is not the right home for them. ### Not multi-application The `ThisApplication` singleton is per-process. When two applications share a process (test harnesses, plugin ecosystems), only one persona can be established. When isolation between two applications matters, they need separate processes. This is consistent with the philosophy's *simple-is-better-than- complex* principle. A multi-application machinery would be heavier; the per-process model is sufficient for almost every use case. ### Not stateful across restarts The `ThisApplication` state lives in process memory. When the process restarts, the singleton is reconstructed from the persona — but the override map and the runtime context are fresh. When state needs to *persist*, the application writes it to the settings file or to disk. The chapter-16 capture-and-replay pattern is the explicit mechanism for cross-process state recovery. The discipline does not pretend that runtime overrides survive process boundaries without that machinery. ## Summary - **`codecraftai-this-app`** provides the identity and presence machinery every running process in the portfolio uses. Two-phase startup: `establish_app_persona(...)` first (sets identity), then `ThisApplicationSingleton()` (gets the configured singleton). Calling the singleton without establishing the persona raises `SemanticError`. - The persona has **seven fields**: `name`, `logger_name`, `log_basename`, `presence_dir`, `log_directory`, `settings_file`, `extensions`, `service_name`. Three are required; the rest have sensible defaults derived from `name`. - **`ThisApplication`** exposes seven read-only properties (the persona fields) plus four methods (`context_insert`, `context_lookup`, `get_variable_value`, `override_variable_value`, `resolve_variable_set`). - The **variable-descriptor model** has three classes: `AppVarDesc` (the data), `AppVarSet` (a group of related descriptors), `AppVarAccessor` (typed read-through at the call site). Each subsystem ships its own descriptor set. - **Variable resolution** has a deterministic order: override → environment → settings → default. Inspectable; predictable. - **Override propagation**: `override_variable_value` writes to the override map *and* to `os.environ` (for subprocess inheritance) *and* hot-swaps matching fields on `ThisApplication` for a small set of identity-critical variables. - The **shared context** is a path-keyed key-value store accessible through `ThisApplication.context`. Well-known paths live in `ContextPaths`, a string enum that's both a name and a path string for grep-and-rename safety. - **Variable descriptors with `context_path`** automatically publish their resolved values to the context, making the resolution pipeline observable end-to-end. - The tier now also **owns the absorbed runtime foundation** (moved here from the retired `codecraftai-runtime`): `RuntimeVariablesDesc` / `RuntimeVariables` and `resolve_runtime_variables` (runtime descriptor set), the well-known-paths helpers in `paths.py` (several activation-gated via `JobType.Unknown`), the `ActivationProfile` / `JobType` enumerations, and the `activate_profile_*` sequence in `activation.py`. Plain testbase runs *without* this layer; `codecraftai-integration-runtime` layers it back on for integrated runs. - The machinery is *not* a generic dependency-injection container, *not* multi-application, *not* persistent across restarts. Each scoping choice is deliberate. The next chapter introduces the extension and factory machinery — the precedence-ordered factory model that lets the portfolio plug new behaviour into existing extension points without modifying the consumers of those extensions. # Chapter 7 — Extension and Factory Machinery > *Some packages need to plug new behaviour into existing extension > points without modifying the consumers of those extensions. > The factory machinery in `codecraftai-extension` is the formal > mechanism that makes this possible — declaratively, with > precedence ordering, with substitutability.* This chapter introduces [`codecraftai-extension`](https://github.com/code-craft-ai-company/codecraftai-extension/tree/main/source/packages/ccai/extension), the foundation-tier package whose job is to provide the plug-in machinery the rest of the portfolio uses. The package defines three kinds of artifact: - **`FactoryProtocolBase`** — the base of every factory protocol; declares what kind of factory it is. - **`FactoryExtension`** — the base of every concrete factory; registers itself by precedence. - **`SuperFactory`** — the engine that walks discovered factories and produces instances by protocol. The pattern is *not* the GoF Factory pattern in textbook form. It's a *registration-based, precedence-ordered, multi-source* factory model that deals with the realities of an `codecraftai-*`-style portfolio: many packages, each contributing factories that override or supplement the base. ## The problem the machinery solves Consider a sysman feature that needs to *create a network adapter representation* on a per-OS basis. The base sysman package defines a generic `NetworkAdapter` class. RHEL needs a `RhelNetworkAdapter` with NetworkManager integration. Ubuntu needs an `UbuntuNetworkAdapter` with netplan handling. macOS needs a `MacOSNetworkAdapter` with `networksetup` shelling. Windows needs PowerShell-driven implementations. UEFI doesn't need network adapters at all. The naive solution: a big switch statement keyed on the host OS. A `if os == "rhel": return RhelNetworkAdapter()` chain. This works for one extension point and grows untenable across a portfolio. Every new OS adds branches; every new extension point duplicates the switch. The `codecraftai-extension` solution: each per-OS package *registers* a factory that produces its OS-specific implementation. The `SuperFactory` discovers all registered factories at startup, sorts them by precedence, and dispatches calls. Adding a new OS is a *new package* with a registered factory; the consumer code doesn't change. ## The three artifact types ### `FactoryProtocolBase` — the contract for protocols [`factoryprotocolbase.py`](https://github.com/code-craft-ai-company/codecraftai-extension/blob/main/source/packages/ccai/extension/factory/factoryprotocolbase.py) defines the marker base for any factory protocol declaration. In its entirety: ```python class FactoryProtocolBase(Protocol): factory_protocol_name = None def __init_subclass__(cls, *args, **kwargs): if cls.factory_protocol_name is None: errmsg = ( "The 'factory_protocol_name' class level field must be set on derived protocol types." ) raise RuntimeError(errmsg) return super().__init_subclass__(*args, **kwargs) ``` Two things to note: - The base is a `typing.Protocol`, *not* an abstract base class. Implementers don't inherit from `FactoryProtocolBase` directly; they inherit from a *derived protocol* that declares the methods of the protocol. - `__init_subclass__` enforces that every derived protocol sets `factory_protocol_name` to a unique string. A protocol that forgets this raises `RuntimeError` at class definition time — *before* any instances are created. A concrete protocol declaration in a higher tier looks like: ```python class NetworkAdapterFactoryProtocol(FactoryProtocolBase): factory_protocol_name = "network_adapter" def get_network_adapter_types(self) -> List[Type[NetworkAdapter]]: ... ``` The protocol declares *one method*: how to enumerate the network-adapter types this factory provides. Different protocols can have different methods; the only universal constraint is `factory_protocol_name`. ### `FactoryExtension` — the base for concrete factories A *concrete factory* is a class that implements one or more factory protocols. It inherits from `FactoryExtension`: ```python class RhelFactoryExtension(FactoryExtension, NetworkAdapterFactoryProtocol): PRECEDENCE = 100 # higher = later = wins overrides @classmethod def get_network_adapter_types(cls) -> List[Type[NetworkAdapter]]: return [RhelNetworkAdapter] ``` Two things this declares: - The class implements `NetworkAdapterFactoryProtocol` (declared by the higher tier). - The class has a `PRECEDENCE` class attribute — a number that controls dispatch order. Higher numbers win when multiple factories implement the same protocol. The `PRECEDENCE` value is the chapter's substantive design choice. When two factories implement the same protocol, the `SuperFactory`'s default behaviour is to walk them by descending precedence and use the first one that produces a non-None result. This means: a project that wants to override the default RHEL implementation ships its own factory with a higher `PRECEDENCE`; the override automatically wins. ### `SuperFactory` — the dispatch engine [`superfactory.py`](https://github.com/code-craft-ai-company/codecraftai-extension/blob/main/source/packages/ccai/extension/factory/superfactory.py) is the engine. Its job: discover all registered factory classes at startup, sort them by precedence, and provide dispatch methods for protocol-specific lookups. The discovery happens through a known package namespace: `ccai.factories`. Every package that ships factory classes puts them under that namespace; the `SuperFactory` scans `sys.path` for the namespace, walks every package found, imports every module, and collects every `FactoryExtension` subclass: ```python def scan_ccai_factories_namespace() -> List[str]: search_paths = set([p for p in sys.path]) rel_amr_factories = os.path.dirname(os.path.dirname(os.path.dirname(__file__))) search_paths.add(rel_amr_factories) modules_found = [] for dir_path in search_paths: check_dir = os.path.join(dir_path, "ccai", "factories") if os.path.exists(check_dir) and os.path.isdir(check_dir): check_dir_modules = scan_for_descendant_modules(check_dir) modules_found.extend(check_dir_modules) return modules_found ``` The pattern is *namespace-package-driven*. Python's namespace package mechanism lets multiple installed packages contribute to the same dotted namespace — `codecraftai-sysman-linux-redhat` contributes `ccai.factories.sysman.linux.redhat`, `codecraftai-sysman-linux-ubuntu` contributes `ccai.factories.sysman.linux.ubuntu`, and so on. The `SuperFactory` doesn't care which package contributed which factory; it sees them all in the unified namespace. After discovery, the `SuperFactory` sorts by precedence (using `operator.attrgetter('PRECEDENCE')`) and exposes dispatch methods. ## The five dispatch modes The `SuperFactory` provides five dispatch methods, each suited to a different consumption pattern: ### `create_instance_by_order(get_type_method)` Walks all factories in precedence order, calls `get_type_method` on each, and returns an instance of the *last* type returned (the lowest-precedence factory's type). This is the "fallback to default" mode — the highest-precedence factory's contribution can be `None` (returning nothing) and the dispatch falls through to lower-precedence factories. ### `create_instance_for_each(get_type_method)` Walks all factories and returns a list of instances, one per factory that returned a non-None type. Use when the consumer wants *all* contributed implementations. ### `get_override_types_by_order(get_type_method)` Like `create_instance_by_order` but returns the *type*, not an instance. The caller instantiates the type itself. Use when the consumer needs control over construction arguments. ### `iterate_override_types_for_each(get_type_method, iterate_type_lists=True)` A generator that yields each contributed type one by one. When a factory's `get_type_method` returns a *list* of types, the generator yields each type in the list separately (controlled by the `iterate_type_lists` parameter — note: per the recent hygiene-sweep typo fix, the parameter name was corrected from the previous `interate_type_lists`). This is the workhorse dispatch — most consumer code uses this to iterate every contribution. ### `iterate_override_types_for_each` use-case A typical consumer: ```python super_factory = ConfiguredSuperFactorySingleton() for adapter_type in super_factory.iterate_override_types_for_each( NetworkAdapterFactoryProtocol.get_network_adapter_types ): register_adapter_type(adapter_type) ``` The consumer doesn't know which packages contributed which adapter types. It walks the contributions in precedence order and registers each one. The same loop works whether the host is RHEL, Ubuntu, macOS, or has multiple OS-specific factories installed. ## How factories register A package that wants to contribute a factory does three things: ### 1. Create the namespace-package directory The package's source layout includes `ccai/factories//`: ``` codecraftai-sysman-linux-redhat/ └── source/packages/ └── ccai/ └── factories/ └── sysman/ └── linux/ └── redhat/ ├── __init__.py └── networkadapterfactory.py ``` The directories are *namespace packages* (no `__init__.py` until the leaf, or empty `__init__.py`). This is what lets multiple installed packages contribute under the same `ccai.factories.*` prefix without conflicting. ### 2. Define the factory class `networkadapterfactory.py` contains: ```python from ccai.extension.factory.factoryextension import FactoryExtension from ccai.sysman.common.networkadapterfactoryprotocol import NetworkAdapterFactoryProtocol from ccai.sysman.linux.redhat.networkadapter import RhelNetworkAdapter class RhelNetworkAdapterFactoryExtension(FactoryExtension, NetworkAdapterFactoryProtocol): PRECEDENCE = 100 @classmethod def get_network_adapter_types(cls): return [RhelNetworkAdapter] ``` The class is a `FactoryExtension` and implements the protocol. Its `PRECEDENCE` is set; its method returns the contributed type(s). ### 3. Install the package `pip install codecraftai-sysman-linux-redhat`. Python's namespace package mechanism merges the package's `ccai.factories.*` contribution into the `ccai.factories` namespace. The `SuperFactory`'s next discovery pass finds the new factory. That's the full registration. No central registry, no manual imports, no init-time configuration. The package is installed; the factory is discovered. ## The precedence ordering `PRECEDENCE` is a class attribute — an integer. Higher numbers *win* when the dispatch reaches a "use the first one" mode. The convention across the portfolio: - `PRECEDENCE = 100` — the *base* contribution from a foundation package. - `PRECEDENCE = 200` — a *standard override* from a per-OS or per-environment package. - `PRECEDENCE = 300+` — a *project-specific override* from a consuming application. The numbers are conventions, not enforced. When two factories have the same `PRECEDENCE`, the dispatch order is *insertion order* (which is determined by namespace-walking order — not predictable across reinstalls). The discipline is: don't share `PRECEDENCE` values between factories that implement the same protocol. ## What the factory machinery is *not* Three deliberate non-features. ### Not a service locator The factories return *types*, not *instances*. The consumer instantiates the type and manages the lifetime of the instance. This is consistent with the philosophy's *explicit-is-better-than-implicit* principle — service-locator patterns hide instance lifetime, which complicates resource management. The five dispatch methods include both type-returning forms (`get_override_types_by_order`, `iterate_override_types_for_each`) and instance-returning forms (`create_instance_by_order`, `create_instance_for_each`). Consumers that want types use the first; consumers that want instances use the second. Both are explicit about which mode they're using. ### Not runtime-mutable Factories are discovered at startup and sorted once. After that, the factory list is fixed. Dynamically registering a factory at runtime — say, after a plugin is installed — requires re-running discovery. The shipped `SuperFactory` doesn't expose an API for this; consumers that need it call the discovery functions directly. This is consistent with chapter-1's *stability-over-performance* principle. A runtime-mutable factory list would buy hot-reload support at the cost of locking and synchronisation; the shipped form is simpler and covers the typical use case. ### Not a dependency-injection container The factories produce types that satisfy a protocol. They do *not* declare dependencies between types, do *not* automatically construct dependency graphs, do *not* manage transitive lifetimes. Each factory is independent; each factory call is independent; composition is the consumer's responsibility. When the consumer's needs grow into "I want a type *and* its dependencies wired up," the right answer is either to extend the factory protocol (each factory now returns a wired instance) or to bring in an actual DI container (a separate tool entirely). The chapter-2 *if-it's-hard-to-explain-try- another-way* principle gates the decision. ## How `codecraftai-this-app` configures the SuperFactory The `codecraftai-this-app` package's `wellknown.py` exposes a helper: ```python def ConfiguredSuperFactorySingleton(): global CONFIGURED_SUPER_FACTORY if CONFIGURED_SUPER_FACTORY is None: # ... discovery + configuration ... CONFIGURED_SUPER_FACTORY = build_super_factory() return CONFIGURED_SUPER_FACTORY ``` The pattern: *one* `SuperFactory` instance per process, lazily constructed on first use. Subsystems that need factory dispatch call `ConfiguredSuperFactorySingleton()` and use the returned instance; the discovery happens once. The `AppPersona.AM_EXTENSIONS` field (chapter 6) carries a semicolon-separated list of *additional* extension modules the `SuperFactory` should scan beyond the default `ccai.factories` namespace. This is the project-specific extension hook — a project ships its own factories under a different namespace and points the persona at it. ## Cross-references The factory machinery is used by: - **Volume 4 (Sysman)** for the per-OS implementation dispatch catalogued in chapter 2 of that volume. - **Volume 5 (Interop)** for the protocol implementation registry — DNS, SSH, serial, power, etc. Each protocol implementation registers as a factory under `ccai.factories.interop.*`. - **Volume 6 (Landscape)** for the device-detail-mapper registry — UPnP devices, networking devices, etc. The pattern is consistent: each package contributes factories; the consumer dispatches by protocol; the discovery is automatic. The chapter-23 of Volume 3 (Extending the Task Model) is the *opposite* pattern — there, the workflow YAML's `tasktype: module@Class` reference is *direct*, not factory-mediated. The choice is deliberate: workflow tasks are *named explicitly* by the workflow author, who wants visibility into which task is selected; cross-OS implementations are *dispatched* by the runtime, which abstracts the choice from the consumer. Both patterns coexist; the choice between them is the chapter-1 *explicit-vs-implicit* question applied to extension dispatch. ## Summary - **`codecraftai-extension`** ships the registration-based, precedence-ordered factory machinery that lets packages plug new behaviour into existing extension points without modifying the consumers. - Three artifact types: **`FactoryProtocolBase`** (the marker for a factory protocol; enforces unique `factory_protocol_name`), **`FactoryExtension`** (the base for concrete factories; declares `PRECEDENCE`), **`SuperFactory`** (the dispatch engine). - Discovery is **namespace-package-driven**. Packages ship factories under the `ccai.factories.*` namespace; the `SuperFactory` scans `sys.path` for the namespace and imports every contributed module. Adding a new factory is a new package install; no central registry. - The **five dispatch modes** — `create_instance_by_order`, `create_instance_for_each`, `get_override_types_by_order`, `iterate_override_types_for_each`, plus the per-protocol custom dispatchers — give consumers a choice between type-returning and instance-returning forms. - **Precedence convention**: 100 (base), 200 (per-OS / per-env override), 300+ (project-specific override). Higher wins. - The machinery is *not* a service locator (returns types, not managed instances), *not* runtime-mutable (discovery happens at startup), *not* a DI container (no transitive dependency graph). - `ConfiguredSuperFactorySingleton()` from `codecraftai-this-app/wellknown.py` is the per-process singleton. Discovery is lazy on first call; the `AppPersona.AM_EXTENSIONS` field configures additional namespaces beyond `ccai.factories`. - Used by Volume 4 (sysman per-OS dispatch), Volume 5 (interop protocol registry), Volume 6 (device-detail mappers). The pattern is universal across the portfolio. The next chapter introduces the runtime variables and well-known paths — the `codecraftai-runtime` package that builds on chapter 6's identity machinery to give every running process a structured view of its runtime environment. # Chapter 8 — Runtime Variables and Well-Known Paths > *Application identity is structural; runtime is operational. > The runtime layer is what every running process inherits the > moment its persona is established and its activation profile > is selected.* The previous chapter introduced the extension and factory machinery. This chapter introduces the *runtime* variable set and the *well-known path* helpers every higher tier uses. These modules now live inside [`codecraftai-this-app`](https://github.com/code-craft-ai-company/codecraftai-this-app/tree/main/source/packages/ccai/thisapp), under the `ccai.thisapp` namespace alongside the chapter-6 persona machinery. (Historically they were a separate `codecraftai-runtime` package; that package is being retired and now holds thin re-export shims that forward `ccai.runtime.X` to `ccai.thisapp.X`. Update any `ccai.runtime.*` imports to `ccai.thisapp.*`.) > **As-built note.** The runtime variable set, the path helpers, > the activation profiles, and the enumerations were all folded > into `codecraftai-this-app` so that the variable-descriptor model > and the operational variables ship from one tier. This also let > `codecraftai-testbase` become near-standalone: testbase does *not* > import `RuntimeVariables` or these path helpers at all. See the > sidebar later in this chapter, "What testbase actually uses". The runtime-related contents of `ccai.thisapp` are organised into these modules: - **`runtimevariables.py`** — `RuntimeVariablesDesc` and `RuntimeVariables` extending the chapter-6 persona variable set, plus `DefaultValue`, the factory defaults, and `resolve_runtime_variables`. - **`paths.py`** — well-known directory helpers (cache, presence, output, results, diagnostics, test-results, by-products, temp, summary-template). - **`activation.py`** — activation-profile commands that configure logging and behaviour for the running process. - **`enumerations.py`** — `JobType` and `ActivationProfile`. The programmatic override helpers (`optionoverrides.py`) have not yet been moved into `ccai.thisapp`; they currently still live in `codecraftai-runtime` (see the section on option overrides below). ## The runtime variable set [`runtimevariables.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/runtimevariables.py) extends `PersonaVariablesDesc` (chapter 6) with the variables every running process needs for *operational* concerns — distinct from chapter-6's *identity* concerns. ```python class RuntimeVariablesDesc(PersonaVariablesDesc): AM_ACTIVATION_PROFILE = AppVarDesc(default=None) AM_AUTOMATION_POD = AppVarDesc(default=DefaultValue.NotSet) AM_BUILD_RELEASE = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.BUILD_RELEASE) AM_BUILD_BRANCH = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.BUILD_BRANCH) AM_BUILD_NAME = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.BUILD_NAME) AM_BUILD_URL = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.BUILD_URL) AM_RUN_ID = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.RUNID) AM_PIPELINE_ID = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.PIPELINE_ID) AM_PIPELINE_NAME = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.PIPELINE_NAME) AM_JOB_ID = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.JOB_ID) AM_JOB_LABEL = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.JOB_LABEL) AM_JOB_NAME = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.JOB_NAME) AM_JOB_SEED = AppVarDesc(default=default_job_seed, context_path=ContextPaths.JOB_SEED) AM_JOB_TYPE = AppVarDesc(default=DefaultValue.NotSet, context_path=ContextPaths.JOB_TYPE) AM_INTERACTIVE_CONSOLE = AppVarDesc(default=False) AM_DEBUG_BREAKPOINTS = AppVarDesc(default=None, context_path=ContextPaths.DEBUG_BREAKPOINTS) AM_DEBUG_DEBUGGER = AppVarDesc(default=None, context_path=ContextPaths.DEBUG_DEBUGGER) # ... and many more ``` The descriptors fall into five thematic groups: ### Activation and pod identity - **`AM_ACTIVATION_PROFILE`** — names the activation profile the process is running under (Command, Console, Job, Service — the `ActivationProfile` enum from `enumerations.py`). - **`AM_AUTOMATION_POD`** — the apod identifier (Volume 3 chapter 5's apod from the landscape). ### Build provenance - `AM_BUILD_RELEASE`, `AM_BUILD_BRANCH`, `AM_BUILD_FLAVOR`, `AM_BUILD_NAME`, `AM_BUILD_URL` — the build identity of the code currently running. Carried into traces and result reports for retrospective analysis. ### Run, pipeline, job identity - `AM_RUN_ID` — UUID for the current run. - `AM_PIPELINE_ID` / `AM_PIPELINE_NAME` / `AM_PIPELINE_INSTANCE` — CI pipeline metadata. - `AM_JOB_ID` / `AM_JOB_LABEL` / `AM_JOB_NAME` / `AM_JOB_SEED` / `AM_JOB_TYPE` — the running job's identity. ### Debugging knobs - `AM_INTERACTIVE_CONSOLE` — boolean indicating whether the process has an attached interactive console. - `AM_DEBUG_BREAKPOINTS` — set of named breakpoints the process should honour (the `breakpoints:` field from the chapter-7 runtime YAML in Volume 3). - `AM_DEBUG_DEBUGGER` — which debugger to attach (`debugpy`, `pydevd`, `none`). ### Diagnostics policy - `AM_DIAGNOSTICS_POSTRUN` / `AM_DIAGNOSTICS_PRERUN` — whether to capture diagnostics around runs. - `AM_DIAGNOSTICS_TRACEBACK_POLICY_OVERRIDE` — override the default traceback policy (the chapter-1 errors-never-silent discipline applied to traceback verbosity). ### `AM_JOB_SEED` — the reproducibility seed The `AM_JOB_SEED` descriptor uses a *factory function* as its default, not a literal value: ```python def default_job_seed(): rtnval = str(uuid4()) return rtnval AM_JOB_SEED = AppVarDesc(default=default_job_seed, context_path=ContextPaths.JOB_SEED) ``` The factory returns a fresh UUID on every resolution. When the caller doesn't override the seed, every run gets a new UUID; when the caller *does* override (passing a captured seed for replay — the chapter-16 of Volume 3 capture-and-replay pattern), the override pipeline takes precedence. This is the descriptor model's *graceful fallback* for values that need to be unique per run but reproducible when explicitly fixed. ## `RuntimeVariables` — the typed accessor For consumer convenience, the package ships `RuntimeVariables` — an `AppVarAccessor`-based class that parallels `RuntimeVariablesDesc`: ```python class RuntimeVariables(PersonaVariables): AM_ACTIVATION_PROFILE = AppVarAccessor(RuntimeVariablesDesc.AM_ACTIVATION_PROFILE) AM_RUN_ID = AppVarAccessor(RuntimeVariablesDesc.AM_RUN_ID) AM_JOB_SEED = AppVarAccessor(RuntimeVariablesDesc.AM_JOB_SEED) # ... and so on ``` A consumer reads `RuntimeVariables.AM_RUN_ID` and gets the resolved UUID. The accessor calls `ThisApplicationSingleton().get_variable_value(descriptor)` under the hood; the call site doesn't need to know. This is the *typed-read-through* pattern from chapter 6 applied at scale. Forty-some descriptors; forty-some accessors; the call sites are uniform. ## Activation profiles [`enumerations.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/enumerations.py) defines the `ActivationProfile` enum with four values: - **`Command`** — the process is running as a CLI command. Logs go to a standard file plus the console; user-interactive. - **`Console`** — the process is an interactive console. Logs go to a file (away from the screen) so log output does not disrupt user interaction; the output directory is a temp directory. - **`Job`** — the process runs as a job that collects its by-products (logs, results) under an output folder. A job carries a `JobType` (see below) — Build, Infrastructure, Orchestration, or TestRun. - **`Service`** — the process is a long-running service. Logs go to a rotating file handler; non-interactive. The companion `JobType` enum names the kinds of job: `Unknown` (the default until activation sets it), `Build`, `Infrastructure`, `Orchestration`, and `TestRun`. As the path section below shows, the test-reporting path helpers are *activation-gated* on `AM_JOB_TYPE` being something other than `Unknown`. [`activation.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/activation.py) provides four `activate_profile_*` functions — one per profile: `activate_profile_command`, `activate_profile_console`, `activate_profile_service`, and `activate_profile_job(job_type)` (the job activation takes the `JobType` to run under). Each function: 1. **Guards** against duplicate activation — reads `RuntimeVariables.AM_ACTIVATION_PROFILE` and raises if already set. A process may activate exactly one profile. 2. **Sets the active profile** through `override_variable_value`. 3. **Runs common setup** (`_activate_profile_common`) that resolves the output directory appropriate to the job type. 4. **Configures logging** appropriate to the profile (file path, format, level, handler type). 5. **Returns** with the process ready to run. ```python def activate_profile_command(): # Guard against attemps to activate more than one, activation profile. if RuntimeVariables.AM_ACTIVATION_PROFILE is not None: errmsg = "An attempt was made to activate multiple environment activation profiles. profile={}".format( RuntimeVariables.AM_ACTIVATION_PROFILE ) raise SemanticError(errmsg) this_app = ThisApplicationSingleton() this_app.override_variable_value(RuntimeVariablesDesc.AM_ACTIVATION_PROFILE, ActivationProfile.Command) _activate_profile_common(this_app) log_directory = RuntimeVariables.AM_LOG_DIRECTORY log_basename = RuntimeVariables.AM_LOG_BASENAME logging_location = LoggingLocation(log_directory, log_basename) console_level = RuntimeVariables.AM_LOG_LEVEL_CONSOLE file_level = RuntimeVariables.AM_LOG_LEVEL_FILE activate_standard_file_logging(logging_location, console_level, file_level) return ``` The pattern is consistent across the four activation functions — guard against duplicate activation; set the profile; common setup; profile-specific logging. (One honest wrinkle: `activate_profile_job` currently raises a plain `RuntimeError` rather than `SemanticError` on the duplicate-activation guard, where the other three raise `SemanticError`.) The application's startup code calls *one* of the activate functions: ```python from ccai.thisapp.persona import establish_app_persona from ccai.thisapp.activation import activate_profile_command establish_app_persona( name="my-tool", logger_name="my-tool", log_basename="my-tool", ) activate_profile_command() # ... rest of the application ... ``` Two lines of startup; both required; both clearly named. The chapter-1 *explicit-is-better-than-implicit* discipline applied to process initialization. ## Well-known paths [`paths.py`](https://github.com/code-craft-ai-company/codecraftai-this-app/blob/main/source/packages/ccai/thisapp/paths.py) provides helpers for the directories every running process needs. Each helper returns an *absolute path*, creating the directory if it doesn't exist. Module-level globals cache the resolved paths: ```python DIR_PRESENCE_DIRECTORY = None DIR_CACHE_DIRECTORY = None DIR_DIAGNOSTICS_DIRECTORY = None DIR_RESULTS_DIRECTORY = None DIR_SHARED_STORE_DIRECTORY = None DIR_TESTRESULTS_DIRECTORY = None DIR_TESTCASE_BYPRODUCTS_DIRECTORY = {} ``` Each helper checks its global, computes the path on first call, caches it, and returns. Subsequent calls return the cached value. The pattern keeps path resolution fast without exposing the caching mechanism to callers. The always-available helpers — these resolve from the persona / output variables and do not require a job activation: - **`get_presence_directory()`** — returns the application's well-known presence directory (`AM_PRESENCE_DIR`). - **`get_directory_for_cached_files()`** — returns the per-application cache directory (`/cache`). Used for intermediate artifacts that persist across runs. - **`get_path_for_output()`** — returns the output directory (`AM_OUTPUT_DIRECTORY`) where results and artifacts are deposited. - **`get_path_for_shared_store()`** — returns the configured shared-store directory (raises `ConfigurationError` if `AM_SHARED_STORE_DIRECTORY` is unset). - **`get_temporary_directory()` / `get_temporary_file(suffix, prefix)`** — a `temp/` directory under the output directory, and a temp file within it via `tempfile.mkstemp`. Used by Volume 3's `ShellScript` task (chapter 19) to create the per-task temp script. - **`collect_python_modules(search_dir, max_depth)`** — utility for module discovery; used by the chapter-7 factory machinery for namespace scanning. The *activation-gated* helpers — these raise `SemanticError` unless the process was activated as a job (i.e. `AM_JOB_TYPE` is something other than `JobType.Unknown`). They resolve paths that only make sense inside a results-collecting job: - **`get_path_for_artifacts(label)`** — `/artifacts/