Chapter 6 — Topology Files

Step ordering is a flat list. Component dependency is a graph. When a flat list pretends to be a graph, inserting a step in the middle becomes a logic puzzle.

The previous chapter introduced Landscape — the inventory of who and what is in scope for an orchestration run. A Landscape lists participants. It does not say how those participants relate to each other.

Two test runners might both be in the apod, but in one orchestration they form a redundant cluster (the loss of one is tolerable) and in another they are a leader-follower pair (the loss of the leader is not). Two infrastructure services might both be in the landscape, but one orchestration has a hard ordering between them ("staging-db must be live before staging-api can start") while another has no such constraint.

Relationships of this kind are the third configuration category: Topology.

Topology is the relationships, not the participants

The boundary between Landscape and Topology is sharp:

Question	Category
Is this host part of the run?	Landscape
Does this host depend on that host?	Topology
What hostname does this host have?	Landscape
What role does this host play in this orchestration?	Topology (when role varies by orchestration)
What is this host?	Landscape (`deviceType`)
How does this host relate to its peers?	Topology

The same physical Landscape can host multiple Topologies. The four runners in a staging-east apod can be:

Topology A — flat: four independent workers, no coordinator, loss-tolerant.
Topology B — leader-follower: runner-1 is leader; runner-2, runner-3, runner-4 are followers.
Topology C — two clusters: runner-1 and runner-2 form a Tier-1 high-availability pair; runner-3 and runner-4 form a Tier-2 batch pool.

Same hosts. Same Landscape file. Three different Topology files. A pipeline switches between them by setting AM_TOPOLOGY_NAMES (per the fixed-interface principle from chapter 7). Without the separation, every relationship change would require duplicating the Landscape — the four-runner Landscape would become three near-copies, and any change to the underlying hosts would have to be propagated across all of them.

This is the chapter's central architectural claim: Topology is its own file because relationships vary independently from inventory.

A note on the schema

A Landscape file has a shipped schema and a shipped example (codecraftai-automation-kit/userguide/examples/landscape.yaml). A Topology file at the time of writing has the configuration plumbing in place — codecraftai-configuration loads topology files through the standard MergeMap mechanism and exposes them through AM_CONFIG_TOPOLOGY_NAMES and the TOPOLOGY_CONFIGURATION_MAP, and the TopologyIntegrationLayer class is shipped as the runtime hook — but the schema content is consumer-defined. Different orchestration teams have different notions of what relationships matter to them.

There is one exception to the consumer-defined schema rule. The framework consumes a clusters section with a fixed shape: each cluster has a name, a group, a list of nodes, and a list of spares. The shape is consumed by LandscapeOperationalLayer._create_clusters which materialises LandscapeDeviceCluster objects from the data. This is the framework's only structural requirement on the Topology file's content — every other top-level section the user defines is parsed as raw MergeMap data and passed through to the user's Topology subclass for interpretation.

That single canonical clusters shape is also what enables the verification contract from chapter 21: the framework knows how to verify a clusters section ("are these nodes really cluster members? Is the spare really standing by?") because it knows the shape. User-defined sections require user-supplied verification logic in a Topology subclass.

This chapter therefore proposes a canonical schema with three concerns: clusters (the framework's shape, extended slightly), dependencies, and topology-specific roles. These three cover the relationship questions that recur across every orchestration the author has built. A reader who needs more (routing tables, replication topologies, sharding strategies) extends the schema additively; the TOPOLOGY_CONFIGURATION_MAP is a MergeMap of arbitrary YAML, so new top-level keys are free to add — but the user is then responsible for subclassing Topology to interpret them (Volume 6 chapter 7 walks the subclassing mechanism).

The proposed Topology schema

A Topology file has four top-level sections: environment (which landscape this Topology applies to), clusters (groups that act together), dependencies (directional ordering), and roles (member roles within this Topology).

# topology-staging-east-leader-follower.yaml
environment:
    label: staging-east-leader-follower
    landscape: staging-east

clusters:
    - name: runner-cluster
      members:
          - runner-1
          - runner-2
          - runner-3
      coordinator: runner-1
      cardinality:
          min: 2
          max: 4

dependencies:
    - source: smoke-tests
      requires:
          - staging-api
          - staging-db
    - source: staging-api
      requires:
          - staging-db

roles:
    - member: runner-1
      role: cluster-leader
    - member: runner-2
      role: cluster-worker
    - member: runner-3
      role: cluster-worker

Walk each section.

`environment` — naming and binding

environment:
    label: staging-east-leader-follower
    landscape: staging-east

Two fields:

label — the human-readable identity of this Topology. Appears in logs, in result reports, in time-travel debugging sessions. Distinct from the Landscape's label so that "ran against Landscape staging-east with Topology staging-east-leader-follower" is unambiguous in retrospective analysis.
landscape — which Landscape this Topology applies to. The loader does not enforce the binding (a single Topology file can describe relationships that apply across multiple Landscapes); the field is documentary, telling a reader which Landscape the Topology was designed against. Practically, the cross-validation step at orchestration startup checks that every member name referenced in clusters, dependencies, and roles exists in the resolved Landscape.

`clusters` — members that act together

A cluster is a named group of Landscape members that the orchestration treats as a logical unit. The LandscapeDeviceCluster class in codecraftai-landscaping is the runtime model the orchestration interacts with; the Topology file is the configuration that produces the runtime cluster.

clusters:
    - name: runner-cluster
      members:
          - runner-1
          - runner-2
          - runner-3
      coordinator: runner-1
      cardinality:
          min: 2
          max: 4

Four fields per cluster:

name — the cluster's identifier; orchestrations refer to clusters by this name.
members — a list of Landscape member names. These must resolve in the bound Landscape.
coordinator (optional) — the member that plays the coordinator role for this cluster. If unset, the cluster is leaderless. When set, the orchestration's coordinator-aware steps use this field rather than electing one at runtime.
cardinality (optional) — soft and hard bounds on the cluster size. min is the threshold below which the cluster is considered failed; max is informational and lets growth-aware steps know when to stop adding members. Either may be omitted.

A Landscape member can belong to multiple clusters in the same Topology — runner-1 could be in both a runner-cluster and a monitoring-cluster if the orchestration treats it both as a worker and as a metrics collector. The discipline does not prohibit this; it merely requires that each cluster name resolves to a clear membership.

`dependencies` — directional ordering

A dependency is a one-directional "X requires Y" relationship between Topology entities. Members, services, and clusters can all appear on either side:

dependencies:
    # smoke-tests can't run until staging-api and staging-db are live
    - source: smoke-tests
      requires:
          - staging-api
          - staging-db

    # staging-api can't start until staging-db is live
    - source: staging-api
      requires:
          - staging-db

    # the runner cluster (the cluster as a whole) can't operate
    # until staging-api is reachable
    - source: runner-cluster
      requires:
          - staging-api

The orchestration uses dependencies for two things:

Sequencing. Steps that operate on staging-api are not started until steps that bring staging-db to a ready state have completed. The orchestration's flow engine resolves the dependency graph and emits a topologically-sorted execution order. This is what "the flat list of steps becomes an output of topology resolution" from the chapter's opening means.
Tear-down. When the orchestration ends, dependencies run in reverse: tear-down of staging-db is gated on tear-down of staging-api (so the API isn't left dangling without its database). Implicit tear-down ordering through hand-maintained shutdown sequences is one of the most error-prone patterns in pipeline engineering; making the graph explicit here puts the graph through code review.

The graph must be acyclic. A cyclic dependency (A requires B; B requires A) is a configuration error caught at load time. The error message names the cycle so the engineer can break it.

`roles` — topology-specific assignments

A Landscape member already carries a role field — taskerserver, runner, dut. That role is the member's Landscape-time identity, the thing that's true regardless of which orchestration runs. A Topology overlay can introduce a Topology-time role that varies by orchestration:

roles:
    - member: runner-1
      role: cluster-leader
    - member: runner-2
      role: cluster-worker
    - member: runner-3
      role: cluster-worker

The same runner-1 is runner in the Landscape and cluster-leader in this Topology. In a different Topology (topology-staging-east-flat.yaml) the same runner-1 is just worker, with no leadership status.

The role vocabulary is application-defined. The framework makes no distinction between cluster-leader and quorum-witness and canary-runner; those are names the orchestration code recognises and acts on. The Topology file is where the names are bound to members.

A common usage pattern: orchestration code queries the resolved Topology for "all members with role X" and dispatches work accordingly. When cluster-leader work needs to happen, the code asks the Topology, gets runner-1, and routes the call. The Landscape is unchanged.

A complete worked example

Here is a Topology that pairs with the staging-east Landscape from chapter 5. The Landscape had three runners and two services (staging-db, staging-api). This Topology arranges them as a leader-follower runner cluster with a clear database-then-API ordering:

# topology-staging-east-leader-follower.yaml
environment:
    label: staging-east-leader-follower
    landscape: staging-east

clusters:
    - name: runner-cluster
      members:
          - runner-1
          - runner-2
          - runner-3
      coordinator: runner-1
      cardinality:
          min: 2
          max: 3

dependencies:
    # services come up in dependency order:
    # staging-db must be live before staging-api starts
    - source: staging-api
      requires:
          - staging-db
    # the cluster needs the api before any work happens
    - source: runner-cluster
      requires:
          - staging-api
    # smoke tests run last, after everything else is live
    - source: smoke-tests
      requires:
          - runner-cluster
          - staging-api
          - staging-db

roles:
    - member: runner-1
      role: cluster-leader
    - member: runner-2
      role: cluster-worker
    - member: runner-3
      role: cluster-worker

The orchestration that loads this Topology obtains a graph it can walk:

staging-db ─────► staging-api ─────► runner-cluster
       │                  │                  │
       └──────────────────┴──────────────────┴──────► smoke-tests

The flow engine's startup sequence is [staging-db, staging-api, runner-cluster, smoke-tests]. The tear-down sequence is the reverse. Neither was hand-maintained.

Now consider the alternate Topology — same Landscape, flat arrangement:

# topology-staging-east-flat.yaml
environment:
    label: staging-east-flat
    landscape: staging-east

clusters:
    - name: runner-cluster
      members:
          - runner-1
          - runner-2
          - runner-3
      cardinality:
          min: 1   # any one runner is enough

dependencies:
    - source: smoke-tests
      requires:
          - runner-cluster
          - staging-api
          - staging-db

roles:
    - member: runner-1
      role: cluster-worker
    - member: runner-2
      role: cluster-worker
    - member: runner-3
      role: cluster-worker

No coordinator. Lower cardinality threshold. All members play the same role. The Landscape file is identical; only the Topology file differs. The pipeline runs against either by setting AM_TOPOLOGY_NAMES=staging-east-flat or AM_TOPOLOGY_NAMES=staging-east-leader-follower — no other change.

Composition with the rest of the categories

Topology references Landscape members by name. It is referenced by nothing else; the resolved Topology is consumed by the orchestration's flow engine and by orchestration-aware code querying for roles and clusters.

Landscape  ◄──── references ──── Topology
   │                                 │
   │                                 │
   │ also referenced by              │ consumed by
   │                                 │
   ▼                                 ▼
Data Profiles, Credentials      Orchestration flow engine
                                Cluster-aware orchestration code
                                Role-based dispatch

The references go one way: Topology to Landscape. This is what makes Landscape the structural truth and Topology a layered overlay. Changing a Topology never invalidates a Landscape; changing a Landscape may invalidate the Topologies built on top of it (the cross-validation step at startup catches dangling member references).

The Runtime profile (chapter 7) parameterises how the orchestration walks the Topology — timeouts on dependency-readiness checks, parallelism within cluster operations — without altering the Topology. Same Topology, different Runtimes give different operational behaviour over the same relationship graph.

Credentials (chapters 9–12) are referenced by the Landscape, not the Topology. Topology is the relationship layer; access control remains a Landscape-and-credential concern.

Multiple Topologies per Landscape: the architectural payoff

The volume's earlier chapters argued for separation of concerns across the six categories. Topology is the chapter where the payoff is most concrete.

Without Topology files, an orchestration that needs leader-follower behaviour today and flat behaviour tomorrow has two options:

Two Landscape files that duplicate the inventory but differ in some structural field (role per client). This forces every change to the underlying inventory — a host's IP changes, a credential rotates — to be made twice. Eventually the two files drift, and one orchestration is run against a stale copy.
A single Landscape with a code-side switch (an environment variable, a CLI flag, a hardcoded conditional) that selects between leader-follower and flat behaviour. This is the Chapter 2 "conditional thicket" failure mode in fresh costume.

Topology files make this a non-problem. One Landscape, multiple Topologies, the relationship choice expressed as data and selected by name. An orchestration that needs to support both modes selects between them by changing one line in the CI workflow:

# .gitlab-ci.yml — leader-follower run
integration:
  variables:
    AM_LANDSCAPE_NAMES: "staging-east"
    AM_TOPOLOGY_NAMES: "staging-east-leader-follower"
    AM_RUNTIME_NAMES: "fast-ci"

# .gitlab-ci.yml — flat run
integration:
  variables:
    AM_LANDSCAPE_NAMES: "staging-east"
    AM_TOPOLOGY_NAMES: "staging-east-flat"
    AM_RUNTIME_NAMES: "fast-ci"

Three categories each named by their AM_*_NAMES variable, each selectable independently. This is the configurations-as-content discipline from chapter 7 applied to the relationship layer.

What Topology is not

Three things to keep out of a Topology file. Each will be tempting; resist all three.

Not a Landscape

Topology references Landscape members; it does not declare them. A Topology that contains a host: 192.167.1.21 field for a member is a Topology that has absorbed Landscape data. Resist. The reason is the same as in the previous chapter — when one file is the source of truth for inventory, every other file references it, without duplication.

If a member appears in a Topology but not in the bound Landscape, the cross-validation step at startup raises a ConfigurationError naming the missing member. This is the contract that lets readers trust a Topology in isolation: every reference resolves, or load fails loudly.

Not a workflow

Topology says what depends on what; the workflow (chapter 20) says what to do, in what order, with what fallbacks. These are related but distinct: the dependency graph from a Topology is an input to workflow planning, not the workflow itself.

A Topology that contains imperative steps (- run: deploy.sh) has crossed into workflow territory. Resist. The workflow YAML is where steps live; Topology is purely declarative relationship data.

Not a runtime configuration

Topology says what the relationships are; runtime says how the orchestration treats them. A timeout_seconds field on a cluster is misplaced; the timeout applies to how the orchestration verifies cluster readiness, which is runtime. The Topology should say "cluster runner-cluster exists with these members"; the runtime profile should say "give each cluster-readiness probe 30 seconds."

The test of "does this belong in the Topology?" is the same as for Landscape: would two different orchestration runs against the same relationship graph need different values of this field? If yes, it's runtime; if no, it's structural and belongs in the Topology.

Migration: from implicit step ordering to explicit Topology

A pipeline that has accumulated step ordering through hand-maintained sequences migrates to a Topology in three steps:

Extract the dependency graph. Walk the existing pipeline's step list and identify the implicit dependencies. When step N is "wait for the database to be ready," that's a dependency from step N+1 to a staging-db Landscape entry. When step M is "deploy the API after the database," that's a dependency from staging-api to staging-db.
Identify the clusters and roles. Wherever the existing pipeline assumes that some hosts coordinate or that one host is the leader, those assumptions become cluster definitions and role assignments in the Topology.
Replace the step list with workflow steps that consume the Topology. The taskplus workflow (chapter 20) goes from a list of explicit ordering to a set of steps each of which declares its required Topology entities; the FlowEngine sequences them automatically.

After migration, the pipeline's ordering is no longer imperative. Adding a step is a Topology change ("smoke-tests requires runner-cluster") plus a workflow change ("here's the smoke-tests step"); the position of the step in the execution order is computed from the graph, not specified by hand.

Where Topology doesn't fit

Honest scoping notes:

Trivial single-step pipelines. A pipeline with three sequential steps and no branching does not need an explicit Topology. The discipline applies when relationships are non-trivial — multiple participants, multiple dependencies, role variation across runs.
Fully ad-hoc workflows. A one-time investigation or a developer's debug session does not need a Topology file. The loader's default-topology fall-through (which produces an empty graph) handles this case — the orchestration runs without Topology constraints, and the workflow's explicit step ordering carries the day.
Streaming graph generation. Topologies that change shape every few seconds — autoscaling clusters where membership shifts during a run — strain the file model. The pattern there is to generate a snapshot Topology at orchestration start (from the service-discovery output), then treat it as the Topology for the run. Out of scope for this chapter; consistent with the discipline.

The discipline scales from "no relationships, single host" to "many participants with varied relationships across multiple Topologies." Above that scale, Topologies compose through the standard MergeMap overlay (chapter 14), the same way Landscapes do.

Summary

A Topology is the relationships between Landscape members, expressed as a separate file because the same physical inventory can host multiple relationship arrangements.
The proposed schema has three concerns: clusters (members that act together, optionally with a coordinator and cardinality bounds); dependencies (one-directional "X requires Y" relationships used for sequencing and tear-down); roles (Topology-time role assignments distinct from Landscape-time roles).
The configuration plumbing for Topology — MergeMap loading, AM_TOPOLOGY_NAMES, the TopologyIntegrationLayer — is shipped; the schema content is consumer-defined, and this chapter proposes the canonical shape.
Topology references Landscape members by name; the cross- validation step at startup catches dangling references. No data flows from Topology back into Landscape.
The architectural payoff: one Landscape can host many Topologies. Switching between leader-follower and flat behaviour is a one-line change to AM_TOPOLOGY_NAMES, not a duplicated Landscape file.
A Topology is not a Landscape (no inventory), not a workflow (no imperative steps), and not a runtime configuration (no behaviour knobs).
Migration from implicit step ordering to explicit Topology is mechanical: extract the dependency graph, identify clusters and roles, replace the imperative step list with Topology-aware workflow steps.

The next chapter — already drafted — is the thesis chapter on Runtime files vs Environment Variables. After that, Part II finishes with chapter 8 on Data Profiles.