Chapter 11 — Credentials at Scale

Volume 3 walks credentials at the orchestration layer: resolution, rotation, scoping, environment spanning. This chapter walks the same vocabulary at the job layer — when one test job needs credentials for twelve hosts and fifteen services, what does the framework do, and how does the activation chain make it look simple?

Volume 3 Part III (chapters 9 through 12) is the canonical treatment of credentials in the codecraftai-* portfolio. It walks the credential vocabularies (basic auth, SSH keypairs, API tokens, OAuth, certificates, derived credentials), the resolution model (vault, file, environment-as-fallback, prompt, derived), rotation in flight, and credential portability across local-dev / CI / staging / prod.

This chapter is the job-layer view of the same machinery. A distributed test job's credentials story has three concerns that the orchestration story doesn't fully address:

• How credentials get from the orchestration to the job at hand-off (chapter 7's verified contract sets it up; this chapter walks the consumption side).
• How a single job uses credentials for many hosts without the wiring becoming N×M complex.
• How credentials flow from the controller (where they're resolved) to the hosts under test (where they're used) — or do they not flow at all?

The answers are clean. Credentials live on the controller; the controller authenticates to the hosts; the hosts never see the credential material. The framework's machinery makes this idiomatic.

Credentials in the Landscape — declared, not resolved

A device declaration in the Landscape (Volume 1 chapter 9 walked the shape) references credentials by name:

- deviceType: network/ssh
  host: 192.168.1.30
  credentials:
      - casey-node      # ← named reference, not material
  features:
      isolation: true

The casey-node string is a credential reference. The actual material — the SSH keypair, the password, the API token — lives in the Credentials configuration category (Volume 3 chapter 9). At activation time, the framework's CredentialManager resolves the reference to the material.

This separation is critical. The Landscape file can be checked into version control because it has no secret material; only references. The Credentials configuration is what actually has secrets, and it's encrypted (Volume 3 chapter 11 walks Fernet encryption) or fetched from a vault.

The CredentialManager — one resolver per Pod

CredentialManager (in codecraftai-landscaping) is the job's credential lookup service. Initialised by LandscapeConfigurationLayer.initialize_credentials() during fire_activate_configuration() (chapter 6 walked the flow), it holds the resolved credentials for the duration of the job.

The lookup is by name:

manager = LandscapeSingleton().credentials_manager
ssh_cred = manager.lookup_by_name("casey-node")

The returned ssh_cred carries the actual auth material: private key, optional passphrase, username, etc. The framework's SshAgent for any device declaring casey-node gets the credential from the manager and uses it to authenticate.

Two properties of the manager:

1. Single-pass resolution. The manager resolves all credentials at activation time. A name that doesn't resolve fails activation (chapter 7's Landscape verification catches it). Once activation succeeds, the manager's lookups are pure dict reads — no further resolution-side failures during the test.
2. Per-Pod scope. Each Pod has its own manager. The credentials for Pod A and Pod B are independent. A credential rotation that affects Pod A doesn't touch Pod B's manager mid-run.

How a credential reaches a host

A test issuing an SSH call against a device:

device = LandscapeSingleton().device("casey-01")
status, stdout, stderr = device.ssh.run_cmd("uptime")

The flow underneath:

1. device.ssh is the device's SSH ProtocolExtension, attached at activation time (chapter 6).
2. The ProtocolExtension wraps an SshAgent (chapter 10).
3. The agent holds a reference to the device's resolved SshCredential (looked up by name from the CredentialManager at activation time).
4. The agent's run_cmd opens an SshSession; the session uses the credential to authenticate against the device's host (192.168.1.30 in our example).
5. The host validates the auth. If valid, the session is established and the command runs.

The credential never travels to the host as data. The host receives an authenticated SSH connection; the auth material stays on the controller. This is the standard SSH model — the client proves possession of the credential through the auth handshake without sending the material.

The implication for the test author: credentials are an input to the controller's wire calls, not data the test ever sees. A test never has reason to pass a credential through its body; the framework wires credentials to devices at activation time and the test's calls inherit the auth.

Composable credential lists in a device declaration

A device's credentials: is a list, not a single name:

- deviceType: network/ssh
  host: 192.168.1.30
  credentials:
      - casey-node
      - emergency-recovery
      - ci-runner

The list is composable per the universal-composability property (Volume 3 chapter 3, Vol 0 ch.9). At activation, the framework loads each named credential and the SshAgent has all of them available. Different operations can use different credentials:

# Default — uses primary_credential (first in the list).
device.ssh.run_cmd("uptime")

# Specific user — opens a session with a different credential.
admin_session = device.ssh.open_session(user="ci-runner")
admin_session.run_cmd("systemctl reload service")

The first name in the list is the primary credential. The others are alternates the test can switch to. The list is useful when:

• Different tasks need different privilege levels (regular user for most operations; root for some).
• A primary credential might rotate during a long run; the alternate is the fallback.
• Different test phases have different identity requirements (CI runner during setup; user-level during tests).

Credentials per service vs per device

The Landscape's services section (Volume 1 chapter 9) carries its own credentials:

infrastructure:
    services:
        - serviceType: network/service
          name: someservice
          credentials:
              - adminuser

Service credentials work the same way as device credentials — named, resolved at activation, never flowing to remote hosts. The framework exposes service credentials through the service's ProtocolExtension (e.g., service.api.client) just as device credentials are exposed through device.ssh.

A test using both a device and a service:

device = LandscapeSingleton().device("casey-01")
service = LandscapeSingleton().service("someservice")

# Device call uses device's primary credential (casey-node).
device.ssh.run_cmd("verify-state")

# Service call uses service's primary credential (adminuser).
service.api.post("/admin/refresh", data={...})

Two different credentials, two different remote endpoints, zero credential material in the test code.

Cross-host concurrency and credential safety

Chapter 10 walked intra-test concurrency — fanning a command out to N hosts via a ThreadPoolExecutor. The credential implications:

• Each device's SshAgent holds its own credential. Concurrent calls from a fan-out to N devices use N independent credentials, with no cross-contamination.
• Credentials are not shared between threads in the thread pool. Each thread's call goes through the device's agent, which uses the device's credential. No mutex needed; no race.
• The Aspects object passed to each call is Freezable (chapter 10) — same instance shared across the fan-out cannot be mutated.

Credential safety in concurrent fan-out is therefore architectural: the framework's design (per-device agents, frozen aspects, no shared credential cache) makes the parallelism safe by construction. The test author writes straightforward concurrent code; the framework holds the invariants.

Credential rotation during a long run

A four-hour integration suite outlives many credential rotation windows. Volume 3 chapter 11 walks rotation as a configuration discipline; the job-side question is: what happens when a credential rotates mid-run?

Three patterns the framework supports:

TTL caching with rotation

The CredentialManager (per its Volume 3 ch.11 contract) can cache resolved credentials with a TTL. When the TTL expires mid-run, the next lookup re-resolves through the underlying backend (vault, file, etc.). If the backend rotated, the new material is fetched.

For SSH agents, the rotation is transparent to the test: the next session opened against the device picks up the freshly-resolved credential. Sessions already open continue with the old credential until they close.

Composable list with rotation fallback

When a primary credential rotates and the SSH session fails auth on next use, the agent can fall back to the alternate credentials in the device's list. The test sees the fall-back transparently; the JSOS record can flag that a fallback occurred for post-run review.

Explicit rotation events

For tests that exercise rotation (testing the rotation itself), the test is the actor that triggers the rotation. The test's pattern:

def test_rotation_does_not_disrupt_active_sessions():
    """
        Verify that rotating credentials does not disrupt active
        sessions.
    """
    device = LandscapeSingleton().device("casey-01")
    long_running_session = device.ssh.open_session()

    # Trigger rotation through the credential vault's API.
    rotate_credential("casey-node")

    # Verify the long-running session still works.
    status, _, _ = long_running_session.run_cmd("uptime")
    rtnval = (status == 0)
    assert rtnval

    # Open a new session — should use the rotated credential.
    new_session = device.ssh.open_session()
    status, _, _ = new_session.run_cmd("uptime")
    rtnval = (status == 0)
    assert rtnval
    return

The test, as the single actor, drives the rotation; the framework's credential machinery handles the consequences (active sessions hold old creds; new sessions get fresh ones).

What the controller owns vs what the host owns

The clear ownership separation that the framework imposes:

The line is sharp. The host never has data the controller gave it from the credential. The controller never has data the host gave it about its own auth state. The two sides trust the SSH handshake to validate identity without exchanging secrets in either direction.

This is the foundation of the determinism model from chapter 2. If credentials flowed bidirectionally, the test could become non-deterministic depending on what the host sent back about its auth state. The unidirectional ownership keeps the test's behaviour entirely a function of the controller's state.

What this chapter set up

The credentials story at the job layer is short because the framework's design does most of the work:

• Credentials are declared by name in the Landscape.
• The CredentialManager resolves them at activation.
• ProtocolExtensions hold credential references; agents use them.
• Credential material never flows to remote hosts.
• Concurrent fan-out is safe by construction.
• Rotation is handled at the manager level, transparent to most tests; tests that exercise rotation drive it as the actor.

A reader who understands chapter 6 (activation), chapter 10 (the wire), and Volume 3's credential vocabulary has the full picture. This chapter assembled the parts.

What the next chapter does

Chapter 12 walks data profiles in remote workers — the analogous story for connection profiles to data stores. Where credentials are about authentication, data profiles are about connection topology (which database, which queue, which cache). The framework's discipline is the same: declare by name in the Landscape, resolve at activation, expose through ProtocolExtensions. The chapter is short because the discipline is uniform.