Chapter 10 — Credential Resolution

The codecraftai-credentials package gives you the vocabulary. The package does not give you a way to fetch the secret value. This chapter is about that gap and how to close it.

The previous chapter introduced the eleven shipped credential classes and showed worked examples in which secret values appeared as ${VAULT:/secret/...} references rather than literal strings. That syntax was not part of the shipped package — it was a preview. This chapter is the discipline that makes the syntax meaningful.

The chapter is the most substantial design exercise in the volume. The reason is that the portfolio's deliberate decision to ship a vocabulary layer with no resolution backends puts a real engineering load on the reader: you will build the resolver yourself, and the choices you make will shape your team's security posture for the rest of the project's life. The chapter makes those choices concrete by proposing an architecture, the interfaces it exposes, and a recipe for the most common backends.

The gap to close

Recall the shape of the load path through the shipped package:

credentials.yaml ──► YAML parser ──► dict ──►
   CredentialManager.load_credentials(credential_info=dict) ──►
      validate() per credential ──►
      dispatch by category tag ──►
      construct typed credential object ──►
      register in self._credentials[identifier]

The CredentialManager takes a parsed dictionary — credential_info — and treats every string value as opaque. When the YAML contains password: hunter2, the resulting BasicCredential.password is the literal string "hunter2". When the YAML contains password: ${VAULT:/secret/staging/db/rw}, the resulting password is the literal string "${VAULT:/secret/staging/db/rw}". The package does not interpret the syntax; it merely stores the value.

This is the gap. Between the literal ${VAULT:...} reference and a usable secret, something has to fetch the value from a real secrets store. That something is the resolver layer. It is not shipped. It is the reader's responsibility.

Three reasons for the choice:

1. Backends are environment-specific. A team using HashiCorp Vault has different needs from a team using AWS Secrets Manager, from a team using a static encrypted file. Shipping one would privilege one environment over others; shipping ten would bloat the package with dependencies most users don't need.
2. Resolution is the security boundary. The point at which a credential reference becomes a credential value is the point at which audit, rotation, and access control happen. Forcing the reader to design that boundary deliberately produces a better security posture than handing them a black-box default.
3. The vocabulary stays portable. A credentials YAML written against the eleven shipped types and the ${SCHEME:path} syntax is the same in every environment. The resolver swaps; the credentials file does not.

The reference syntax

The proposed convention is ${SCHEME:path} — a bash-style parameter-expansion form, where the SCHEME names a backend and the path is the backend-specific lookup key. Examples:

Three things the syntax gets right:

• The scheme is explicit. A grep for ${VAULT: finds every Vault reference in a project. No ambiguity, no implicit defaults.
• The path is opaque to the parser. The resolver decides how to interpret it. The reference syntax does not constrain backend semantics.
• The reference is recognizable in logs. When a ${VAULT:...} string accidentally appears in an error message or a debug dump, it is immediately distinguishable from a literal secret. This is a security defence-in-depth — the syntax is loud, so a leak of a reference is catastrophically less bad than a leak of a value.

The syntax is bash-style on purpose. Shell users recognise the shape immediately. YAML keeps the string intact (the curly braces do not require escaping). Most editors will not "helpfully" parse the reference into something else.

The Resolver interface

The resolver is the piece you build. It has a small interface:

from typing import Optional


class CredentialReference:
    """
        A parsed credential reference of the form ${SCHEME:path}.
    """
    def __init__(self, scheme: str, path: str, raw: str):
        self.scheme = scheme    # e.g. "VAULT"
        self.path = path        # e.g. "/secret/staging/db/rw"
        self.raw = raw          # the original "${VAULT:/secret/...}" string
        return


class CredentialBackend:
    """
        Abstract interface for a backend that resolves credential references.
    """

    SCHEME: str = None  # subclasses override; the dispatch key

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        """
            Given a credential reference whose scheme matches this backend,
            return the materialised secret value as a string.

            Return None if the reference cannot be resolved (the resolver
            then tries the next backend, or raises ConfigurationError if
            no backend resolves the reference).
        """
        raise NotImplementedError("CredentialBackend subclasses must implement resolve")


class CredentialResolver:
    """
        Walks the credential configuration data, finds ${SCHEME:path}
        references, and replaces each with the value the matching backend
        returns.
    """

    def __init__(self, backends: list):
        # backends are tried in order; the first whose SCHEME matches the
        # reference is asked to resolve.  If it returns None, the resolver
        # raises rather than falling through to a different backend with the
        # same scheme.
        self._backends = backends
        self._scheme_to_backend = {b.SCHEME: b for b in backends}
        return

    def resolve_dict(self, data: dict) -> dict:
        """
            Walk a parsed YAML dict, replace every ${SCHEME:path} string
            with the resolved value.  Return a new dict; do not mutate the
            input.
        """
        return self._walk(data)

    # ... _walk, _resolve_string implementation in the worked example below

The interface is small for a reason: every method has a single responsibility, and the responsibilities don't grow. Adding a new backend means writing a new CredentialBackend subclass, not modifying the resolver.

The shipped backends (none) and the ones you build

The shipped scope of codecraftai-credentials is zero backends. The following sections walk through the most common backends a reader will build. Each subsection is a recipe — interface contract, implementation sketch, when to use it, when not to.

File backend

The simplest backend. Reads the secret value from a file on disk. Useful for:

• Containerised deployments where secrets are mounted as files (Kubernetes Secret mounted to /run/secrets/..., Docker Swarm secrets, AWS ECS secrets-as-files).
• Local development where the secret is in ~/.ccai/secrets/ protected by filesystem permissions.
• CI where the runner has been pre-populated with secret files at a well-known path.

class FileBackend(CredentialBackend):
    SCHEME = "FILE"

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        path = os.path.abspath(
            os.path.expanduser(os.path.expandvars(reference.path))
        )
        if not os.path.exists(path):
            return None
        with open(path, "r") as fh:
            value = fh.read().rstrip("\n")
        return value

Two design decisions worth noting:

• expanduser and expandvars are applied to the path so ${FILE:~/.ccai/secrets/db} and ${FILE:$XDG_RUNTIME_DIR/secrets/db} both work.
• Trailing newline stripping because most tools that write secrets to files (heredocs, echo, vault's vault read -field=...) leave a trailing newline that breaks the credential value.

When not to use the file backend: when the secret needs to rotate without a process restart. A file value is read once at resolution time; rotation requires re-resolution. Chapter 11 covers rotation.

Environment-variable backend

Reads the secret value from a process environment variable. Useful for:

• Migrating from existing env-var-based pipelines. Lets you migrate the configuration to typed credentials before migrating the backend.
• CI vendors whose secret store exposes secrets as environment variables and not as files (some older Jenkins setups).
• Tests, where you want the secret to come from a controlled fixture.

class EnvBackend(CredentialBackend):
    SCHEME = "ENV"

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        return os.environ.get(reference.path)

One design decision:

• Returns None on missing var, not an empty string. A missing environment variable should fall through to the next backend (if any) or raise a clear "no backend resolved this reference" error. An empty string would be silently accepted as the secret value, which is almost certainly a bug.

When not to use the env-var backend: as the primary backend in production. It exists to bridge the gap from a previous discipline to the new one. Once migration is complete, the env-var backend can be removed.

Prompt backend (local-dev only)

Prompts the developer for the secret value at process start. Useful for local development where the developer would rather type a password than copy a token into a file.

class PromptBackend(CredentialBackend):
    SCHEME = "PROMPT"

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        if not sys.stdin.isatty():
            return None  # not interactive; let other backends try
        return getpass.getpass(prompt=f"{reference.path}: ")

Two design decisions:

• Returns None on non-TTY so the same credentials YAML can be used in CI without the prompt firing. A ${PROMPT:...} reference in CI silently falls through to the next backend; a missing fall-through resolves to a clear error.
• getpass.getpass rather than input() so the secret is not echoed to the terminal.

When not to use prompt: in any non-interactive context. The fall-through above is the safety net.

HashiCorp Vault backend

The most common production secrets backend. Uses the hvac library (or your team's wrapper).

class VaultBackend(CredentialBackend):
    SCHEME = "VAULT"

    def __init__(self, vault_addr: str, vault_token: str):
        import hvac
        self._client = hvac.Client(url=vault_addr, token=vault_token)
        return

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        # path format: /secret/path/to/key#field
        path, field = reference.path.split("#", 1) if "#" in reference.path else (reference.path, "value")
        try:
            response = self._client.secrets.kv.v2.read_secret_version(
                path=path.lstrip("/")
            )
        except Exception:
            return None
        return response["data"]["data"].get(field)

Three design decisions:

• The Vault path supports a #field suffix so a single Vault secret can hold multiple fields and the credential reference picks one. ${VAULT:/secret/staging/db#username} and ${VAULT:/secret/staging/db#password} resolve to the two fields of the same Vault entry.
• The Vault client is constructed at backend creation time, not per-reference. Vault auth happens once; subsequent reads are cheap.
• Exceptions are swallowed and converted to None. A real resolver would log the exception (with the reference path but not the response body) so the engineer can see why a Vault lookup failed without leaking the secret if it did succeed.

When not to use Vault: when your team doesn't have a Vault deployment. Vault is excellent but operationally non-trivial; for teams that already have one, this backend is the right answer. For teams that don't, file + a separate secrets-distribution channel (Kubernetes Secrets, cloud-native KMS) is often simpler.

Cloud-native secret-manager backends

AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — each has its own client library and its own quirks, but the backend shape is the same:

class AwsSecretsManagerBackend(CredentialBackend):
    SCHEME = "AWSSM"

    def __init__(self, region: str):
        import boto3
        self._client = boto3.client("secretsmanager", region_name=region)
        return

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        # path format: secret-name#field (for JSON secrets) or just secret-name
        secret_name, field = reference.path.split("#", 1) if "#" in reference.path else (reference.path, None)
        try:
            response = self._client.get_secret_value(SecretId=secret_name)
        except Exception:
            return None
        if "SecretString" not in response:
            return None
        secret_string = response["SecretString"]
        if field is None:
            return secret_string
        try:
            data = json.loads(secret_string)
        except json.JSONDecodeError:
            return None
        return data.get(field)

Same pattern for GCP and Azure with the relevant SDK calls. When your environment is single-cloud, one backend is enough. When your environment is multi-cloud, ship both, and let the ${SCHEME:path} syntax route each reference to the right one.

Keychain backend (macOS local dev)

For developers on macOS who want to keep secrets in the system Keychain instead of in plain files:

class KeychainBackend(CredentialBackend):
    SCHEME = "KEYCHAIN"

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        # path format: service/account
        try:
            service, account = reference.path.split("/", 1)
        except ValueError:
            return None
        try:
            result = subprocess.run(
                ["security", "find-generic-password", "-s", service, "-a", account, "-w"],
                capture_output=True, text=True, check=True,
            )
        except subprocess.CalledProcessError:
            return None
        return result.stdout.rstrip("\n")

When the developer hasn't yet stored the secret in Keychain, the resolver falls through to the next backend (typically prompt). The Keychain entry then gets created by hand or via a setup script.

Derived backend

For credentials that compose from other credentials. A worked example: a database connection where the password is the same as the SSH password for the host the database runs on:

- identifier: legacy-db-on-runner-1
  categories: [basic]
  username: app
  password: ${DERIVED:ci-runner-ssh#password}

The DERIVED backend looks up another credential in the already-loaded set and returns one of its fields:

class DerivedBackend(CredentialBackend):
    SCHEME = "DERIVED"

    def __init__(self, manager):
        # the manager parameter must be the partially-loaded credential
        # manager.  The derived backend is special: it can only resolve
        # references to credentials that have already been resolved.
        self._manager = manager
        return

    def resolve(self, reference: CredentialReference) -> Optional[str]:
        try:
            cred_id, field = reference.path.split("#", 1)
        except ValueError:
            return None
        try:
            cred = self._manager.lookup_credential(cred_id)
        except ConfigurationError:
            return None
        return getattr(cred, field, None)

The derived backend is order-sensitive: the credential it derives from must already be loaded when the derivation is resolved. The resolver enforces this by performing two passes (resolve all non-derived references first, then resolve derived ones).

The derived backend is also the one place the resolver discipline gets cycle-prone: a cred-A derived from cred-B derived from cred-A is a cycle that would loop forever. A real resolver detects cycles and raises a clear error.

The resolution-order recipe

When a ${SCHEME:path} reference's scheme matches more than one backend (which the proposed dispatch model prevents — exactly one backend per scheme — but more generally, when a reference could be resolved by multiple paths), the discipline follows a strict preference order:

1. Vault / cloud secret manager preferred over everything. If a secret can be in a centralised audited store, that's where it should be.
2. File backend preferred over env or prompt. Files have filesystem permissions; environment variables don't.
3. Environment variable as a bridge during migration. Should fade out of production over time.
4. Prompt only in interactive contexts. Never the production answer.
5. Derived as a special case, resolved in a second pass.

The proposed dispatch model — exactly one backend per scheme — makes this less an "order" question and more a "default scheme" question. Most credentials should reference the centralised store directly. The order matters when a credential could be looked up multiple ways, in which case the discipline says: prefer the form your secret-handling auditor would prefer.

Pre-load vs post-lookup resolution

The resolver pattern can fit at two seams:

Pre-load resolution (recommended default)

Resolve all references before calling CredentialManager.load_credentials. The materialised values become the credential field values; the manager and the rest of the orchestration never see the references.

# orchestration startup
resolver = build_resolver(...)
with open("config/credentials/credentials-staging-east.yaml", "r") as fh:
    raw = yaml.safe_load(fh)

# Resolve every ${SCHEME:path} in the raw dict.  resolved_credentials
# now contains literal secret strings.
resolved = resolver.resolve_dict(raw)

# Hand the resolved dict to the manager.  The manager doesn't know
# that the values were ever references.
mgr = CredentialManagerSingleton(
    credential_info=resolved,
    credential_uris=["config/credentials/credentials-staging-east.yaml"],
)

Pre-load resolution is the recommended default because:

• The manager and the rest of the orchestration are simpler.
• Failure to resolve fails the orchestration at startup, not at the moment of first credential use.
• The audit log of "which secrets were fetched at startup" is a single resolved-batch event, not a stream of per-lookup events.

Post-lookup resolution

Resolve references on demand the first time a credential is looked up. Wrap the manager:

class LazyResolvingManager:
    def __init__(self, raw_manager: CredentialManager, resolver: CredentialResolver):
        self._raw = raw_manager
        self._resolver = resolver
        self._resolved_cache = {}
        return

    def lookup_credential(self, key: str):
        if key in self._resolved_cache:
            return self._resolved_cache[key]
        cred = self._raw.lookup_credential(key)
        # walk cred.as_dict(), resolve ${SCHEME:path} fields, rebuild
        materialised = self._materialise(cred)
        self._resolved_cache[key] = materialised
        return materialised

Post-lookup resolution is the right answer when:

• The orchestration uses only a small fraction of declared credentials per run. Resolving everything at startup is wasted work.
• Secrets are short-lived and need to be re-fetched periodically (the cache becomes a TTL cache; the next chapter on rotation builds on this).
• The startup time budget is tight and the resolver is slow (some vault backends have noticeable latency at first connect).

Most teams should start with pre-load resolution and only move to post-lookup if a specific need surfaces.

Host scoping

The shipped codecraftai-credentials package has no notion of host scope — a credential applies wherever its identifier is referenced. This is sometimes too permissive. A credential that lets you SSH to the staging runner pool should not silently work against the production runner pool just because both apods reference it by the same identifier.

The proposed discipline: extend the categories tag with host-scope annotations, and have the resolver enforce them at lookup time.

- identifier: ci-runner-ssh
  categories:
    - ssh
    - host-scope:staging-east
    - runner-fleet

The resolver wraps lookup_credential to check that the active landscape's label matches the host-scope annotation:

def lookup_credential(self, key: str):
    cred = self._raw.lookup_credential(key)
    scopes = [c.split(":", 1)[1] for c in cred.categories if c.startswith("host-scope:")]
    if scopes:
        active_landscape = LandscapeSingleton().label
        if active_landscape not in scopes:
            raise ConfigurationError(
                f"Credential {key} is scoped to {scopes} but active landscape is {active_landscape}"
            )
    return cred

This is a discipline addition the reader builds. It is one example of why having the resolver layer be yours matters: the host-scoping rule that fits your environment is not the same as the rule that fits a different team's environment.

Putting it together: a complete resolver

A complete CredentialResolver plus a small set of backends, wired into orchestration startup, looks like this:

import os
import re
import yaml
from typing import Any, Dict, List, Optional

# ... CredentialBackend, FileBackend, EnvBackend, VaultBackend
# ... defined as in the recipes above

REFERENCE_RE = re.compile(r"^\$\{([A-Z]+):(.+)\}$")


class CredentialResolver:
    def __init__(self, backends: List[CredentialBackend]):
        self._scheme_to_backend = {b.SCHEME: b for b in backends}
        return

    def resolve_dict(self, data: Dict[str, Any]) -> Dict[str, Any]:
        return self._walk(data)

    def _walk(self, value):
        if isinstance(value, dict):
            return {k: self._walk(v) for k, v in value.items()}
        if isinstance(value, list):
            return [self._walk(v) for v in value]
        if isinstance(value, str):
            return self._resolve_string(value)
        return value

    def _resolve_string(self, raw: str) -> str:
        m = REFERENCE_RE.match(raw)
        if not m:
            return raw  # not a reference, leave alone
        scheme, path = m.group(1), m.group(2)
        backend = self._scheme_to_backend.get(scheme)
        if backend is None:
            raise ConfigurationError(
                f"No backend registered for scheme {scheme} in reference {raw}"
            )
        ref = CredentialReference(scheme=scheme, path=path, raw=raw)
        value = backend.resolve(ref)
        if value is None:
            raise ConfigurationError(
                f"Backend {scheme} could not resolve reference {raw}"
            )
        return value


def build_resolver_for_environment(profile: str) -> CredentialResolver:
    """
        Build the resolver for a particular environment profile.
        The set of backends differs per profile — this is the
        environment-specific bit (chapter 12 expands on this).
    """
    if profile == "ci":
        return CredentialResolver([
            VaultBackend(vault_addr=os.environ["VAULT_ADDR"],
                         vault_token=os.environ["VAULT_TOKEN"]),
            FileBackend(),
            EnvBackend(),
        ])
    elif profile == "local-dev":
        return CredentialResolver([
            KeychainBackend(),
            FileBackend(),
            PromptBackend(),
        ])
    elif profile == "staging":
        return CredentialResolver([
            VaultBackend(vault_addr=...,
                         vault_token=...),
            FileBackend(),
        ])
    else:
        raise ConfigurationError(f"Unknown profile {profile!r}")


# Wired into orchestration startup
def bootstrap_credentials(profile: str, credentials_path: str) -> None:
    with open(credentials_path, "r") as fh:
        raw = yaml.safe_load(fh)

    resolver = build_resolver_for_environment(profile)
    resolved = resolver.resolve_dict(raw)

    CredentialManagerSingleton(
        credential_info=resolved,
        credential_uris=[credentials_path],
    )
    return

Sixty lines — including the reference-parsing regex, the recursive walk, the per-backend dispatch, and the per-environment configuration — for a complete resolver. The same sixty lines work for every project in your portfolio.

What the resolver does not do

Three things to keep out of the resolver:

Not authentication

The resolver fetches secrets. It does not perform authentication against systems using those secrets. A BasicCredential whose password the resolver materialised is still inert; the consumer's HTTP client is what authenticates.

This matters because the resolver runs once per orchestration startup, but authentication runs every time a credential is used. Conflating the two would force authentication failures at the wrong seam.

Not access-control policy

The resolver fetches secrets. It does not decide whether the caller is allowed to fetch a particular secret. That is a property of the backend's own access control (Vault policies, IAM roles, KV store ACLs).

A team's credential discipline should rely on the backend's permission system to enforce who-can-read-what, not on the resolver. When the resolver fails to fetch (because the caller lacks permission), the backend returns the failure; the resolver surfaces it as a clear error.

Not rotation

The resolver fetches the current value at the time it runs. When a credential rotates, the cached resolved value goes stale. The chapter on credentials in flight (chapter 11) is the rotation discipline; the resolver is one of the building blocks but not the whole story.

For long-running orchestrations (anything over an hour), the post-lookup resolution model with TTL caching becomes important; chapter 11 expands on the pattern.

The migration path

For a team adopting the discipline against an existing pipeline:

1. Stand up one backend. Pick the one that matches your environment — Vault if you have it, AWS Secrets Manager if you're on AWS, file backend backed by Kubernetes Secrets if you run on K8s. Do not try to ship six backends on day one.
2. Migrate the credential file to use the ${SCHEME:path} syntax for one credential at a time. Keep the old environment variables as a fallback during migration by wiring the EnvBackend.
3. Validate at every step. Each migrated credential is a separate change; each change is reviewable; each change can be rolled back independently.
4. Remove the EnvBackend once every credential has migrated.
5. Add host-scoping annotations as a second pass once the primary backends are stable.

The migration takes weeks, not days. This is fine — the discipline compounds over time. A single rotated production credential that didn't leak through environment variables to a CI log is, by itself, worth the migration.

Summary

• The codecraftai-credentials package treats secret values as opaque strings. Resolution — turning ${VAULT:/path} into the actual secret — is the consumer's responsibility, deliberately so.
• The proposed reference syntax is ${SCHEME:path}, with each scheme corresponding to a backend. The SCHEME is greppable, the path is opaque to the parser, and the form is loud enough to notice if it leaks into a log.
• The backend interface is small: one resolve(reference) method returning the materialised value or None. Adding a backend is a new subclass and a new dispatch entry.
• Common backends a reader will build: File, Env (legacy bridge), Prompt (local dev), Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, Keychain (macOS local dev), Derived (cred-from-cred).
• Pre-load resolution (resolve before load_credentials) is the recommended default; post-lookup resolution is the answer when rotation or sparse usage matters.
• Host-scoping is a discipline addition the reader builds on top of the categories vocabulary; the resolver enforces it at lookup time.
• The complete resolver — regex, recursive walk, per-environment backend selection, orchestration wiring — is roughly sixty lines and is reusable across projects.
• The resolver is not authentication, not access-control policy, and not rotation. Each is a distinct concern with its own seam.

The next chapter is credentials in flight — rotation during long runs, encryption at rest of credential files, audit trails, and the "never log a credential" discipline made operational.