Chapter 12 — Data Profiles in Remote Workers

Volume 3 chapter 8 walks data profiles at the orchestration layer. This chapter walks the analogue at the job layer: when a distributed test job needs to talk to a database, a message queue, or a cache, what does the framework do? The answer is short — it follows the same pattern as credentials from chapter 11, applied to a different category.

A data profile in the codecraftai-* portfolio is a connection profile for a data store. Where a credential answers "how do I authenticate," a data profile answers "where do I connect." The two go together — most data-store calls need both — but they're different categories with different configuration files.

This chapter is short. The discipline that holds for credentials (chapter 11) holds for data profiles, with the same shape and the same boundaries. The brevity is the payoff of universal patterns: when the same shape applies, the documentation is the same shape too.

Declared by name, resolved at activation, never a hardcoded URI

A data profile is declared in the Landscape's services section (or, equivalently, referenced from a device or as a top-level Landscape entity, depending on the project's convention):

infrastructure:
    services:
        - serviceType: database/postgres
          name: orders-db
          connection_profile: orders-db-profile

The connection_profile: orders-db-profile is a named reference to a data profile. The actual connection details (host, port, database name, schema, connection-pool size) live in the Data Profiles configuration category (Volume 3 chapter 8) — composable via MergeMap like every other category.

A representative data profile file:

# data-profiles/orders-db-profile.yaml
name: orders-db-profile
type: database/postgres
host: db.staging-east.internal
port: 5432
database: orders
schema: public
pool:
    min_connections: 2
    max_connections: 10
    timeout_seconds: 30

The file has no credentials. Authentication is a separate concern (chapter 11); the data profile says only "where to connect." Credentials are referenced from the same service declaration:

- serviceType: database/postgres
  name: orders-db
  connection_profile: orders-db-profile
  credentials:
      - orders-db-app    # ← the role the test connects as

Two named references in the service declaration. The framework resolves both at activation; the test gets a connection-ready service object without ever seeing host strings or auth material in test code.

The DataProfileManager — the parallel to CredentialManager

DataProfileManager (analogous to CredentialManager from chapter 11; in codecraftai-dataprofiles) is the job's data-profile lookup service. Initialised during landscape activation. Holds the resolved profiles for the duration of the job.

manager = LandscapeSingleton().dataprofiles_manager
profile = manager.lookup_by_name("orders-db-profile")

# profile carries the connection details
db_host = profile.host
db_port = profile.port

In practice the test rarely calls lookup_by_name directly. The framework wires the profile through the service's ProtocolExtension:

service = LandscapeSingleton().service("orders-db")

# The service has a 'db' ProtocolExtension wired by the
# database-Coordinator at activation time.
result = service.db.execute("SELECT count(*) FROM orders")

Same shape as device.ssh.run_cmd(...) from chapter 11 — the ProtocolExtension is the test's surface; the underlying machinery (connection pool, cursor, transaction) is hidden.

Composable data profile lists

Data profiles are composable like every other category. A service can reference multiple profiles:

- serviceType: database/postgres
  name: orders-db
  connection_profile:
      - orders-db-base
      - orders-db-staging-overrides

The framework merges them via MergeMap (later names override earlier names where keys collide) and the service uses the merged result. Common patterns:

• Base + environment overrides. A base-postgres-profile with sensible pool defaults; a staging-postgres-profile with staging-specific host and timeout adjustments.
• Pool size variants. A small-pool for unit-test databases; a large-pool for integration runs.
• Schema variants. A read-only profile pointing at a read replica; a read-write profile pointing at the primary.

The composability is what makes the four-environment ladder from Volume 3 chapter 13 work. Local-dev uses local-postgres + dev-overrides; CI uses local-postgres + ci-overrides; staging uses local-postgres + staging-overrides. Same suite of tests, four profiles, three differences.

What flows from the controller to the host

A connection profile leads to a connection from the controller to the data store. The flow:

• Controller resolves the profile via the DataProfileManager.
• Controller's data-store ProtocolExtension uses the profile's host / port / database / schema to construct a connection.
• Controller authenticates using the credential (chapter 11 describes this side).
• Connection is established; queries run from the controller to the store.

The data profile's content is not sensitive (the same way credentials are). Hosts under test that need their own connections to the data store typically do so through their own configuration — the framework's job is to give the controller a connection for its test code, not to push connection profiles to the hosts.

If a test exercises a host's connection to a data store (e.g., "does this app server connect to the database?"), the test:

• Uses the controller's connection (via the data profile) to inspect the database state.
• Uses the host's API or admin interface to check the host's connection state.
• Compares.

The framework's data profiles are the controller's, not the host's. The host has its own configuration story, owned by the project.

Per-test isolation through profile selection

A common pattern: a test wants its own sandbox connection. The framework supports this through profile-name selection at the test level:

@testbase.attach.parameter(orders_db_profile, identifier="db")
def test_can_query_orders(db):
    """
        Verify a query against the orders database.
    """
    rtnval = db.execute("SELECT count(*) FROM orders").rowcount
    assert rtnval >= 0
    return

The orders_db_profile here is the originator that yields a profile-bound connection. The Volume 1 chapter 3 resource-factory model applies: per-test scoping means each test gets a fresh connection (with whatever transaction isolation the database provides) and the connection is released at test end.

A project that wants per-test isolation more strongly (e.g., a fresh schema per test) writes a custom originator:

@testbase.factories.resource_factory(scope=ResourceLifespan.Test)
@testbase.attach.parameter(orders_db_profile, identifier="profile")
def isolated_db_session(profile):
    """
        Yields a per-test isolated DB session with a fresh schema.
    """
    schema = f"test_schema_{uuid.uuid4().hex[:8]}"
    conn = profile.connect()
    conn.execute(f"CREATE SCHEMA {schema}")
    conn.execute(f"SET search_path TO {schema}")
    try:
        yield conn
    finally:
        conn.execute(f"DROP SCHEMA {schema} CASCADE")
        conn.close()
    return

The originator is per-test (lifespan Test), gets the profile from the framework, creates a fresh schema, yields the connection, drops the schema on cleanup. The test sees a clean isolated database. The Volume 1 chapter 3 dependency graph composes the orders_db_profile (yields a profile) → isolated_db_session (yields a per-test connection) chain correctly.

Streaming-data profiles for queues and event buses

Data profiles aren't database-specific. Message queues, event buses, key-value stores, time-series databases all use the same shape:

# A Kafka topic profile
- serviceType: messaging/kafka
  name: orders-events
  connection_profile: orders-events-profile
  credentials:
      - kafka-consumer

# data-profiles/orders-events-profile.yaml
name: orders-events-profile
type: messaging/kafka
brokers:
    - kafka-1.staging-east.internal:9092
    - kafka-2.staging-east.internal:9092
topic: orders.events.v2
consumer_group: ci-test-runner
auto_offset_reset: latest

The framework's discipline is uniform: declare by name, resolve at activation, ProtocolExtension wires the connection, credential authenticates. The test uses service.kafka.consume(...) exactly as it uses service.db.execute(...).

What the framework does NOT do

A short list of things the framework deliberately stays out of, leaving them to the test or to the data store's own machinery:

• Connection pool management at the data-store level. The framework gives the test a connection (or a pool, depending on the ProtocolExtension); how the data store handles the pool is the data store's concern.
• Transaction boundaries. A test that wants explicit transaction control uses the connection's transaction API; the framework doesn't impose a per-test transaction envelope.
• Schema migrations. Bringing the database to a known state is test fixture territory. The framework can yield a connection; it doesn't run migrations.

These are project / domain concerns, not framework concerns. The framework's job is to connect; the project's job is to use.

What the next chapter does

Chapter 13 walks result aggregation — the JSOS streams from N workers across the Pod arriving at one coordinator, the preservation of per-host timelines, the failure-mode where one worker's stream stops mid-run. Where Volume 1 chapter 5 walked single-host streaming, chapter 13 walks the multi-host aggregation.