Daemon high availability

Agor daemon HA is an explicit, constrained deployment mode. Independently started daemon replicas share PostgreSQL as the durable state and coordination plane and Redis as the Socket.IO/Feathers notification plane. Redis is not a task lock, tenant authority, or durable event log.

The accepted profile is constrained-active-active. Adding REDIS_URL to a standalone daemon does nothing: HA activates only with deployment.mode: ha (or AGOR_DEPLOYMENT_MODE=ha) and fails startup unless every required assertion passes.

The checked-in runnable variant is a development-only, auth-resolved multi-tenant smoke environment. Its identity picker deliberately allows impersonating fixed test personas; tenant propagation after the picker uses the normal signed external-launch and Agor JWT boundaries.

Required topology

Every replica needs:

• the same PostgreSQL database and completed HA migrations;
• the same private Redis deployment and a deployment-unique Redis prefix;
• identical, explicitly supplied AGOR_JWT_SECRET and AGOR_MASTER_SECRET values (32+ characters);
• Engine.IO session affinity at ingress while polling remains enabled;
• execution.allow_web_terminal: false for external; shared-local may opt into owner-local ephemeral terminals;
• execution.managed_envs_execution_mode: webhook-only, or the explicitly configured bounded external command profile; and
• one explicit execution topology.

HA Codex and Claude credential operations additionally require Linux flock (provided by the util-linux package) and an explicit user_home_locking: cross-replica-flock operator assertion. The official image includes the executable. Without both requirements, auth-file operations fail closed rather than using a local-only or age-stealable filesystem lock. A delegated/external executor image must provide /usr/bin/flock itself.

Executor storage contract

shared_filesystem is not a blanket requirement for every HA or Agor Cloud deployment. It is an assertion tied to execution_topology:

• shared-local: either daemon may launch a local executor against a branch working copy, so the working-copy/repository paths must exist at the same path on both daemon containers. The Compose smoke stack mounts agor-ha-user-home at /home/agor and a nested, stable agor-ha-home at /home/agor/.agor. Its sandbox overlays a durable tenant/user-specific directory from ~/.agor/tenants/<tenant>/homes/<user> as the executor home, rather than exposing the shared daemon home as the user’s credential home. Registered repositories and branch workspaces remain shared at stable paths. These volumes preserve files, not the executor process: this topology does not by itself guarantee that an executor survives loss of its container. shared_filesystem: true is required.
• external: daemon pods do not own tenant workspaces. execution.executor_command_template sends work to the external execution substrate. shared_filesystem must be false/omitted. This is the intended Cloud shape.

HA also requires an explicit execution.executor_storage contract. These values do not create volumes or mounts; they tell Agor what the launcher or Compose topology guarantees for every executor invocation:

• user_home: replica-local, shared, or persistent-per-user;
• user_home_locking: local-only or cross-replica-flock;
• branch_workspace: replica-local, shared, or persistent-per-branch; and
• base_repository: replica-local, shared, or unavailable.

persistent-per-user means the same isolated home is selected from trusted tenant/user identity for auth helpers, Tasks, replicas, and restarts. persistent-per-branch means the same durable working copy is mounted at the absolute branch.path recorded in PostgreSQL. Executor homes and branches may be separate volumes; they do not need to be one global shared filesystem. An auth-resolved HA deployment must use persistent-per-user; startup rejects a shared or replica-local credential home rather than allowing users or tenants to share CLI credentials.

When execution.sandbox.sdk_home_mode: per_branch is enabled, the tenant data root’s branch-homes/<branch-id>/ tree carries each branch’s relocated SDK config/state. Each Session is immutably stamped to use either that tree or its historical execution home; every replica and delegated launcher must honor the Session stamp rather than inferring the choice from current deployment or branch configuration. OpenCode is refused for branch-scoped Sessions because its native credential file cannot currently be separated from its writable XDG state. Local Codex subscription auth is supported only with the per-user sandbox: each replica opens the exact caller’s persistent auth.json and projects that pinned inode into the branch home for one executor without copying it. Credential replacement/logout records a durable termination request for every affected actor Task, including Tasks owned by another replica; the owning executor cooperatively quiesces through the normal HA termination path. A delegated substrate owns any equivalent credential overlay. The tree needs the same shared-storage and cross-replica-flock guarantee documented above for homes/: a prompt or Task dispatched to replica B must see exactly what replica A wrote, and concurrent writers within one branch home rely on the same cross-client flock semantics. Agor does not add a separate executor_storage key for it — place branch-homes/ on the same backing store as homes//worktrees/, at the same absolute path on every replica.

cross-replica-flock asserts that an exclusive flock on one user’s home is observed by every daemon/storage client that can mutate that home. Agor cannot infer this reliably from a mount path. Do not assert it for NFS mounted with local_lock, or for NFS/CIFS/storage configurations whose cross-client locking has not been verified. local-only (and omission) keeps HA Codex and Claude credential operations unavailable while leaving unrelated HA capabilities usable.

When base_repository: unavailable, native worktrees are invalid because their .git file points into the registered base repository’s .git/worktrees metadata. Agor therefore requires clone-only branch policy. A clone-mode branch carries its own .git directory and can be mounted without the base checkout. Requiring full rather than shallow clones is a separate policy.

Auth-resolved multi-tenancy also requires an explicit clone-only branch policy, even when a shared-local topology exposes the base repository. Native worktrees share repository metadata across tenant working copies, so hosted mode rejects them. Set default_mode: clone as well as allowed_modes: [clone]; the server-owned default covers onboarding and API callers that omit storage_mode.

PostgreSQL already owns repo/branch metadata and most coordination, but local worktrees, agent runtime homes, and tools that inspect a working directory are still a filesystem tail. The flag prevents a two-daemon deployment from claiming safe local execution when only one replica can see those paths. config.yaml is separately mounted read-only in the example because current bootstrap treats an existing config as immutable.

How sticky Socket.IO ingress works

A persisted WebSocket is an open TCP connection terminated by one daemon through the proxy. The proxy does not copy live connection state to another daemon. Redis lets either daemon deliver packets to clients attached to the other daemon.

Before WebSocket upgrade, Engine.IO may use multiple HTTP polling requests with one session ID. All /socket.io/ requests for that logical connection must therefore reach the same daemon. The example nginx configuration uses consistent client-IP hashing for /socket.io/, forwards upgrade headers, and sets a 120-second proxy timeout (above Agor’s 85-second heartbeat window). Ordinary HTTP/REST uses a separate round-robin upstream and is not sticky.

If a daemon dies, its existing sockets close; their Engine.IO session IDs do not migrate. Socket.IO/Feathers clients reconnect to a healthy daemon with a new authenticated namespace handshake, then refetch PostgreSQL-backed state. The Redis adapter does not provide connection-state recovery or missed-event replay.

For Agor Cloud, configure the platform load balancer or ingress so that:

1. /socket.io/ supports polling and WebSocket upgrades;
2. affinity is cookie-, connection-, or source-based for the whole Engine.IO session (cookie affinity is preferable when many clients share one NAT address);
3. idle/read timeouts exceed 85 seconds;
4. only ready pods receive new sessions; and
5. ordinary REST routes remain free to balance across ready pods; and
6. authentication, refresh, and external-launch abuse limits are enforced at a shared edge/WAF layer rather than multiplied per daemon.

The Compose IP hash is deliberately minimal and testable, not a recommendation for large NAT-heavy fleets.

Compose readiness limitation: Docker health checks gate the example’s initial nginx startup, but open-source nginx does not continuously consume later Docker health-state changes for its static upstream list. It can passively fail over after a connection error, but /readyz: 503 alone does not remove a still-listening daemon from this smoke ingress. Production ingress/orchestration must actively route only to ready replicas as required above.

Runnable Compose smoke variant

docker-compose.ha.yml uses explicitly named daemon-a and daemon-b, PostgreSQL, fanout-only Redis (RDB and AOF disabled), a one-shot preparation service that reconciles the declared agentic tools and then runs migrations, a Compose-only development launch issuer, and nginx. It does not use deploy.replicas. Because the example has exactly one nginx ingress, nginx also provides a coarse shared per-IP limiter for authentication/refresh/launch paths and logs the selected upstream. That limiter proves the fleet choke-point topology; it is not a credential-aware production quota and would not be shared across multiple nginx replicas.

The checked-in HA smoke configuration declares clone-only branch storage and every first-party managed agentic-tool integration so the harness can exercise each one under HA. Before either daemon starts, the preparation job runs agor install --sync into the shared /home/agor/.agor volume. Production deployments should instead declare the exact agentic_tools.installed list they support; changing that list requires rerunning preparation with package-registry access. Agentic-tool installation and user credentials are separate: users still need valid credentials for the selected integration.

export AGOR_REDIS_KEY_PREFIX=my-company-production
export AGOR_JWT_SECRET="$(openssl rand -hex 32)"
export AGOR_MASTER_SECRET="$(openssl rand -hex 32)"
export AGOR_EXTERNAL_LAUNCH_SHARED_SECRET="$(openssl rand -hex 32)"
export HA_PORT=3030
export AGOR_HA_PUBLIC_ORIGIN=http://localhost:3030
 
docker compose -f docker-compose.ha.yml -p agor-ha up -d --build
docker compose -f docker-compose.ha.yml -p agor-ha ps
curl -fsS http://localhost:3030/livez
curl -fsS http://localhost:3030/readyz
docker compose -f docker-compose.ha.yml -p agor-ha logs --tail=150 daemon-a daemon-b redis dev-launcher ingress

AGOR_HA_PUBLIC_ORIGIN must be the exact scheme, host, and port that browsers use for the ingress. The Compose file applies it as both the daemon public base URL and the credentialed CORS origin on every replica. The checked-in .agor.yml ha variant honors an inherited value (needed when a public HTTPS proxy fronts the branch ingress), or otherwise renders a development HTTP value from Agor’s host address and branch port. OAuth on PostgreSQL/HA requires the former to be a safe browser-reachable HTTPS origin; the development HTTP default cannot start a provider flow. The Agor Cloud rollout therefore remains dependent on agor-cloud#468 provisioning that public origin contract. Until it does, mcpOAuth reports false and OAuth start refuses before provider discovery. daemon.public_url is reserved for internal daemon/executor callbacks and is never an OAuth browser redirect origin.

The Agor-managed ha variant is intentionally disposable. Opening it lands on /dev-auth/, a separate sidecar page where the developer chooses Acme or Globex and an admin or member persona. The picker requires no password. It creates a one-time code and signs the selected identity for Agor’s existing external-launch exchange; Agor then JIT-provisions the tenant-scoped user and issues its normal access/refresh JWTs. Use a normal and incognito window (or two browser profiles) to keep two personas active simultaneously because the Agor tokens are stored per browser origin.

The sidecar is mounted only by the checked-in Compose file and is not copied into the daemon or UI product images. Do not expose this smoke stack to an untrusted network or copy its fixed personas, development assertion secret, or passwordless picker into a production deployment.

Compose volumes survive up and ordinary Agor environment stop/start operations, so JIT-projected personas and their tenant data survive too. Nuke the disposable environment to start from empty Acme and Globex tenants.

In-place development cutover: The ha variant previously used local authentication and the static default tenant. It now deliberately models the Cloud identity boundary instead. Existing saved lifecycle commands remain usable because Compose has a development-only assertion-secret fallback, but rows in the old default tenant are not reassigned to Acme or Globex. Before first use of the new picker, nuke an existing disposable HA environment to start clean. Preserve or migrate anything valuable explicitly rather than expecting a selected persona to see it.

Keep generated secrets in a secret manager and rotate them as an all-replica operation. The example’s unencrypted Redis connection stays on its private Compose network. Across a shared or untrusted network, use rediss://, certificate verification, Redis ACL authentication, private networking/firewalls, and credentials limited to the required Pub/Sub/control commands and deployment channels.

General HA startup requires an explicit AGOR_ADMIN_PASSWORD unless external-launch authentication owns bootstrap. The checked-in Compose profile takes the latter path: external identity authority skips the local first-run account, and the first selected persona is projected inside its signed tenant scope.

Configuration contract

deployment:
  mode: ha
  redis:
    # Prefer REDIS_URL when credentials are present.
    url: rediss://redis.internal:6380/0
    # Share this only among replicas of one mutually trusted deployment.
    # Use a distinct prefix for every other Agor deployment using this Redis.
    key_prefix: my-company-production
    connect_timeout_ms: 5000
    startup_timeout_ms: 15000
    request_timeout_ms: 5000
    reconnect_base_delay_ms: 100
    reconnect_max_delay_ms: 2000
  ha:
    support_profile: constrained-active-active
    execution_topology: external # or shared-local
    ingress_affinity: true
    environment_health_monitor:
      scan_interval_ms: 5000
      max_idle_interval_ms: 30000
      startup_offset_max_ms: 3000
      scan_batch_size: 32
      max_in_flight: 8
      http_timeout_ms: 1000
      claim_lease_ms: 15000
      shutdown_drain_timeout_ms: 5000
execution:
  executor_command_template: 'cell launch --tenant {tenant_id} -- {command}'
  executor_response:
    max_response_bytes: 8388608
    max_active_requests: 16
    timeout_ms:
      default: 300000
      by_command:
        branch.files.read: 60000
    external_protocol: executor-response-v1
    # In practice prefer the per-replica environment override described below.
    origin_url: http://agor-daemon-0.agor-daemon-headless:3030
  executor_storage:
    user_home: persistent-per-user
    # Operator assertion: the backing store propagates flock across clients.
    user_home_locking: cross-replica-flock
    branch_workspace: persistent-per-branch
    base_repository: unavailable
  branch_storage:
    default_mode: clone
    allowed_modes: [clone]
    allow_shallow_clones: false
  allow_web_terminal: false
  managed_envs_execution_mode: webhook-only
database:
  dialect: postgresql
daemon:
  # Required for external executors to call/reconnect through the daemon fleet.
  public_url: https://agor-daemon.internal.example

key_prefix is the Socket.IO and internal realtime trust namespace, not a cosmetic label. All replicas in one deployment must use the same value, and independent deployments must use different values even if their tenant IDs do not currently overlap. Reusing it makes those deployments participants in the same Redis adapter/control plane.

daemon.public_url remains the fleet URL for ordinary executor Feathers and Socket.IO calls. It is not a safe request-response origin. Set AGOR_EXECUTOR_RESPONSE_ORIGIN_URL independently on every replica to an executor-reachable Pod IP or stable per-Pod DNS name; it overrides the shared YAML origin_url. The initiating replica owns the waiter and response bytes. Redis does not forward or persist them, and another replica does not take over after waiter loss. Timeout, disconnect, or replica restart fails the request without an automatic retry.

For shared-local, replace the external template with shared_filesystem: true and declare the actual storage guarantees; its executor callback defaults to the local daemon. The Compose smoke stack declares shared branch/base-repository storage, persistent-per-user home storage, and cross-replica flock semantics: agor-ha-user-home mounts /home/agor, while the historical agor-ha-home is nested at /home/agor/.agor so PostgreSQL workspace paths remain valid across upgrades. Sandbox mode keeps each tenant/user’s home store below that nested shared volume and overlays the exact store at ~ for its Tasks. Auth-file operations point to the same store directly, so either daemon and the later Codex or Claude Task agrees on .codex/auth.json or .claude/.credentials.json. Generation-fenced HA mutations run in the authority-owning daemon rather than a detached credential writer. Codex device and Claude paste-back attempts use PostgreSQL ownership; Redis never carries their codes, PKCE material, or credentials.

shared-local may set allow_web_terminal: true. A terminal create runs over the browser’s sticky Socket.IO connection and returns a daemon-boot-fenced, branch-scoped attachment. PTY traffic remains local to that replica. Replica loss is a visible disconnect; the user explicitly reconnects to create a new PTY bridge. There is no live PTY migration or Redis replay. external must keep terminals disabled until its terminal/workspace runtime provides an owner-affine callback contract.

External topology requires an HTTP(S) daemon.public_url that the execution substrate can reach through the daemon fleet. Environment overrides include AGOR_DEPLOYMENT_MODE, AGOR_HA_SUPPORT_PROFILE, AGOR_HA_EXECUTION_TOPOLOGY, AGOR_HA_SHARED_FILESYSTEM, AGOR_HA_INGRESS_AFFINITY, REDIS_URL, AGOR_REDIS_KEY_PREFIX, the AGOR_REDIS_*_MS timeout/backoff variables, and AGOR_HA_ENV_HEALTH_SCAN_INTERVAL_MS, AGOR_HA_ENV_HEALTH_MAX_IDLE_INTERVAL_MS, AGOR_HA_ENV_HEALTH_STARTUP_OFFSET_MAX_MS, AGOR_HA_ENV_HEALTH_SCAN_BATCH_SIZE, AGOR_HA_ENV_HEALTH_MAX_IN_FLIGHT, AGOR_HA_ENV_HEALTH_HTTP_TIMEOUT_MS, AGOR_HA_ENV_HEALTH_CLAIM_LEASE_MS, and AGOR_HA_ENV_HEALTH_SHUTDOWN_DRAIN_TIMEOUT_MS. The health claim lease must be at least five seconds longer than the HTTP timeout. Storage guarantees remain YAML-only so an accidental environment toggle cannot claim that mounts exist.

The Redis prefix separates deployments operationally. It is never tenant authorization.

Health and failure semantics

Realtime events are at-most-once notifications across an outage window. Exactly-once durable mutation, work claims, task identity, and gateway occurrence admission are PostgreSQL concerns.

Security and event inventory

• The maintained @socket.io/redis-adapter carries complete Socket.IO packets across trusted Redis Pub/Sub. Redis traffic is neither signed nor encrypted by the adapter.
• Feathers channel membership is process-local and is not replicated by the adapter. Agor sends a bounded JSON relay envelope (tenantId, service path/event metadata, dispatched payload) with serverSideEmit. Each receiving daemon reruns tenant and branch/user publication logic against its own connections before entering the Feathers dispatcher.
• Authentication, refresh, raw session-token/API-key resolution, external-launch credentials, MCP OAuth/token endpoints, Codex/OpenCode auth, terminals, and secret-derived Artifact query results are denied from the Feathers Redis relay. After-hooks’ redacted dispatch value is relayed instead of an unredacted service result.
• Native cross-replica packets in this profile include cursor-moved, cursor-left, presence-updated, presence-left, tenant-scoped repo:cloneError, tenant/branch terminal cleanup metadata, and tenant-qualified MCP OAuth completion/disconnect/Catalog hints. Board-bearing presence packets target only tenant-and-board-qualified rooms admitted through the normal board visibility service. server-info is socket-local. OAuth packets are UX hints only: PostgreSQL attempts, grants, and client-registration leases remain authoritative; supported Codex/Claude auth flows use PostgreSQL and direct responses rather than Redis OAuth packets. Terminal input/output is always socket-local and must never enter Redis.
• Ordinary authorized Feathers data, including transcript/message streaming, may traverse the private Redis plane. Do not deliberately place bearer/API/executor/MCP/terminal tokens, OAuth/PKCE/device credentials, GitHub setup state, external-launch credentials, Artifact grants/environment, or secret-derived results in realtime payloads.
• Cursor-room joins call the authenticated Feathers boards.get path. Navbar board-association subscriptions call authenticated boards.find, silently omit unavailable IDs, and join a distinct low-frequency tenant-and-board-qualified room. Authentication replacement/logout revokes prior tenant/user/board room membership.
• Per-socket and daemon-local HTTP rate limits are process-local best effort, not a fleet security quota. The single-ingress Compose limiter is a coarse topology proof. Production must enforce security-sensitive quotas at a shared edge/WAF or durable authority layer.

No current product cross-replica notification requests client broadcast acknowledgements. Feathers method callbacks terminate on the daemon that handled the method. The adapter supports broadcast acknowledgements, but any future product event using them needs explicit multi-replica tests before support is claimed.

Managed-environment observation semantics

Each active starting or running branch is eligible for bounded periodic discovery. One replica holds a PostgreSQL lease for one observation. Releasing a completed observation writes a durable database-time cooldown, so another replica cannot multiply the configured polling cadence. If an owner dies without releasing, another replica can take over after database-time lease expiry even when a prior cooldown exists. Every result commit locks and rereads the branch and rejects an expired or replaced token, changed lifecycle generation, inactive status, or archived row. No database transaction remains open during the HTTP request.

• A healthy starting observation promotes the environment to running only after any active lifecycle command has settled.
• A reachable non-success response is recorded as unhealthy. A network error or timeout during starting preserves the existing startup grace behavior; the same failure during running is recorded as unhealthy, without changing the lifecycle status.
• A missing health URL is recorded as unknown. A URL rejected by the health-check security policy is recorded as unhealthy. URLs and embedded credentials are not logged by the worker.
• Branch events only wake discovery early. Periodic scans recover missed events and newly eligible rows.
• A coordination failure makes /readyz return 503. The loop retries with bounded jitter/backoff; the first successful scan restores readiness. Endpoint health results never change daemon readiness.
• Graceful shutdown stops new discovery, aborts HTTP work, drains within the configured bound, and conditionally releases current tokens when time remains. Otherwise lease expiry is authoritative.

Current support matrix

[!WARNING] Codex device-attempt, import, and logout writers are generation-fenced, but the Codex provider runtime can still read and rewrite its canonical .codex/auth.json. It does not yet have Claude’s canonical-file mask plus daemon-issued short-lived-token containment. That runtime-writer parity risk remains a separate follow-up; this change neither fixes it nor changes the existing Codex capability behavior.

Rollout and rollback checklist

1. Quiesce OAuth exchanges/refreshes and stop every daemon, then back up PostgreSQL and any shared-local workspace volume.

2. Apply all migrations before admitting new-version daemons. Migration 0077_environment_health_ha adds branch observation leases/generations and the narrow discovery policy. Migration 0078_mcp_oauth_pending_flows replaces legacy MCP OAuth grant/callback authority, migration 0082_github_install_state moves one-time GitHub setup state into PostgreSQL, migration 0091_codex_device_auth_attempts adds Codex poll leases plus completion generations, migration 0100_claude_oauth_attempts adds Claude paste-back attempt and mutation authority, PostgreSQL migration 0102_mcp_oauth_client_registrations adds the encrypted fleet-wide DCR lease/CAS authority, and 0103_oauth_authority_watermark_reconciliation repairs the old b0585d76 timestamp collision only after matching its complete archived schema fingerprint and exactly one applied migration-ledger row at the legacy watermark with the archived SHA-256 hash (discarding those old registrations so reconnect is fresh). It also verifies the complete final DCR and Claude authority schemas and fails closed on malformed or future shapes. SQLite intentionally has no DCR registration schema and retains process-local standalone registration. These protocol migrations require agor db migrate --yes --offline-cutover on an existing database. Use an all-at-once cohort replacement: old daemons lack the registration/attempt and credential-file fencing and must not overlap or advertise the new capabilities. The later 0103 watermark makes pre-final daemons reject the migrated database as ahead. Mixed relay revisions intentionally do not exchange events. Main’s 0101_environment_command_discovery stays unchanged. Both OAuth timestamps follow it. The 0102 bootstrap leaves any existing DCR relation for 0103’s exact validation in the same offline transaction, preserving already-final rows while retaining the archived legacy repair and malformed-schema refusal. This also permits upgrading the previously reviewed b058ea35 head without recreating its DCR table.

   Migration 0105_mcp_oauth_grant_attribution also requires an offline cutover and retires historical shared MCP OAuth grants. Follow the shared-grant upgrade checklist: inventory affected servers per tenant, identify reauthorization administrators, quiesce OAuth exchanges/refreshes before backup and migration, and verify each server after cutover. Restoring a backup cannot undo provider-side authorization-code consumption or token rotation.

3. Provision private Redis with health monitoring, ACL/TLS/network controls, and a unique prefix. Do not rely on Redis for persistence/replay.

4. Distribute identical stable config and secrets. Set stable AGOR_DAEMON_INSTANCE_ID values for diagnostics only.

5. Choose and verify shared-local or external; declare the executor storage contract from actual mounts, and do not set shared_filesystem merely because HA is enabled. Before using base_repository: unavailable, inventory historical branches for worktree rows.

6. Verify /socket.io/ polling-to-WebSocket affinity and REST distribution separately.

7. Configure /readyz for traffic removal and /livez for process restart decisions.

8. Confirm an authenticated Feathers health.find on every replica reports mode: ha, supportProfile: constrained-active-active, the expected fine-grained capability booleans, distinct instance/boot diagnostics, and ready Redis clients. In particular, do not interpret completionCallbackDurableAdmission: true as pre-admission recovery or gatewayListeners: true as outbound exactly-once; the corresponding negative capability flags remain visible. codexCredentialFiles reflects replica consistency and tenant safety; codexDeviceAuth requires an exact-user sandbox or delegated route with persistent-per-user and cross-replica locking. claudeOAuth and claudeAuth additionally require shared-local execution without an executor template and the concrete per-user bubblewrap immutable-parent and authority-leaf masks. The default-off provider-policy flag independently controls whether the OAuth endpoint and UI are advertised. environmentHealthMonitor: true confirms that the fenced observer is advertised, and githubInstall: true confirms that the durable setup callback is admitted. The public HTTP GET /health intentionally exposes only the configured instance label and dependency summary.

9. Exercise daemon loss, Redis loss/recovery, cross-replica authorized delivery, and negative authorization tests before production traffic.

10. Validate only the gateway providers and task permission modes you plan to enable; “daemon starts” is not full product HA certification.

To roll back the binaries, stop the HA cohort and remove every new daemon from ingress. Agor refuses to start when the database migration watermark is newer than the binary’s journal, so a binary that predates final 0103 cannot simply run against a database where 0103_oauth_authority_watermark_reconciliation was applied. Restore the tested pre-cutover backup, or use a deployment-specific coordinated schema-and-ledger rollback with every daemon stopped. Do not bypass that refusal. If the selected rollback binary still knows the current migration watermark, any abandoned 0091 Codex, 0100 Claude, and 0102 DCR rows remain inert only because its constrained-HA capability guards stay off; cancel or invalidate attempts before later re-upgrading. Never overlap cohorts or advertise MCP/Claude HA OAuth from a binary that does not own the corresponding generation, lease, and CAS protocols.

Retaining additive tables is safe only for a rollback binary whose journal includes their migration watermark. A deliberate rollback below that watermark is a separate recovery procedure with every old and new daemon stopped: restore a tested pre-migration backup, or use a deployment-specific, tested procedure that coordinates both the reverse DDL and the corresponding drizzle.__drizzle_migrations ledger entries. Removing any schema without its matching ledger change leaves the migration recorded as applied, so a later upgrade will not recreate it. Reverse 0078, 0100, 0102, or 0103 only through a tested backup restore or coordinated schema-and-ledger procedure. REDIS_URL may stay present because standalone ignores it. Do not rotate shared secrets or switch database engines during an emergency rollback.

Opt-in integration harness

The repository harness is intentionally not part of the default test run. Against only the checked-in HA Compose project it proves two ready identities and activation of the distributed environment-health loop on both replicas (not occurrence-level fencing), activation of shared Task queue/reconciler loops, REST distribution, JIT login for Acme and Globex through the picker/exchange boundary, cross-tenant board read/watch/event denial, same-tenant private-board denial, cross-replica one-shot GitHub setup callback consumption plus raw-state exclusion from nginx access and upstream-failure logs, an actual 429 from the single-ingress sensitive-route limiter, fail-closed opaque personal API keys in auth-claim-only tenancy, polling upgrade affinity, authorized Feathers and native-room delivery with a bounded duplicate-observation window, anonymous denial, device/Claude-attempt status on both replicas, authenticated daemon reconnect/restart, and Redis readiness/liveness loss and recovery. Both replicas must report an identical capability matrix, including the containment-dependent Claude flags. It writes a simulated Codex auth file through A, inspects it through B’s executor route, logs out through B, and observes it absent through A. It also starts, wrong-state submits, and races replacement Claude attempts across replicas; external Claude source and execution-home changes fence the winner, the route change deletes the old canonical credential before publishing the override, and restoring the canonical route leaves cross-replica logout working. It never completes a real provider attempt and asserts the simulated tokens are absent from daemon/ingress logs. PostgreSQL two-pool tests additionally prove refresh-versus-login/logout/route ordering, loser adoption, Claude/Codex claim and replacement, source-change, cross-provider invalidation, removal-first/finalize-first deletion, standalone/HA generation transitions, canonical non-aliasing, generation, and file-lock races; a standalone test exercises actual delegated-home-key reuse through the global queue. Cleanup runs in finally, restores stopped branch services, and removes any created auth file, API key, or tenant-owned board after both success and assertion failure.

export COMPOSE_PROJECT_NAME=agor-ha-integration
export AGOR_REDIS_KEY_PREFIX=agor-ha-integration
export AGOR_JWT_SECRET='replace-with-32-plus-stable-characters'
export AGOR_MASTER_SECRET='replace-with-32-plus-stable-characters'
export AGOR_EXTERNAL_LAUNCH_SHARED_SECRET='replace-with-a-development-assertion-secret'
export AGOR_HA_PUBLIC_ORIGIN=http://localhost:3030
export AGOR_HA_INTEGRATION=1
export AGOR_HA_INTEGRATION_START=1
export AGOR_HA_INTEGRATION_FAILURES=1
pnpm test:ha:docker

The source image intentionally contains no authenticated provider CLI/runtime, so the live harness does not fake an agent Task. PostgreSQL-gated daemon tests instead construct two queue workers/Task services on independent PostgreSQL clients, elect one dispatch intent for one admitted Task, exercise executor-token authentication/reconnect/revocation/bounded use through two Feathers apps backed by independent PostgreSQL clients, and exercise two reconcilers. Passing this harness is a transport, control-plane, and tenant-boundary smoke result—not a claim that the shared-local container preserves an in-flight executor, matches Cloud’s external execution substrate, covers every gateway provider/task permission mode, or makes blocked process-affine features HA-safe.