Daemon high availability

Agor daemon HA is an explicit, constrained deployment mode. Independently started daemon replicas share PostgreSQL as the durable state and coordination plane and Redis as the Socket.IO/Feathers notification plane. Redis is not a task lock, tenant authority, or durable event log.

The accepted profile is constrained-active-active. Adding REDIS_URL to a standalone daemon does nothing: HA activates only with deployment.mode: ha (or AGOR_DEPLOYMENT_MODE=ha) and fails startup unless every required assertion passes.

The checked-in runnable variant is static single-tenant. Realtime publication is tenant-qualified and has adversarial cross-tenant contract tests, but required_from_auth multi-tenant HA is not yet operationally certified.

Required topology

Every replica needs:

• the same PostgreSQL database and completed HA migrations;
• the same private Redis deployment and a deployment-unique Redis prefix;
• identical, explicitly supplied AGOR_JWT_SECRET and AGOR_MASTER_SECRET values (32+ characters);
• Engine.IO session affinity at ingress while polling remains enabled;
• execution.allow_web_terminal: false;
• execution.managed_envs_execution_mode: webhook-only; and
• one explicit execution topology.

Executor storage contract

shared_filesystem is not a blanket requirement for every HA or Agor Cloud deployment. It is an assertion tied to execution_topology:

• shared-local: either daemon may launch a local executor against a branch working copy, so the working-copy/repository paths must exist at the same path on both daemon containers. The Compose smoke stack mounts agor-ha-user-home at /home/agor for CLI-native credential files and a nested, stable agor-ha-home at /home/agor/.agor for registered repositories and branch workspaces. Keeping the historical Agor volume name also preserves the correspondence between PostgreSQL absolute paths and files across variant upgrades. These volumes preserve files, not the executor process: this topology does not by itself guarantee that an executor survives loss of its container. shared_filesystem: true is required.
• external: daemon pods do not own tenant workspaces. execution.executor_command_template sends work to the external execution substrate. shared_filesystem must be false/omitted. This is the intended Cloud shape.

HA also requires an explicit execution.executor_storage contract. These values do not create volumes or mounts; they tell Agor what the launcher or Compose topology guarantees for every executor invocation:

• user_home: replica-local, shared, or persistent-per-user;
• branch_workspace: replica-local, shared, or persistent-per-branch; and
• base_repository: replica-local, shared, or unavailable.

persistent-per-user means the same isolated home is selected from trusted tenant/user identity for auth helpers, Tasks, replicas, and restarts. persistent-per-branch means the same durable working copy is mounted at the absolute branch.path recorded in PostgreSQL. Executor homes and branches may be separate volumes; they do not need to be one global shared filesystem. An auth-resolved HA deployment must use persistent-per-user; startup rejects a shared or replica-local credential home rather than allowing users or tenants to share CLI credentials.

When base_repository: unavailable, native worktrees are invalid because their .git file points into the registered base repository’s .git/worktrees metadata. Agor therefore requires clone-only branch policy. A clone-mode branch carries its own .git directory and can be mounted without the base checkout. Requiring full rather than shallow clones is a separate policy.

PostgreSQL already owns repo/branch metadata and most coordination, but local worktrees, agent runtime homes, and tools that inspect a working directory are still a filesystem tail. The flag prevents a two-daemon deployment from claiming safe local execution when only one replica can see those paths. config.yaml is separately mounted read-only in the example because current bootstrap treats an existing config as immutable.

How sticky Socket.IO ingress works

A persisted WebSocket is an open TCP connection terminated by one daemon through the proxy. The proxy does not copy live connection state to another daemon. Redis lets either daemon deliver packets to clients attached to the other daemon.

Before WebSocket upgrade, Engine.IO may use multiple HTTP polling requests with one session ID. All /socket.io/ requests for that logical connection must therefore reach the same daemon. The example nginx configuration uses consistent client-IP hashing for /socket.io/, forwards upgrade headers, and sets a 120-second proxy timeout (above Agor’s 85-second heartbeat window). Ordinary HTTP/REST uses a separate round-robin upstream and is not sticky.

If a daemon dies, its existing sockets close; their Engine.IO session IDs do not migrate. Socket.IO/Feathers clients reconnect and reauthenticate to a healthy daemon, then refetch PostgreSQL-backed state. The Redis adapter does not provide connection-state recovery or missed-event replay.

For Agor Cloud, configure the platform load balancer or ingress so that:

1. /socket.io/ supports polling and WebSocket upgrades;
2. affinity is cookie-, connection-, or source-based for the whole Engine.IO session (cookie affinity is preferable when many clients share one NAT address);
3. idle/read timeouts exceed 85 seconds;
4. only ready pods receive new sessions; and
5. ordinary REST routes remain free to balance across ready pods; and
6. authentication, refresh, and external-launch abuse limits are enforced at a shared edge/WAF layer rather than multiplied per daemon.

The Compose IP hash is deliberately minimal and testable, not a recommendation for large NAT-heavy fleets.

Compose readiness limitation: Docker health checks gate the example’s initial nginx startup, but open-source nginx does not continuously consume later Docker health-state changes for its static upstream list. It can passively fail over after a connection error, but /readyz: 503 alone does not remove a still-listening daemon from this smoke ingress. Production ingress/orchestration must actively route only to ready replicas as required above.

Runnable Compose smoke variant

docker-compose.ha.yml uses explicitly named daemon-a and daemon-b, PostgreSQL, fanout-only Redis (RDB and AOF disabled), a one-shot migration service, and nginx. It does not use deploy.replicas. Because the example has exactly one nginx ingress, nginx also provides a coarse shared per-IP limiter for authentication/refresh/launch paths and logs the selected upstream. That limiter proves the fleet choke-point topology; it is not a credential-aware production quota and would not be shared across multiple nginx replicas.

export AGOR_REDIS_KEY_PREFIX=my-company-production
export AGOR_JWT_SECRET="$(openssl rand -hex 32)"
export AGOR_MASTER_SECRET="$(openssl rand -hex 32)"
export AGOR_ADMIN_PASSWORD="$(openssl rand -base64 24)"
export HA_PORT=3030
export AGOR_HA_PUBLIC_ORIGIN=http://localhost:3030
 
docker compose -f docker-compose.ha.yml -p agor-ha up -d --build
docker compose -f docker-compose.ha.yml -p agor-ha ps
curl -fsS http://localhost:3030/livez
curl -fsS http://localhost:3030/readyz
docker compose -f docker-compose.ha.yml -p agor-ha logs --tail=150 daemon-a daemon-b redis ingress

AGOR_HA_PUBLIC_ORIGIN must be the exact scheme, host, and port that browsers use for the ingress. The Compose file applies it as both the daemon public base URL and the credentialed CORS origin on every replica. The checked-in .agor.yml ha variant renders this value from Agor’s host address and branch port automatically.

The Agor-managed ha variant is intentionally a disposable development environment and uses the same admin@agor.live / admin login as the other development variants. The Compose file enables that weak credential only through an explicit development-only gate and runs with NODE_ENV=development; the daemon refuses the gate under NODE_ENV=production. Do not expose this smoke stack to an untrusted network or copy that gate into a production deployment. Direct Compose use should omit the gate, run in production mode, and provide a strong generated AGOR_ADMIN_PASSWORD as shown above.

Compose volumes survive up and ordinary Agor environment stop/start operations. In particular, AGOR_ADMIN_PASSWORD is bootstrap-only: changing it after the users table exists does not reset the admin password. Use the existing password or explicitly nuke the disposable smoke environment before expecting the checked-in development credential to bootstrap again.

Keep generated secrets in a secret manager and rotate them as an all-replica operation. The example’s unencrypted Redis connection stays on its private Compose network. Across a shared or untrusted network, use rediss://, certificate verification, Redis ACL authentication, private networking/firewalls, and credentials limited to the required Pub/Sub/control commands and deployment channels.

HA startup requires an explicit AGOR_ADMIN_PASSWORD unless external-launch authentication owns bootstrap. Daemons never race through the standalone generated credential-file path. The password is consumed only when the first admin row is created; it does not reset an existing account.

Configuration contract

deployment:
  mode: ha
  redis:
    # Prefer REDIS_URL when credentials are present.
    url: rediss://redis.internal:6380/0
    key_prefix: my-company-production
    connect_timeout_ms: 5000
    startup_timeout_ms: 15000
    request_timeout_ms: 5000
    reconnect_base_delay_ms: 100
    reconnect_max_delay_ms: 2000
  ha:
    support_profile: constrained-active-active
    execution_topology: external # or shared-local
    ingress_affinity: true
    environment_health_monitor:
      scan_interval_ms: 5000
      max_idle_interval_ms: 30000
      startup_offset_max_ms: 3000
      scan_batch_size: 32
      max_in_flight: 8
      http_timeout_ms: 1000
      claim_lease_ms: 15000
      shutdown_drain_timeout_ms: 5000
execution:
  executor_command_template: 'cell launch --tenant {tenant_id} -- {command}'
  executor_storage:
    user_home: persistent-per-user
    branch_workspace: persistent-per-branch
    base_repository: unavailable
  branch_storage:
    default_mode: clone
    allowed_modes: [clone]
    allow_shallow_clones: false
  allow_web_terminal: false
  managed_envs_execution_mode: webhook-only
database:
  dialect: postgresql
daemon:
  # Required for external executors to call/reconnect through the daemon fleet.
  public_url: https://agor-daemon.internal.example

For shared-local, replace the external template with shared_filesystem: true and declare the actual storage guarantees; its executor callback defaults to the local daemon. The Compose smoke stack declares all three surfaces shared: agor-ha-user-home mounts /home/agor, while the historical agor-ha-home is nested at /home/agor/.agor so PostgreSQL workspace paths remain valid across upgrades. Short-lived auth-file helpers and Tasks on either daemon therefore see the same CLI and Agor workspace state. This is one shared Unix execution identity, not per-user isolation. Codex import/logout may use it with the same simple-mode credential-sharing caveat as standalone; the process-local device-code poller remains gated. External topology requires an HTTP(S) daemon.public_url that the execution substrate can reach through the daemon fleet. Environment overrides include AGOR_DEPLOYMENT_MODE, AGOR_HA_SUPPORT_PROFILE, AGOR_HA_EXECUTION_TOPOLOGY, AGOR_HA_SHARED_FILESYSTEM, AGOR_HA_INGRESS_AFFINITY, REDIS_URL, AGOR_REDIS_KEY_PREFIX, the AGOR_REDIS_*_MS timeout/backoff variables, and AGOR_HA_ENV_HEALTH_SCAN_INTERVAL_MS, AGOR_HA_ENV_HEALTH_MAX_IDLE_INTERVAL_MS, AGOR_HA_ENV_HEALTH_STARTUP_OFFSET_MAX_MS, AGOR_HA_ENV_HEALTH_SCAN_BATCH_SIZE, AGOR_HA_ENV_HEALTH_MAX_IN_FLIGHT, AGOR_HA_ENV_HEALTH_HTTP_TIMEOUT_MS, AGOR_HA_ENV_HEALTH_CLAIM_LEASE_MS, and AGOR_HA_ENV_HEALTH_SHUTDOWN_DRAIN_TIMEOUT_MS. The health claim lease must be at least five seconds longer than the HTTP timeout. Storage guarantees remain YAML-only so an accidental environment toggle cannot claim that mounts exist.

The Redis prefix separates deployments operationally. It is never tenant authorization.

Health and failure semantics

Realtime events are at-most-once notifications across an outage window. Exactly-once durable mutation, work claims, task identity, and gateway occurrence admission are PostgreSQL concerns.

Security and event inventory

• The maintained @socket.io/redis-adapter carries complete Socket.IO packets across trusted Redis Pub/Sub. Redis traffic is neither signed nor encrypted by the adapter.
• Feathers channel membership is process-local and is not replicated by the adapter. Agor sends a bounded JSON relay envelope (tenantId, service path/event metadata, dispatched payload) with serverSideEmit. Each receiving daemon reruns tenant and branch/user publication logic against its own connections before entering the Feathers dispatcher.
• Authentication, refresh, raw session-token/API-key resolution, external-launch credentials, MCP OAuth/token endpoints, Codex/OpenCode auth, terminals, and secret-derived Artifact query results are denied from the Feathers Redis relay. After-hooks’ redacted dispatch value is relayed instead of an unredacted service result.
• Native cross-replica packets in this profile are cursor-moved, cursor-left, presence-updated, and tenant-scoped repo:cloneError. server-info is socket-local. OAuth packets are behind unsupported HA flows; terminal PTY traffic is disabled and must never enter Redis.
• Ordinary authorized Feathers data, including transcript/message streaming, may traverse the private Redis plane. Do not deliberately place bearer/API/executor/MCP/terminal tokens, OAuth/PKCE/device credentials, GitHub setup state, external-launch credentials, Artifact grants/environment, or secret-derived results in realtime payloads.
• Board-presence joins call the authenticated Feathers boards.get path before joining a tenant-qualified native room. Authentication replacement/logout revokes prior tenant/user/board room membership.
• Per-socket and daemon-local HTTP rate limits are process-local best effort, not a fleet security quota. The single-ingress Compose limiter is a coarse topology proof. Production must enforce security-sensitive quotas at a shared edge/WAF or durable authority layer.

No current product cross-replica notification requests client broadcast acknowledgements. Feathers method callbacks terminate on the daemon that handled the method. The adapter supports broadcast acknowledgements, but any future product event using them needs explicit multi-replica tests before support is claimed.

Managed-environment observation semantics

Each active starting or running branch is eligible for bounded periodic discovery. One replica holds a PostgreSQL lease for one observation. Releasing a completed observation writes a durable database-time cooldown, so another replica cannot multiply the configured polling cadence. If an owner dies without releasing, another replica can take over after database-time lease expiry even when a prior cooldown exists. Every result commit locks and rereads the branch and rejects an expired or replaced token, changed lifecycle generation, inactive status, or archived row. No database transaction remains open during the HTTP request.

• A healthy starting observation promotes the environment to running.
• A reachable non-success response is recorded as unhealthy. A network error or timeout during starting preserves the existing startup grace behavior; the same failure during running is recorded as unhealthy, without changing the lifecycle status.
• A missing health URL is recorded as unknown. A URL rejected by the health-check security policy is recorded as unhealthy. URLs and embedded credentials are not logged by the worker.
• Branch events only wake discovery early. Periodic scans recover missed events and newly eligible rows.
• A coordination failure makes /readyz return 503. The loop retries with bounded jitter/backoff; the first successful scan restores readiness. Endpoint health results never change daemon readiness.
• Graceful shutdown stops new discovery, aborts HTTP work, drains within the configured bound, and conditionally releases current tokens when time remains. Otherwise lease expiry is authoritative.

Current support matrix

Rollout and rollback checklist

1. Back up PostgreSQL and any shared-local workspace volume.
2. Apply all migrations before admitting new-version daemons. Migration 0077_environment_health_ha adds branch observation leases/generations and the narrow discovery policy. Use an all-at-once cohort replacement; old daemons do not understand the new capability contract.
3. Provision private Redis with health monitoring, ACL/TLS/network controls, and a unique prefix. Do not rely on Redis for persistence/replay.
4. Distribute identical stable config and secrets. Set stable AGOR_DAEMON_INSTANCE_ID values for diagnostics only.
5. Choose and verify shared-local or external; declare the executor storage contract from actual mounts, and do not set shared_filesystem merely because HA is enabled. Before using base_repository: unavailable, inventory historical branches for worktree rows.
6. Verify /socket.io/ polling-to-WebSocket affinity and REST distribution separately.
7. Configure /readyz for traffic removal and /livez for process restart decisions.
8. Confirm an authenticated Feathers health.find on every replica reports mode: ha, supportProfile: constrained-active-active, the expected fine-grained capability booleans, distinct instance/boot diagnostics, and ready Redis clients. In particular, do not interpret completionCallbackDurableAdmission: true as pre-admission recovery or gatewayListeners: true as outbound exactly-once; the corresponding negative capability flags remain visible. codexCredentialFiles reflects the declared home consistency while codexDeviceAuth remains false. environmentHealthMonitor: true confirms that the fenced observer is advertised. The public HTTP GET /health intentionally exposes only the configured instance label and dependency summary.
9. Exercise daemon loss, Redis loss/recovery, cross-replica authorized delivery, and negative authorization tests before production traffic.
10. Validate only the gateway providers and task permission modes you plan to enable; “daemon starts” is not full product HA certification.

To roll back, stop the HA cohort, remove all but one daemon from ingress, set deployment.mode: standalone, and restart that daemon against PostgreSQL. The added columns and policy are backward-compatible and may remain; rolling the migration back requires dropping the discovery policy/index and seven coordination columns only after every new daemon is stopped. REDIS_URL may stay present because standalone ignores it. Do not rotate shared secrets or switch database engines during an emergency rollback.

Opt-in integration harness

The repository harness is intentionally not part of the default test run. Against only the checked-in HA Compose project it proves two ready identities and activation of the distributed environment-health loop on both replicas (not occurrence-level fencing), activation of shared Task queue/reconciler loops, REST distribution, an actual 429 from the single-ingress sensitive-route limiter, stateless MCP initialization plus stateful rejection, polling upgrade affinity, authorized Feathers and native-room delivery with a bounded duplicate-observation window, anonymous denial, the split Codex contract (auth-file import admitted, device polling blocked), authenticated daemon reconnect/restart, and Redis readiness/liveness loss and recovery. PostgreSQL two-worker tests prove the environment ownership/fencing semantics. Cleanup runs in finally, restores stopped branch services, and removes any created API key or board after both success and assertion failure.

export COMPOSE_PROJECT_NAME=agor-ha-integration
export AGOR_REDIS_KEY_PREFIX=agor-ha-integration
export AGOR_JWT_SECRET='replace-with-32-plus-stable-characters'
export AGOR_MASTER_SECRET='replace-with-32-plus-stable-characters'
export AGOR_ADMIN_PASSWORD='replace-with-a-bootstrap-password'
export AGOR_HA_PUBLIC_ORIGIN=http://localhost:3030
export AGOR_HA_INTEGRATION=1
export AGOR_HA_INTEGRATION_START=1
export AGOR_HA_INTEGRATION_FAILURES=1
pnpm test:ha:docker

The source image intentionally contains no authenticated provider CLI/runtime, so the live harness does not fake an agent Task. PostgreSQL-gated daemon tests instead construct two queue workers/Task services on independent PostgreSQL clients, elect one dispatch intent for one admitted Task, exercise executor-token authentication/reconnect/revocation/bounded use through two Feathers apps backed by independent PostgreSQL clients, and exercise two reconcilers. Passing this harness is a transport/deployment smoke result, not a claim that the shared-local container preserves an in-flight executor, auth-resolved multi-tenancy, every gateway provider, every task permission mode, or the blocked process-affine features are HA-safe.