Launcher daemon: spawn headless workers from launch_requested batches

[justin808]

Migrated from pre-split shakacode/agent-coordination-state#9.

Context

Launching a batch currently means manually pasting prompts into Codex or Claude sessions. Once the HTTP backend and registered batch records exist, coordination state can also act as a launch queue.

Proposal

Add a launcher daemon that turns launch_requested batches into running headless workers on the matching machine.

Suggested scope

• Poll the HTTP API for batches tagged for the local machine.
• Claim before spawning workers.
• Create isolated git worktrees per lane.
• Spawn codex exec or Claude headless workers according to lane host choice.
• Record spawn events and lane thread handles.
• Enforce safety: max concurrency, duplicate-spawn prevention, dry-run mode, stop/pause controls, and no secret logging.

[justin808]

Batch disposition: no-pr-evidence — product decision, default park / P3

Decision: NO-GO for a production/autonomous launcher now. Keep this issue open.

Current main at dd109d00cb32bda5402af24929355f551459b733 does not yet provide the authoritative controls required to spawn unattended processes safely:

• agent-coord has no request-launch, cancel, stop, reservation, or launch-attempt commands.
• generation and instance_id are recorded as metadata, but current claim acquisition does not fence a second process using the same agent_id.
• The Worker exposes generic JSON state CAS, not first-class launch, cancellation, reservation, or process-control endpoints.
• The relational launch, cancellation, fencing, and reservation model in backend design is future architecture.
• sim/bin/llm-worker proves that a host can be started manually for a simulation, but it is not a safe launcher: it starts the autonomous host before the worker claims, inherits ambient environment, uses a temporary directory rather than a managed worktree, and has no duplicate-attempt fence, stop acknowledgement, or crash-adoption protocol.
• The #6 reservation disposition correctly parks runtime reservations until capacity, inbox, blocked-lane, and workspace state are authoritative.
• Issue #41 records that provisioned machine tokens still default to all-state scope.
• There is no launcher implementation PR, production incident artifact, measured manual-launch frequency, or verified customer demand. Those value signals are UNKNOWN.

Product decision required

Choose the boundary before code:

1. Recommended: a generic, single-operator local supervisor belongs in this MIT protocol plane; remote scheduling, policy/UI, and multi-user launch approval belong in the product plane.
2. Decide whether workers may ever receive push/PR credentials directly. The safer v1 uses a sanitized no-secret worker plus a narrowly authorized publisher/broker.
3. Choose fail-closed execution-lease behavior for launcher/backend loss and the first supported host adapter.

Minimum contract

Batch control:

registered -> launch_requested -> launching -> running
running -> stop_requested -> stopped
launching|running -> failed|cancelled

Launch attempt:

pending -> acquired -> starting -> running -> stopping
                            \-> succeeded|failed|cancelled|lost

A launch request must create a unique, generation-fenced attempt keyed by (workspace, batch_id, lane_name, request_generation). Acquisition must atomically validate request version, dependencies, host-limit eligibility, capacity reservation, and target claim; consume the reservation; and create the attempt before spawn. Terminal attempts never revive, and retries require an explicit generation increment.

Stable attempt data includes request/prompt/base-SHA hashes, launcher instance and generation, process identity, worktree path, capability profile, reservation, target claim, and timestamps. The coordinator owns request/cancel transitions; the fenced launcher instance owns attempt transitions.

Safety gates before a real-host adapter

• Dry-run: use the same live evaluator while performing no backend writes, claims, reservations, filesystem mutations, or child starts. Report hashes, eligibility, redacted command/environment, and refusal reasons.
• Duplicate prevention: two launchers racing produce exactly one attempt and one child, including same-agent-ID, repeated-request, retry, and crash-restart cases.
• Worktrees: trusted-repo allowlist, exact base SHA, launcher-owned canonical root, hooks disabled during setup, symlink/dirty/reused-directory/branch-collision rejection, root-bounded idempotent cleanup, and retention for unpushed or uncertain work.
• Secrets and untrusted input: allowlist the child environment; do not inherit coordination, GitHub, cloud, package, SSH-agent, or desktop-session credentials. Launch only a sanitized coordinator-authored goal with exact targets, trust-preflight evidence, trusted base SHA, and capability policy—not raw GitHub bodies/comments. Use short-lived per-attempt capabilities or a bounded publisher/broker.
• Stop/cancel: persist monotonic stop_requested; suppress before spawn; send process-group TERM, wait a bounded grace period, then KILL; record acknowledgements/signals; block automatic restart; release remaining reservation only after the child is terminal.
• Crash recovery: record PID plus process birth identity/session. Adopt only an exact match; otherwise terminate/quarantine and mark lost. Never spawn while a nonterminal attempt exists. Backend loss must expire a short local execution lease and fail closed.
• Auditability: append redacted events for request, refusal, acquire, spawn, running, stop, signal, exit, recovery, release, and retry, with actor/machine and hashes but no secret values.

Ordered implementation boundaries

1. Contract-only ADR/schema for launch request, launch attempt, state transitions, authorization, fencing, and conformance fixtures. No daemon.
2. Atomic backend/CLI request, acquire, stop, acknowledgement, and recovery operations with deterministic contention tests. No subprocess execution.
3. Dry-run policy evaluator, trusted local config, prompt/base/capability hashes, redaction tests, and zero-side-effect proof.
4. Fake-child supervisor with managed worktree, duplicate, stop, crash/adoption, dirty-retention, and secret-canary tests.
5. One opt-in, single-lane simulation canary for one explicitly selected host, one operator/machine, concurrency one. Broaden only after replay evidence.

No cross-repository edits belong in these lanes.

Resume condition

Resume PR 1 only after:

1. PR #37 is terminal and workspace-key encoding is explicit.
2. The maintainer records the protocol-plane versus product-plane ownership decision and credential posture.
3. Authoritative capacity/inbox/blocked-lane state and the Capacity reservation to close the triage plan-then-launch race #6 reservation contract exist, or the first canary is explicitly constrained to concurrency one with no planner-driven capacity claim.
4. First-class instance/generation fencing and persisted stop/cancellation semantics are accepted.
5. Least-privilege launcher/worker credential issuance is available; all-state inherited credentials are not accepted.
6. A replayable demand artifact or explicit maintainer approval justifies moving beyond simulation.

Recommended labels: discussion, P3. No label changes were made.

Process Gap Disposition

• Mechanism target: schema
• Motivating miss: manual headless starts have no authoritative spawn ownership, stop state, or crash-recovery record.
• Replay evidence: two-launcher/one-child race; dry-run zero-diff; hostile-input/secret canary; cancellation before and during spawn; TERM-to-KILL escalation; crash/adoption; backend outage; worktree retention; reservation conversion.
• Park reason: product boundary, authoritative reservation/capacity, cancellation/fencing, and credential model are unresolved.
• Non-goal: scheduler/UI/product planning, account rotation, vendor-cap evasion, arbitrary public issue ingestion, or prose-only launch rules.

QA: not required for this no-code architecture disposition. No files, commits, pushes, PRs, labels, issue state, or cross-repository content were changed by this lane.

[justin808]

Agent evaluation (batch awr-b-0716-1535, read-only lane):

Recommendation: park / P3 — the 2026-07-10 NO-GO stands. Prerequisites have advanced at the contract level only; every runtime control an unattended launcher needs is still missing.

Evidence:

• Re-verified 2026-07-16 against origin/main at 150d79d. Ledger of what changed since the NO-GO baseline (dd109d0):
  • RESOLVED: issue provision-token: require explicit scopes instead of all-state default #41 (machine tokens defaulting to all-state scope) closed completed 2026-07-11 via PR Require explicit scopes when provisioning tokens #46 — one cited blocker is gone.
  • ADVANCED (contract only): capacity reservation contract merged (PR Add capacity reservation state contract #65, ADR 0008) and host-limit contract merged (PR Add schema-first host-limit state contract #61, ADR 0007). Neither has runtime: no reservation or host-limit routes in worker/src/index.ts, no reserve/consume/release or host-limit commands in bin/agent-coord.
  • ADVANCED (ingredient): batch launch prompts are now persisted at registration and retrievable (PRs Expose saved batch launch prompts #50, Preserve batch launch prompts at registration #57, Add missing terminal and launch-prompt changelog notes #64) — a launcher would need exactly this to spawn workers.
  • UNCHANGED: launch_requested exists only in docs/backend-design.md (future architecture), not in the Worker state model; no cancel/stop/launch-attempt endpoints; claim acquisition still does not fence a second process reusing an agent_id (no fencing PR has merged since the baseline); machine/session identity (Track machine and Codex session identity via environment variables #62 / PR Track machine and session identity via environment variables #68) is still open.
• ADR 0008 explicitly lists launchers as a non-goal of the reservation contract, so the Add capacity reservation state contract #65 merge does not imply launcher authorization.
• UNKNOWN: still no production incident artifact, measured manual-launch cost, or demand signal that would justify pulling this forward.

Impact:

• Ergonomics/automation value only today. Manual prompt-pasting works and is the current safety boundary. The risk side is asymmetric: an unattended spawner without fencing, stop acknowledgement, and reservation-backed admission can double-spawn or orphan workers.

Complexity:

• Highest-risk item of the migrated trio. Hard dependency chain before any implementation: (1) Capacity reservation to close the triage plan-then-launch race #6 runtime (reservation admission), (2) claim fencing + identity (Track machine and Codex session identity via environment variables #62 / PR Track machine and session identity via environment variables #68), (3) batch launch_requested/cancel/stop state in the Worker, (4) product decision on the launch boundary (attended dry-run first vs. autonomous daemon), with Track and display host usage-limit exhaustion per machine and host #8 runtime as the eligibility gate for limit-blocked hosts.

Process Gap Disposition:

• Mechanism target: park
• Motivating miss: N/A (no incident; preventive design issue)
• Replay evidence or park reason: prerequisites are contract-only; no runtime admission, fencing, or stop controls exist to make unattended spawning safe
• Non-goal: shipping an autonomous daemon ahead of the reservation runtime and fencing, or inferring authorization from the Add capacity reservation state contract #65 contract merge

Next action:

• Keep open as the parked tracking anchor. Re-evaluate only after the Capacity reservation to close the triage plan-then-launch race #6 runtime slice and PR Track machine and session identity via environment variables #68 land, and after a maintainer picks the launch boundary (recommend starting with an attended, dry-run-only launcher if/when unparked). No implementation batch should pick this up before then.