Claude-Code-Monitor/docs/superpowers/plans/2026-08-03-shipyard-parity-lanes.md

# Shipyard parity for CCAM lanes — full plan

**Source of truth for "Shipyard":** `~/MyDrive/Projects/ResearchAndDevelopment/AgentWorkflow/`
(the "Parallel Feature Harness"). Every capability below traces to a real file
there, so each port can be checked against the original rather than a memory of it.

**Goal:** bring the whole harness into CCAM — isolated per-lane runtime and data,
per-feature state with archive, proof gallery, cross-lane locks, the
`ship-feature` pipeline with its QC agents, and the optional integrations.

**Non-goal (explicit):** OS-level sandboxing. Shipyard does not containerize
either. Its isolation is *resource namespacing on the host*: separate working
copy, ports, database, Redis logical DB, `.env`, upload dir. All lanes run as the
same user, share the network, and can read any file the user can. Anything
stronger is a separate design (§ Future).

---

## Status

| | Subsystem | Depends on | Status |
|---|---|---|---|
| **A1** | Slots, ports, profile hooks, detached lifecycle | — | ✅ **done** 2026-08-03 |
| **A2** | Data isolation: `.env`, database, Redis index | A1 | ✅ **done** 2026-08-03 |
| **A3** | Stack detection + profile scaffolding | A2 | ✅ **done** 2026-08-04 |
| **B** | Per-feature state + archive | — | ✅ **done** 2026-08-04 |
| **C** | Proof gallery | B | planned |
| **D** | Cross-lane named locks | — | ✅ **done** 2026-08-04 |
| **E** | `ship-feature` skill + QC agents | A2·B·C·D | planned |
| **F** | Integrations (tracker / dev-QC / CI) | E | planned |

A1 spec: `docs/superpowers/specs/2026-08-03-lane-runtime-isolation-design.md`.

**Detail below tapers on purpose.** A2 is written at implementation grade because
everything it needs is known. A3, B, C, D are at design grade — decisions named,
shapes fixed, exact code left to their own spec. E and F are at scope grade: E's
skill is a port of a 15-stage document whose every `bin/…` call must be rewritten
against command surfaces that B–D have not built yet, and writing that in detail
now would be writing against imagined APIs. Each subsystem still gets its own
spec → plan → implement cycle; this document is the map, not a substitute.

---

## Invariants every phase must preserve

Carried from `CLAUDE.md` and proven in A1. Re-check each at review time.

1. **CCAM does not orchestrate.** It offers primitives; the session sequences them.
   No chaining, no queue, no retry, no gate evaluation.
2. **The runtime never writes `stage`, `status` or `notes`** — only allocation
   facts. `status=running` means an agent is working, not that a server listens.
3. **Liveness is computed, never stored.** Anything CCAM does not control
   (a process, a port, a deploy) is re-derived on read.
4. **Never build a shell command as a string.** `execFile`/`spawn` with argv
   arrays. Hooks are the deliberate exception and are spawned with a fixed argv.
5. **Config is parsed, never sourced.** Hooks execute on purpose; declarations
   are only read.
6. **Adopted lanes are never destroyed and never written into.**
7. **Every destructive path re-runs its own guard**, never trusts its caller.

---

## A2 — Data isolation

**Goal:** a lane gets its own database, its own Redis logical DB, its own `.env`,
and its own data directories, so two lanes can run their stack at once without
sharing state. This is the half of Shipyard's isolation A1 did not cover.

**Traces to:** `bin/lane-env-seed.sh`, the `createdb`/`dropdb` blocks in
`bin/lane-bootstrap.sh` / `lane-up.sh` / `lane-reset.sh` / `lane-remove.sh`,
`_common.sh:lane_db` / `lane_db_url` / `lane_redis_url` / `lane_upload_dir`.

### Decisions to settle first

| Question | Proposed answer | Why |
|---|---|---|
| Who creates the database? | A `db-create.sh` / `db-drop.sh` **hook** (allowlist already exists in A1) | CCAM must not assume Postgres. `createdb` vs `mysqladmin create` vs `touch foo.db` genuinely differ; the template ships Shipyard's docker-compose Postgres commands so a Postgres user gets parity by copying. |
| Who rewrites `.env`? | **CCAM**, not a hook | Mechanical and identical across stacks: parse `KEY=VALUE`, replace a declared set, keep the rest. Pushing it to every repo would duplicate ~30 lines and re-lose the `JWT_SECRET` lesson below in each. |
| Where do database credentials live? | `~/.ccam/secrets.env`, mode `0600` | Machine-level, not repo-level (Shipyard's `config/secrets.env`). **Never** returned by any route and never logged; `GET /runtime` may report key *names* present, never values. |
| Does a lane without a database break? | No | `DB_PREFIX` empty (the default) means "no per-lane database": no name allocated, no hook called, no dead code. Same for `REDIS=0`. |

### Profile additions

```bash
# .ccam/profile/profile.env  (all optional; empty = feature off)
DB_PREFIX="myapp_l"          # lane in slot 3 -> myapp_l3 ; empty = no per-lane DB
DB_KIND="postgres"           # informational for A3's scaffolder; A2 gates on DB_PREFIX
DB_URL_SCHEME="postgresql"   # DATABASE_URL scheme
REDIS=1                      # 1 = allocate logical index = slot
ENV_FILES="backend/.env"     # files to seed, relative to the lane
ENV_SOURCE="backend/.env"    # source path in the source repo (defaults to ENV_FILES)
ENV_REWRITE="DATABASE_URL REDIS_URL UPLOAD_DIR"   # keys CCAM overwrites per lane
ENV_PRESERVE="JWT_SECRET"    # keys kept from the lane's EXISTING file on a --force refresh
UPLOAD_SUBDIR="backend/data/uploads"   # exported as UPLOAD_DIR
```

`DB_NAME`, `DATABASE_URL`, `REDIS_URL`, `UPLOAD_DIR`, `TEST_DATABASE_URL`
(`<DB_NAME>_test`, for `ci-gate`) join the hook environment contract.

### Steps

1. **`server/lib/secrets.js`** — read `~/.ccam/secrets.env` with the A1 parser
   (`lane-profile.js:parseEnvFile`, reuse it). Warn once and continue when the
   file is absent; refuse to load it when its mode is group/world-readable.
2. **`server/lib/lane-slots.js`** — extend `slotFacts`: `dbName`,
   `databaseUrl`, `redisUrl`, `uploadDir`, `testDatabaseUrl`. Same one place
   every slot-derived fact already comes from.
3. **`server/lib/lane-env.js`** (new) — `seedEnv(lane, profile, { force })`.
   Copies `<source_repo>/$ENV_SOURCE` → `<lane>/$ENV_FILES` when missing (or on
   `--force`), then rewrites `$ENV_REWRITE` keys **in the file**, so the file is
   correct on its own rather than merely masked by runtime exports.
   - **Port the two hard-won behaviours verbatim.** A `--force` refresh
     preserves `$ENV_PRESERVE` from the lane's existing file: swapping in the
     source's `JWT_SECRET` 401s a running lane until reboot. And a missing source
     `.env` falls back to `.env.example` with a loud warning, not silent success.
   - Refuses on an adopted lane (`assertManaged`): that file is the user's real
     working config.
4. **`server/lib/lane-services.js`** (new) — `ensureDatabase` / `dropDatabase`
   calling the `db-create` / `db-drop` hooks.
   - `dropDatabase` runs `assertManaged` **and** asserts
     `DB_NAME === slotFacts(lane.slot).dbName` before spawning anything. Only a
     name CCAM derived can ever be dropped; nothing from a request reaches it.
5. **Wire into the lifecycle** (`lane-runtime.js`, `routes/lanes.js`):
   - *provision* → allocate slot → `seedEnv` → `bootstrap` → `db-create` →
     `migrate` → `seed`
   - *up* → `seedEnv` (repair) → ensure DB exists → `migrate` **every boot**
     (Shipyard's note is right: lanes drift while idle and a stale schema
     cascade-fails the whole e2e suite) → `seed` only if the DB was just created
   - *reset* → down → git reset (existing) → `bootstrap` → clear `LANE_DIRS` →
     drop + create + `migrate` + `seed`, unless `--keep-db`
   - *remove* → down → `db-drop` for `<db>` and `<db>_test` → existing teardown
6. **Preflight + confirmation** — `lane-preflight.js` reports the database name a
   `reset`/`remove` will drop, and the confirm dialog echoes it back like the
   existing counted facts. Dropping a database is the most destructive thing this
   whole roadmap adds; it gets the same echo-or-refuse contract as the rest.
7. **CLI** — `ccam lanes reset --keep-db`; `ccam lanes runtime` gains the DB name
   and Redis index rows.
8. **Profile template** — `server/data/profile-templates/postgres-compose/` with
   `db-create.sh` / `db-drop.sh` holding Shipyard's exact
   `docker compose exec -T $DB_SERVICE createdb/dropdb` commands.

### Verify

- Fixture repo with a SQLite `db-create.sh` (`touch $DB_NAME.db`) — no Docker
  needed in CI: provision → file exists; reset → file recreated; remove → gone.
- `.env` seeding: keys in `ENV_REWRITE` replaced, every other line byte-identical;
  `--force` preserves `ENV_PRESERVE` from the existing file; missing source falls
  back to `.env.example` and warns.
- `dropDatabase` with a tampered `DB_NAME` throws **before** spawning.
- `seedEnv` and `dropDatabase` both throw `ENOTMANAGED` on an adopted lane.
- Secrets: a `0644` `~/.ccam/secrets.env` is refused; `GET /runtime` response
  contains no secret value (assert on the serialized JSON).
- Two lanes up at once, each writing its own database, neither seeing the other's
  rows — the actual point of the phase.

### Risks

- **Dropping the wrong database.** Mitigated by derived-name-only, the managed
  guard, and the echoed preflight. Review this path twice.
- **Secrets leaking into a log.** `runHook` streams hook output to the websocket;
  a hook that echoes `$DATABASE_URL` publishes a password to every browser tab.
  Add a redaction pass over hook output for values that came from `secrets.env`.

---

## A3 — Detection and scaffolding

**Goal:** `ccam lanes profile init <repo>` writes a working `.ccam/profile/`, so
adopting a repo is one command instead of nine hand-written hooks. This is the
gap that left Shipyard's own `profiles/` empty.

**Design grade.** The presets must be derived from the profiles actually written
during A1/A2 — that is why this phase is last in A, and its spec should start by
reading them.

- **Detect** from file signals, never from guessing: `package.json` (scripts
  `dev`/`build`/`start`, `vite`/`next`/`express` deps), `docker-compose.yml`
  (services named `postgres`/`mysql`/`redis`), `pyproject.toml` /
  `requirements.txt` (`django`/`fastapi`), `manage.py`, `alembic.ini`,
  `prisma/schema.prisma`, `go.mod`, `Gemfile`.
- **Scaffold, don't interpret.** Detection writes a concrete, readable
  `.ccam/profile/` the user owns and edits. The runtime never re-detects at boot;
  a wrong guess is fixed by editing a file, not by changing CCAM.
- Unknown values are written as explicit `TODO:` markers rather than plausible
  defaults — a wrong default that boots something is worse than a refusal.
- **`ccam lanes profile check`** — the `harness-doctor` equivalent: profile
  parses, declared hooks exist and are executable, no `TODO:` left, declared
  ports are free, `secrets.env` has what the declared database needs. Must exit
  non-zero on any of these.
- Templates in `server/data/profile-templates/<preset>/`.

**Verify:** a fixture repo per preset scaffolds, `profile check` passes, and
`ccam lanes up` boots it — the same end-to-end proof A1 used, once per preset.

---

## B — Per-feature state and archive

**Goal:** a lane's history survives switching features. Today `clearLane` erases;
Shipyard archives into `state/laneN/<slug>.json` with an `.active` pointer and
keeps every past feature browsable.

**Independent of A** — can be built in parallel.

- **Schema:** `lane_features` (`lane_id`, `slug`, `title`, `branch`, `stage`,
  `stage_since`, `status`, `gate_decision`, `ci_status`, `qc_dev`, `stages`,
  `links`, `notes`, `archived_at`), unique on `(lane_id, slug)`, plus
  `lanes.active_feature_id`. The `lanes` row stays the live view so nothing
  downstream breaks.
- `clearLane` becomes: snapshot the row into `lane_features` with `archived_at`,
  then reset. Nothing is lost by tidying up.
- **Slug canonicalization is the load-bearing detail.** The slug keys three
  things — the state row, the branch, and (in C) the proof directory. Port
  Shipyard's `state.sh activate` rule exactly: drop a `feat/` prefix, `/` and
  spaces to `-`, keep `[A-Za-z0-9._-]`, refuse empty, and **echo the canonical
  slug back** so callers store what the server stored. Reuse
  `worktree.js:slugify` only if its output is identical; otherwise a separate
  function with its own test. Do not let the two drift.
- Routes `GET/POST /api/lanes/:id/features…`; CLI `ccam feature list|activate|show`.
- UI: a feature picker that swaps the detail panel to an archived snapshot while
  the lane keeps running.

**Verify:** activate → clear → activate a second slug → the first is still
browsable with its final stage intact and the live row is clean; a slug with
slashes and spaces lands as one flat segment; `DELETE /api/lanes/:id` cascades.

---

## C — Proof gallery

**Goal:** the screenshots QC agents capture are visible in the dashboard, grouped
by feature and phase. Depends on B for the grouping key.

- Store at `<lane>/.playwright-mcp/proof/<slug>/{qc-local,qc-dev,ticket}/`.
- **Port `_common.sh:ensure_proof_link`.** Clone-root `proof/` becomes a symlink
  into the canonical directory, because screenshots land there whenever an MCP's
  `--output-dir` is not yet in effect. It exists to fix a real class of stranded
  evidence; skipping it recreates the bug.
- **Path containment is the entire security surface.** Resolve, `realpath`, and
  assert the result is inside the lane's proof root. No request path reaches `fs`
  unresolved. This is the one part of C worth reviewing carefully.
- Routes: manifest, file, delete. UI: thumbnails grouped feature → phase,
  lightbox, `+N` overflow, the 🎫 ticket-report link.

**Verify:** traversal attempts (`../`, absolute, symlink out of the root) all
rejected; a proof written to the clone root still appears in the manifest.

---

## D — Cross-lane named locks

**Goal:** serialize the steps that thrash a shared machine — builds, e2e runs —
across all lanes. **Independent; the smallest phase; ship it whenever.**

`server/lib/lane-lock.js` today is an in-process promise chain: it serializes
work *within one lane, within one process*. This is the other axis and must be a
**separate module** (`named-lock.js`); conflating them would be a subtle bug.

- `mkdir` for atomicity, owner file `lane<slot> <epoch>` — port `lane-lock.sh`.
  No `flock` dependency, so Shipyard's macOS `brew install flock` prerequisite
  disappears.
- **Time-based staleness with a floor.** A holder older than `LOCK_MAX_HOLD`
  (default 2700s) is broken on the next acquire, but the floor of 300s stays:
  it is what stops a caller from force-breaking a live lock via an env var.
- `acquire` heartbeats the waiting lane (~60s) so waiting never reads as stalled.
- **Port the etiquette text into `docs/LANES.md`.** Waiting is normal; never kill
  a holder, never delete the lock directory, never shrink `LOCK_MAX_HOLD`. Agents
  read docs, and this is the rule they break.
- `GET /api/locks`, `ccam lock status|acquire|release`, and a lock indicator on
  the lane card so a human can see *why* a lane is sitting still.

**Verify:** second acquire blocks then succeeds after release; a holder backdated
past `LOCK_MAX_HOLD` is broken; one backdated *within the floor* is not, even
with `LOCK_MAX_HOLD=1`; release from a non-holder is refused and leaves the lock.

---

## E — `ship-feature` and the QC agents

**Goal:** port Shipyard's driving skill and its five agents. Needs A2·B·C·D.

**This is where "CCAM does not orchestrate" is preserved by construction:** the
skill runs in the session and calls `ccam` commands; the dashboard still only
records.

- **Pipeline template** `server/data/pipelines/ship-feature.json` — the stages
  with aliases and `detect` rules in the existing node format. `default.json`
  untouched; a lane opts in via `pipeline`.
- **Skill** `.claude/skills/ship-feature-lane/SKILL.md` — every `"$HARNESS/bin/…"`
  call rewritten to its `ccam` equivalent (`state.sh set` → `ccam stage`;
  `activate` → `ccam feature activate`; `lane-ci-gate.sh` → `ccam lanes hook
  ci-gate`; `lane-up.sh --qc` → `ccam lanes up --qc`; the `.harness-lane` marker
  disappears because CCAM already resolves a lane from `cwd`).
- **Keep, do not trim, the hard-won rules.** Each encodes a real failure: the
  turn-liveness rule (every turn between stage 1 and done leaves a re-invoker
  pending), backgrounded-`sleep` polling, the e2e active-poll (a hung suite never
  fires its completion re-invoke), no-retry-cap/phase-clock, one-driver-per-MCP,
  never-push-base, and the test-only / localized re-entry fast paths.
- **Drop what CCAM makes unnecessary:** the `@@HARNESS_ROOT@@` install-time
  placeholder and the zsh `source _common.sh` warning. Keep an
  integration-mismatch check even though its original cause is gone — silently
  skipping ticket/dev-QC while reporting success is the failure that matters.
- **Agents** → `.claude/agents/`: `qc-local`, `dev-qc`, `senior-gate-reviewer`,
  `ticketer`, `pr-reviewer`. Shipyard generates these per lane with embedded
  credentials; port that as `ccam lanes agents install`, writing to
  `<lane>/.claude/agents/`, git-excluded, creds at mode `0600`.
- **`ccam lanes sync-base`** — port `lane-sync-dev.sh`. The valuable part is the
  **migration-number collision pre-check (exit 5)**: another lane's migration
  landed on the base with the number yours uses. Generic form driven by
  `MIGRATIONS_DIR` + a filename-number pattern. Also port the keep-ours merge
  driver for `GENERATED_MERGE_PATHS` plus the post-merge `regen` hook — it kills
  the single most common cross-lane conflict.

**Verify:** the skill is prose, so verification is a real dry run on a scratch
repo walking stages 0→8, confirming every `ccam` command it names exists and
behaves as documented. Each of those commands gets a CLI test.
`sync-base --check` gets a unit test with a fixture collision.

---

## F — Integrations

**Goal:** tracker, dev-site QC, CI deploy-wait. All off by default, read from
`<repo>/.ccam/profile/integrations.env`, exposed as `ccam lanes integration <name>`
(exit 0/1) and in `GET /runtime`.

- **Tracker** — file one ticket per feature, idempotent (update, never
  duplicate), writing `proof/<slug>/ticket/REPORT.html`.
- **Dev-site QC** — post-merge browser QC against the deployed site with a
  per-lane account; owns the `qc_dev` field; checkpoints to
  `RESULTS.partial.md` so a died agent resumes instead of restarting.
- **CI deploy-wait** — port `ci-job.sh` (`status`/`failures`/`rerun`/`cancel`) as
  `ccam ci`, which is what makes the skill's flake-triage rule executable.
- **`ccam lanes mcp sync`** — port `lane-mcp-sync.sh`: writes the lane's
  `.mcp.json` with Playwright servers pinned to the lane's own `--output-dir`.
  This is what stops two lanes' browser QC from sharing a profile directory.

**Verify:** all toggles off → the runtime reports them off and the skill's
stage-skip path runs; a toggle on in the file but off via the API → the mismatch
check fires instead of silently degrading.

---

## Order

```
A1 ✅ ──▶ A2 ✅ ──▶ A3 ✅
                     │
B ✅ ──▶ C ─────────────┼──▶ E ──▶ F
D ✅ ────────────────────┘
```

B next (it blocks E, and its presets must be derived from the profiles
actually written during A1/A2). B, C are independent (D done) of the A chain and of
each other except C→B — take D first if the pain right now is lanes thrashing
the machine, B first if it is `clear` erasing history.

**Per phase, every time:** spec → plan → implement → `npm run test:server` +
`npm run test:client` → docs (`update-project-docs`) → review. No phase is done
without its own end-to-end manual run, the way A1 proved detachment by killing
the server and watching the stack survive.

---

## Future: container isolation

Real isolation beyond Shipyard — per-lane container with its own filesystem view,
network namespace and resource limits.

The hard part is not the container. It is that Claude Code, its hooks and its MCP
servers must run *inside* it, so the hook → API path crosses a container
boundary, and the `cwd` a hook reports becomes a container path that no longer
matches the lane's host `cwd`. That breaks `resolveLaneByCwd` — the binding
between a session and a lane, and the assumption the entire lane model rests on.
`~/.claude` would also have to be mounted or synthesized per lane.

A design document of its own, not a phase of this one.