docs+test: the young-run window in SERVER-MODE/LANES; run.test plants its link in the real drop dir

run.test.js hardcoded /tmp/agenthub-drops-<uid>, which drops left for
<data dir>/drops long ago — it passed only while an old build's folder
survived in /tmp (gone after a reboot). Full suites: server 2887/2887,
client 1250/1250.

ARCHITECTURE.md

@@ -2323,7 +2323,7 @@ The same shape is used by `GET /status`, `POST /check`, and the `update_status`
 - **Branch coverage.** A feature branch or detached HEAD would need branch-aware integration (rebase vs merge vs switch) and merge-conflict handling. That belongs in the user's shell, not in a background daemon — which is why those situations get a printed `git fetch` and no button.
 - **Ownership.** A plugin-cache install is not ours to rewrite by hand — its directory names are content hashes Claude Code manages. So that case does not fast-forward anything; it calls `claude plugin update` and then re-runs `plugin-bootstrap.js` from the directory that tool produced. The residual limit is only `/reload-plugins`, which is REPL-scoped: an interactive session keeps its old skills/commands until the user reloads, even though the dashboard server has already moved.
 
-What `--ff-only` buys is the absence of the first three problems inside the supported cases: the move is either a clean fast-forward or it aborts, `npm run setup`/`build` failures reject the request before any restart is scheduled, and the restart itself goes through `self-restart.js` + the detached `scripts/restart-helper.js` rather than an in-process guess at the supervisor.
+What `--ff-only` buys is the absence of the first three problems inside the supported cases: the move is either a clean fast-forward or it aborts, `npm run setup`/`build` failures reject the request before any restart is scheduled, and the restart itself goes through `self-restart.js`: a server our own service unit runs (its cgroup ends in `/agenthub.service`, or it is our launchd job) is restarted BY that manager (`systemctl restart --no-block`, via `sudo -n` for the server-mode system unit; launchd's KeepAlive after an exit), anything else by the detached `scripts/restart-helper.js`. A detached replacement under a `Restart=always` unit held the port while the unit restarted every 3 s, each start exiting "already running" (v2.7.2).
 
 ---
 

CLAUDE.md

@@ -1,6 +1,6 @@
 # Claude Code Working Guide
 
-_Last synced: v2.7.2 (2026-10-12) — an account's own agent instructions go before every workspace's (#67); Chat ⇄ terminal keeps the unsent prompt; a server without S3 never ends a member's run._ A change that makes any statement in this file, `AGENTS.md`, or a README wrong — a moved path, a new or changed extension point, a retired rule — updates that file in the same change-set and bumps this marker. **Bump = replace the version/date and a ≤ 1-line summary; never append a changelog here** — details belong in the commit message and `docs/`, because this whole file is loaded into every session.
+_Last synced: v2.7.3 (2026-10-12) — Update now restarts through our own service unit; a start refuses a missing folder (ENOCWD)._ A change that makes any statement in this file, `AGENTS.md`, or a README wrong — a moved path, a new or changed extension point, a retired rule — updates that file in the same change-set and bumps this marker. **Bump = replace the version/date and a ≤ 1-line summary; never append a changelog here** — details belong in the commit message and `docs/`, because this whole file is loaded into every session.
 
 ## Project mission
 - Maintain a reliable local-first dashboard for Claude Code session monitoring.
@@ -9,7 +9,7 @@ _Last synced: v2.7.2 (2026-10-12) — an account's own agent instructions go bef
 ## Repo map
 - `server/`: Express API, hook ingestion, SQLite access, websocket broadcast (includes release-based update detection — `lib/update-check.js` reads the repo's own releases API and `routes/updates.js` exposes status/check/apply, refusing to self-update a `~/.claude/plugins/cache/` install, plus `lib/workflow-ingest.js` which ingests on-disk Workflow-tool run journals — fleets that emit no hooks — and `lib/codex-ingest.js`, which mirrors a `codex` lane's running/idle from the mtime of Codex's own rollout journals (skipping a lane whose hooks are current), `lib/codex-usage.js`, which reads the same journals into `sessions` (`tool = 'codex'`) and `token_usage`, and `lib/codex-hooks.js`, which installs AgentHub's opt-in hooks into Codex's `hooks.json`, and their Antigravity counterparts `lib/antigravity-usage.js` (agy's conversation files → `tool = 'antigravity'` sessions and tokens) and `lib/antigravity-hooks.js` (the named hook `agenthub` in `~/.gemini/config/hooks.json`), and `lib/cursor-usage.js` (Cursor's agent transcripts → `tool = 'cursor'` sessions, no tokens)).
 - `client/`: React + Vite UI.
-- `scripts/`: hook installer/handler, import, seed, cleanup utilities. (Update detection lives server-side in `server/lib/update-check.js`, keyed on the forge's **newest release** rather than a commit count. `POST /api/updates/apply` has two modes and picks by install kind, never from the request: a fast-forwardable **checkout** merges the release tag `--ff-only`, rebuilds, and self-restarts via `server/lib/self-restart.js` + the detached `scripts/restart-helper.js`; a **plugin-cache** install is handed to `server/lib/plugin-update.js` + the detached `scripts/plugin-update-helper.js`, which runs `claude plugin marketplace update` → `claude plugin update -y` → `plugin-bootstrap.js --force` from the NEW cache dir. Both write the HTTP response before anything stops the process, and every plugin-chain outcome is recorded to `plugin-update.json` in the runtime dir — outside the cache, so it survives the swap and the next server reports it as `last_plugin_update`. Neither mode can run `/reload-plugins`: it is REPL-scoped to the session that invokes it, so an open Claude Code session still reloads by hand for new skills/commands. Elsewhere users run the printed manual command, surfaced in the UI and by `agenthub update-check`. `scripts/cc-update-all-helper.js` is the same detached pattern for the config panel's **Update all** button — marketplaces → every plugin → AgentHub LAST → one reload — journalling each step to `cc-update-all.json` in the runtime dir.)
+- `scripts/`: hook installer/handler, import, seed, cleanup utilities. (Update detection lives server-side in `server/lib/update-check.js`, keyed on the forge's **newest release** rather than a commit count. `POST /api/updates/apply` has two modes and picks by install kind, never from the request: a fast-forwardable **checkout** merges the release tag `--ff-only`, rebuilds, and self-restarts via `server/lib/self-restart.js` — through our own unit when this process runs in one (`systemctl restart --no-block`, launchd's KeepAlive; a detached replacement beside a `Restart=always` unit crash-loops it), else the detached `scripts/restart-helper.js`; a **plugin-cache** install is handed to `server/lib/plugin-update.js` + the detached `scripts/plugin-update-helper.js`, which runs `claude plugin marketplace update` → `claude plugin update -y` → `plugin-bootstrap.js --force` from the NEW cache dir. Both write the HTTP response before anything stops the process, and every plugin-chain outcome is recorded to `plugin-update.json` in the runtime dir — outside the cache, so it survives the swap and the next server reports it as `last_plugin_update`. Neither mode can run `/reload-plugins`: it is REPL-scoped to the session that invokes it, so an open Claude Code session still reloads by hand for new skills/commands. Elsewhere users run the printed manual command, surfaced in the UI and by `agenthub update-check`. `scripts/cc-update-all-helper.js` is the same detached pattern for the config panel's **Update all** button — marketplaces → every plugin → AgentHub LAST → one reload — journalling each step to `cc-update-all.json` in the runtime dir.)
 - `skills/`: this repo's skills. Because the repo root **is** the `agenthub` plugin root, this is also the plugin's skills directory — every skill here ships to anyone who installs the plugin, addressed as `/agenthub:<name>` (bare `/<name>` works as an alias). Put a skill here only if it should reach plugin users. The plugin ships no subagents and no pipeline skill, and depends on no other plugin (Superpowers included) — do not add an auto-install of one back into the bootstrap.
 - `mcp/`, `statusline/`, `monitoring/`: opt-in integrations that only READ the running dashboard over HTTP — the read-only MCP server (`agenthub mcp`, docs/MCP.md), the Claude Code statusline (`agenthub statusline install|uninstall`, docs/STATUSLINE.md) and Prometheus/Grafana files for `/api/metrics` (`docker-compose.monitoring.yml`, docs/MONITORING.md). Neither the MCP server nor the statusline is ever registered by the bootstrap; only the user's own command writes `statusLine`, never over a foreign one without `--force`, always with a backup. The MCP server and the statusline find the dashboard through `server/lib/dashboard-url.js`, the CLI's own discovery.
 - `.claude-plugin/`: the marketplace plus the root `agenthub` plugin manifest (`"source": "./"` — the whole repo is the plugin). Its hooks are inline in `plugin.json`; its commands live in `plugins/agenthub/commands/`, which is NOT a subdirectory plugin. `scripts/plugin-bootstrap.js` runs from `SessionStart` and owns the writable runtime under `~/.claude/agenthub/runtime/` — it never writes into the plugin cache, which Claude Code replaces on every update. See `docs/PLUGINS.md`.

client/src/pages/Sessions.tsx

@@ -88,6 +88,7 @@ import { SessionRowMenu, workspaceHref } from "../components/SessionRowMenu";
 import { formatDateTime, formatDuration, truncate, fmtCost, isoOrUndefined } from "../lib/format";
 import { effectiveSessionStatus, sessionAwaitingReason } from "../lib/types";
 import type { Session, DashboardEvent } from "../lib/types";
+import { usePermissions } from "../components/AuthGate";
 
 const PAGE_SIZE = 10;
 
@@ -183,14 +184,17 @@ export function Sessions({ embedded = false }: { embedded?: boolean } = {}) {
 
   // Load remote-source labels so remote-origin rows can show a friendly badge
   // instead of a raw `src_…` id. Refreshed when a source's status changes.
+  // Staff only: a developer's /api/remote-sources is a 403 (STAFF_ONLY_PREFIXES).
+  const { seesAll } = usePermissions();
   const loadSourceLabels = useCallback(() => {
+    if (!seesAll) return;
     api.remoteSources
       .list()
       .then((res: { sources: RemoteSource[] }) => {
         setSourceLabels(new Map(res.sources.map((s) => [s.id, s.label])));
       })
       .catch(() => undefined);
-  }, []);
+  }, [seesAll]);
 
   useEffect(() => {
     loadSourceLabels();

client/src/pages/Settings.tsx

@@ -196,7 +196,7 @@ export function Settings({
   section?: "data" | "about" | "notifications";
 } = {}) {
   const { t } = useTranslation("settings");
-  const { isAdmin } = usePermissions();
+  const { isAdmin, seesAll } = usePermissions();
   // Only Data and About show server info; General and Notifications never read it.
   const needsInfo = section === "data" || section === "about";
   const [loading, setLoading] = useState(needsInfo);
@@ -954,10 +954,12 @@ export function Settings({
             <ImportHistory />
           </section>
 
-          {/* ─── REMOTE DATA SOURCES ─── */}
-          <section id="remote-sources" className="scroll-mt-24">
-            <RemoteSources />
-          </section>
+          {/* ─── REMOTE DATA SOURCES (staff only: the server answers 403 to a developer) ─── */}
+          {seesAll && (
+            <section id="remote-sources" className="scroll-mt-24">
+              <RemoteSources />
+            </section>
+          )}
 
           {/* ─── PUSH INGEST + PRIVACY (each its own anchored card) ─── */}
           <PushIngestSettings />

client/src/pages/__tests__/__snapshots__/screens.snapshot.test.tsx.snap

@@ -7178,7 +7178,7 @@ exports[`screen snapshots > Settings — about 1`] = `
                 <span
                   class="text-status-warning/90"
                 >
-                  UI build v2.7.2 (rebuild client to match)
+                  UI build v2.7.3 (rebuild client to match)
                 </span>
               </p>
             </div>

docs/API.md

@@ -357,6 +357,8 @@ Error `409 ESESSIONLIVE` is returned when `resumeSessionId` (or `resume`) names
 
 Error `400 EBADPROMPT` is returned for a `prompt` (or `POST /api/run`'s `initialPrompt`) over 128 KB: no CLI can take one argument that long (Linux's `MAX_ARG_STRLEN`, issue #68) — attach the text as a file instead. A prompt over 4 KB never rides in tmux's argv (one tmux message holds 16 KB, `command too long`): outside Windows it is written to `<data dir>/run-prompts/<run id>.txt` (0600) and the pane's own `sh` reads it back as the CLI's last argument and deletes the file. An `after-start` tool's paste still goes through `tmux set-buffer` and stays under that 16 KB.
 
+Error `409 ENOCWD` is returned when the workspace's folder no longer exists (or is not a directory) — the pane would otherwise start in the account's home under this workspace's name. A folder this server cannot read (`EACCES`) is not refused.
+
 Error `409 ERUNLIVE` is returned if the workspace already has a live run; stop it first with the `stop` action. The same code and status come back when the workspace's tmux pane is already running an agent while a `prompt` or `resumeSessionId` was sent — that argv cannot be typed over a live agent, so the request is refused instead of answered 200 with nothing done. A start that asks for neither still adopts the existing pane silently.
 
 #### Preflight a destructive workspace action

docs/SERVER-MODE.md

@@ -801,7 +801,29 @@ agenthub users provision alice      # the Linux account only (a user made before
   v2.7.2 on it leaves any run and workspace under the users root (read from
   `/etc/agenthub/server.conf`) alone; a build older than v2.7.2 marks their
   runs interrupted or abandoned within seconds, and the proxy gateway then
-  refuses those panes (`401 not a live pane's token`).
+  refuses those panes (`401 not a live pane's token`). A dev server that DOES
+  run in server mode (it read `AGENTHUB_MODE=server` from
+  `~/.config/agenthub/settings.env`) but lacks the unit's
+  `CAP_DAC_READ_SEARCH` cannot stat a member's 0700 `.agenthub/tmux`; from
+  v2.7.3 on that `EACCES` counts as "maybe running" and the member's list is
+  asked through sudo, never read as "no panes". And a run younger than 20 s
+  is never ended by any server: only the server that started a pane writes it
+  into its own cached tmux list, so another one on the same database had not
+  listed it yet and ended a Codex pane seconds after its start.
+- **Update now** under the system unit restarts through
+  `sudo -n systemctl restart --no-block agenthub.service` (from v2.7.3). An
+  older build started a detached replacement instead, which kept the port
+  while `Restart=always` started the unit every few seconds, each start exiting
+  "already running" — `systemctl status agenthub.service` shows
+  `activating (auto-restart)` with `MainPID` 0. Recover with
+  `kill -TERM <pid of the server on the port>`; the unit then starts on the
+  new code.
+- A member's tmux list is re-asked at most every 5 s (one `sudo` each, which
+  writes three `auth.log` lines); a pane the dashboard starts or stops shows at
+  once, one that exits on its own within those 5 s.
+- Claude Code does not carry a folder's trust down from the member's home
+  (which provisioning trusts): a member answers "Trust this folder" once per
+  workspace, in the terminal.
 
 ### Undo and uninstall
 

openapi.yaml

@@ -3,7 +3,7 @@
 openapi: 3.0.3
 info:
   title: Agent Dashboard for Claude Code API
-  version: 2.7.2
+  version: 2.7.3
   description: HTTP API for real-time Claude Code session monitoring, agent lifecycle tracking, analytics, pricing, hooks ingestion, and workflow intelligence.
   contact:
     name: Nguyễn Ngọc Trí Vĩ

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "agenthub",
-  "version": "2.7.2",
+  "version": "2.7.3",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "agenthub",
-      "version": "2.7.2",
+      "version": "2.7.3",
       "hasInstallScript": true,
       "license": "UNLICENSED",
       "dependencies": {

package.json

@@ -1,6 +1,6 @@
 {
   "name": "agenthub",
-  "version": "2.7.2",
+  "version": "2.7.3",
   "bin": {
     "agenthub": "bin/agenthub.js"
   },

scripts/test-server.js

@@ -16,7 +16,9 @@
  * setting never writes the developer's real `~/.config/agenthub/settings.env`,
  * and `AGENTHUB_PRICING_SYNC=0` keeps every suite off the network. The
  * `GIT_DIR`-family variables a git hook exports are dropped too, so a test's
- * own `git init`/`commit` can never land in the real repository.
+ * own `git init`/`commit` can never land in the real repository. Every
+ * inherited `AGENTHUB_*` variable is dropped as well: a dashboard pane carries
+ * `AGENTHUB_MODE=server` and friends, which turned the suites login-gated.
  * @author Nguyễn Ngọc Trí Vĩ <vinnt@smartgift.vn>
  */
 
@@ -34,8 +36,14 @@ const files = fs
 const args = process.argv.slice(2);
 
 const tmuxDir = fs.mkdtempSync(path.join(os.tmpdir(), "agenthub-test-tmux-"));
+// A dashboard pane's own AGENTHUB_* (AGENTHUB_MODE=server, AGENTHUB_USERS_ROOT,
+// its pane token…) would turn the suites into a login-gated server: hundreds
+// of 401s and EFOLDERs that have nothing to do with the code under test.
+const inherited = Object.fromEntries(
+  Object.entries(process.env).filter(([k]) => !k.startsWith("AGENTHUB_"))
+);
 const env = {
-  ...process.env,
+  ...inherited,
   TMUX_TMPDIR: tmuxDir,
   // Never the developer's real ~/.config/agenthub/settings.env: it can pin
   // DASHBOARD_DATA_DIR at their live database.

server/__tests__/dashboard-runs.test.js

@@ -22,8 +22,17 @@ before(() => {
 });
 after(() => fs.rmSync(ROOT, { recursive: true, force: true }));
 
+// Long enough ago that no reconcile treats the run as just started.
 const start = (id, extra = {}) =>
-  runs.recordRun({ id, mode: "interactive", cwd: "/r", status: "running", laneId: 4, ...extra });
+  runs.recordRun({
+    id,
+    mode: "interactive",
+    cwd: "/r",
+    status: "running",
+    laneId: 4,
+    startedAt: Date.parse("2026-01-01T00:00:00Z"),
+    ...extra,
+  });
 
 describe("dashboard-runs", () => {
   it("keeps the previous run of a pane as history when the pane starts again", () => {
@@ -77,6 +86,14 @@ describe("dashboard-runs", () => {
     assert.ok(runs.getRun("agenthub-lane-11"), "another tool's row stays");
   });
 
+  it("never ends a run started a moment ago: another process's tmux list may predate it", () => {
+    start("agenthub-lane-30", { laneId: 30, startedAt: Date.now() - 2000 });
+    start("agenthub-lane-31", { laneId: 31, startedAt: Date.now() - 60_000 });
+    runs.reconcileOrphans(() => false);
+    assert.equal(runs.getRun("agenthub-lane-30").status, "running");
+    assert.equal(runs.getRun("agenthub-lane-31").status, "abandoned");
+  });
+
   it("flags as abandoned only the running rows whose pane is gone", () => {
     start("agenthub-lane-5", { laneId: 5 });
     start("agenthub-lane-6", { laneId: 6 });

server/__tests__/lanes-api.test.js

@@ -427,6 +427,16 @@ describe("lanes changed by another server on the same DB", () => {
 });
 
 describe("lane start input", () => {
+  it("refuses to start in a workspace folder that no longer exists (409 ENOCWD)", async () => {
+    const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "lane-gone-"));
+    const id = (await request("POST", "/api/lanes", { cwd })).body.lane.id;
+    fs.rmSync(cwd, { recursive: true, force: true });
+    // Without the check the pane started in the account's home instead.
+    const r = await request("POST", `/api/lanes/${id}/start`, {});
+    assert.equal(r.status, 409);
+    assert.equal(r.body.error.code, "ENOCWD");
+  });
+
   it("answers 400 for a model that is not a plain model id", async () => {
     const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "lane-bad-model-"));
     const id = (await request("POST", "/api/lanes", { cwd })).body.lane.id;

server/__tests__/run.test.js

@@ -335,17 +335,15 @@ describe("GET /api/run/drop/:name", () => {
   });
 
   it("serves no file that is not a dropped image or text file", async () => {
-    const link = path.join(
-      os.tmpdir(),
-      `agenthub-drops-${os.userInfo().uid}`,
-      "fedcba9876543210.png"
-    );
-    fs.rmSync(link, { force: true });
-    fs.symlinkSync("/etc/passwd", link);
     const pdf = await fetchJson("/api/run/drop-file?name=a.pdf", {
       method: "POST",
       raw: { type: "application/pdf", buf: Buffer.from("%PDF") },
     });
+    // Planted in the real drop dir (the one this drop just made), not a
+    // guessed path: `<data dir>/drops` replaced `/tmp/agenthub-drops-<uid>`.
+    const link = path.join(path.dirname(pdf.body.path), "fedcba9876543210.png");
+    fs.rmSync(link, { force: true });
+    fs.symlinkSync("/etc/passwd", link);
     for (const name of [
       path.basename(pdf.body.path), // a drop, but no image
       "0123456789abcdef.png", // the right shape, never dropped

server/__tests__/s3-unseen-member-runs.test.js

@@ -31,7 +31,15 @@ const { healAllRunIds } = require("../routes/lanes");
 after(() => fs.rmSync(ROOT, { recursive: true, force: true }));
 
 const start = (id, laneId, cwd) =>
-  runs.recordRun({ id, mode: "interactive", cwd, status: "running", laneId, sessionId: `s-${id}` });
+  runs.recordRun({
+    id,
+    mode: "interactive",
+    cwd,
+    status: "running",
+    laneId,
+    sessionId: `s-${id}`,
+    startedAt: Date.parse("2026-01-01T00:00:00Z"),
+  });
 
 describe("a server without S3 on an S3 machine's database", () => {
   it("never ends a member's run, and still ends its own", () => {

server/__tests__/self-restart.test.js

@@ -0,0 +1,88 @@
+/**
+ * @file self-restart.test.js
+ * @description After an update, a server our service unit runs is restarted
+ * by the unit, never by a detached replacement: v2.7.2 on the S3 system unit
+ * left the replacement holding the port while `Restart=always` started the
+ * unit every 3 s, each start exiting "already running".
+ * @author Nguyễn Ngọc Trí Vĩ <vinnt@smartgift.vn>
+ */
+
+const { describe, it } = require("node:test");
+const assert = require("node:assert/strict");
+const { scheduleRestart, owningUnit } = require("../lib/self-restart");
+
+function deps(unit, restartAnswer = "restarted") {
+  const calls = { restart: [], kill: 0, spawn: 0 };
+  return {
+    calls,
+    unit,
+    svc: () => ({
+      restartUserService: (name) => {
+        calls.restart.push(name);
+        return restartAnswer;
+      },
+    }),
+    kill: () => calls.kill++,
+    spawnHelper: () => {
+      calls.spawn++;
+      return { unref() {} };
+    },
+  };
+}
+
+describe("self-restart", () => {
+  it("tells which unit owns this process, from its cgroup or launchd label", () => {
+    const linux = (text) => owningUnit({ platform: "linux", cgroup: () => text });
+    assert.equal(linux("0::/system.slice/agenthub.service\n"), "systemd");
+    assert.equal(
+      linux("0::/user.slice/user-1001.slice/user@1001.service/app.slice/agenthub.service\n"),
+      "systemd"
+    );
+    assert.equal(linux("0::/user.slice/user-1001.slice/session-3.scope\n"), null);
+    assert.equal(linux("0::/system.slice/agenthub.service.d/x\n"), null);
+    assert.equal(
+      owningUnit({
+        platform: "linux",
+        cgroup: () => {
+          throw new Error("no /proc");
+        },
+      }),
+      null
+    );
+    assert.equal(
+      owningUnit({ platform: "darwin", env: { XPC_SERVICE_NAME: "io.smartgift.agenthub" } }),
+      "launchd"
+    );
+    assert.equal(owningUnit({ platform: "darwin", env: {} }), null);
+  });
+
+  it("asks systemd to restart its unit and starts no replacement", () => {
+    const d = deps("systemd");
+    assert.equal(scheduleRestart("/r", d), "service");
+    assert.deepEqual(d.calls.restart, ["agenthub"]);
+    assert.equal(d.calls.spawn, 0);
+    assert.equal(d.calls.kill, 0, "systemd's restart job stops this process");
+  });
+
+  it("exits for Restart=always when systemd cannot be asked — still no replacement", () => {
+    const d = deps("systemd", "failed");
+    assert.equal(scheduleRestart("/r", d), "exit");
+    assert.equal(d.calls.kill, 1);
+    assert.equal(d.calls.spawn, 0);
+  });
+
+  it("exits under launchd, whose KeepAlive starts the new code", () => {
+    const d = deps("launchd");
+    assert.equal(scheduleRestart("/r", d), "exit");
+    assert.equal(d.calls.kill, 1);
+    assert.equal(d.calls.spawn, 0);
+    assert.deepEqual(d.calls.restart, []);
+  });
+
+  it("hands over to the detached helper when no unit owns the server", () => {
+    const d = deps(null);
+    assert.equal(scheduleRestart("/r", d), "helper");
+    assert.equal(d.calls.spawn, 1);
+    assert.equal(d.calls.kill, 1);
+  });
+});

server/__tests__/server-s3-panes.test.js

@@ -248,4 +248,20 @@ describe("run-as: a path under the users root belongs to that user", () => {
       "/opt/agenthub/libexec/write-as"
     );
   });
+  it("counts a member's tmux it cannot stat as maybe there, a missing one as none", () => {
+    const home = path.join(TMP, "users", "zoe");
+    const acct = { name: "zoe", login: "ah-zoe", home, uid: 5002 };
+    assert.equal(runAs.mayHaveTmux(acct), false, "no .agenthub at all");
+    const dir = path.join(home, ".agenthub", "tmux");
+    fs.mkdirSync(dir, { recursive: true });
+    assert.equal(runAs.mayHaveTmux(acct), false, "no socket yet");
+    // The member's own 0700 dir, seen by a dashboard without CAP_DAC_READ_SEARCH.
+    fs.chmodSync(dir, 0o000);
+    try {
+      if (process.getuid() !== 0)
+        assert.equal(runAs.mayHaveTmux(acct), true, "EACCES proves nothing");
+    } finally {
+      fs.chmodSync(dir, 0o700);
+    }
+  });
 });

server/lib/dashboard-runs.js

@@ -258,8 +258,21 @@ function claimToolSessions(sessionId = null) {
 }
 
 const runningStmt = db.prepare(
-  "SELECT id, tool, session_id, cwd FROM dashboard_runs WHERE status IN ('running', 'spawning')"
+  "SELECT id, tool, session_id, cwd, started_at FROM dashboard_runs WHERE status IN ('running', 'spawning')"
 );
+
+/**
+ * How long a new run is never judged dead. A tmux list is a cached answer —
+ * a member's may be up to 15 s old (tmux.js MEMBER_LIST_MAX_AGE_MS) — and only
+ * the process that started a pane writes it into its own cache, so another
+ * dashboard on the same database (a dev server) ended a pane it had simply
+ * not listed yet: the proxy gateway then answered 401 to a live Codex.
+ */
+const YOUNG_RUN_MS = 20_000;
+function isYoungRun(startedAt, now = Date.now()) {
+  const t = Date.parse(startedAt || "");
+  return Number.isFinite(t) && now - t < YOUNG_RUN_MS;
+}
 const endStmt = db.prepare(`
   UPDATE dashboard_runs
   SET status = @status,
@@ -300,7 +313,7 @@ function reconcileOrphans(isAlive = () => false) {
     let changes = 0;
     const { unseenMemberFolder } = require("./linux-accounts");
     for (const row of runningStmt.all()) {
-      if (isAlive(row.id) || unseenMemberFolder(row.cwd)) continue;
+      if (isAlive(row.id) || unseenMemberFolder(row.cwd) || isYoungRun(row.started_at)) continue;
       const interrupted =
         row.tool !== "shell" &&
         !!row.session_id &&
@@ -368,6 +381,7 @@ module.exports = {
   getRun,
   forgetShellRun,
   reconcileOrphans,
+  isYoungRun,
   listInterrupted,
   dismissInterrupted,
   proxyForSession,

server/lib/run-as.js

@@ -133,13 +133,22 @@ function memberHomes() {
 function membersWithTmux() {
   return memberHomes()
     .map((home) => accountForPath(home))
-    .filter((acct) => {
-      try {
-        return fs.statSync(tmuxSocket(acct)).isSocket();
-      } catch {
-        return false;
-      }
-    });
+    .filter(mayHaveTmux);
+}
+
+/**
+ * Whether the member may have a tmux server. Only a missing socket says no:
+ * a member's `.agenthub/tmux` is theirs (0700), and a dashboard without
+ * CAP_DAC_READ_SEARCH (a dev server on the same database) gets EACCES there
+ * — that proves nothing, and reading it as "no server" ended every run of
+ * theirs within seconds. Their list is asked through sudo, which can tell.
+ */
+function mayHaveTmux(acct) {
+  try {
+    return fs.statSync(tmuxSocket(acct)).isSocket();
+  } catch (err) {
+    return err.code !== "ENOENT" && err.code !== "ENOTDIR";
+  }
 }
 
 // uid by login, from /etc/passwd (world-readable; node has no getpwnam).
@@ -357,8 +366,7 @@ function openSafe(file) {
 // at no cost): MEMBER_READ_MAX unless the caller passes its own `max`, and
 // never past the size `fstat` gave — a file growing under the read is cut.
 const MEMBER_READ_MAX = 64 * 1024 * 1024;
-const tooBig = (file) =>
-  Object.assign(new Error(`${file}: too big to read`), { code: "ETOOBIG" });
+const tooBig = (file) => Object.assign(new Error(`${file}: too big to read`), { code: "ETOOBIG" });
 
 /** `fs.readFileSync(file, enc)`, guarded (a member's file: ≤ `max` bytes). */
 function readFileSafe(file, enc, { max = MEMBER_READ_MAX } = {}) {
@@ -369,7 +377,7 @@ function readFileSafe(file, enc, { max = MEMBER_READ_MAX } = {}) {
     if (size > max) throw tooBig(file);
     const buf = Buffer.alloc(size);
     let off = 0;
-    for (let n; off < size && (n = fs.readSync(fd, buf, off, size - off, off)) > 0; ) off += n;
+    for (let n; off < size && (n = fs.readSync(fd, buf, off, size - off, off)) > 0;) off += n;
     return enc ? buf.toString(enc, 0, off) : buf.subarray(0, off);
   } finally {
     fs.closeSync(fd);
@@ -398,7 +406,7 @@ async function readFileSafeAsync(file, enc, { max = MEMBER_READ_MAX } = {}) {
     if (size > max) throw tooBig(file);
     const buf = Buffer.alloc(size);
     let off = 0;
-    for (let n; off < size && (n = (await fh.read(buf, off, size - off, off)).bytesRead) > 0; )
+    for (let n; off < size && (n = (await fh.read(buf, off, size - off, off)).bytesRead) > 0;)
       off += n;
     return enc ? buf.toString(enc, 0, off) : buf.subarray(0, off);
   } finally {
@@ -438,7 +446,10 @@ async function* readLines(file, { maxLine = 64 * 1024 * 1024 } = {}) {
       let start = 0;
       for (let i = view.indexOf(0x0a); i !== -1; i = view.indexOf(0x0a, start)) {
         if (skipping || partLen + (i - start) > maxLine) yield "";
-        else yield text(partLen ? Buffer.concat([...parts, view.subarray(start, i)]) : view.subarray(start, i));
+        else
+          yield text(
+            partLen ? Buffer.concat([...parts, view.subarray(start, i)]) : view.subarray(start, i)
+          );
         parts = [];
         partLen = 0;
         skipping = false;
@@ -502,6 +513,7 @@ module.exports = {
   assertOwnerFits,
   hasLinuxAccount,
   membersWithTmux,
+  mayHaveTmux,
   memberHomes,
   ownerForRead,
   tmuxSocket,

server/lib/self-restart.js

@@ -7,26 +7,85 @@
  * it, and hands the "wait for the old PID to actually die, then spawn the
  * replacement" job to a detached helper script — the dying process can't do
  * that itself without racing its own replacement for the port.
+ *
+ * A server our service unit runs (systemd `agenthub.service`, system or user;
+ * the launchd job) is restarted BY that manager instead: a detached
+ * replacement would hold the port while `Restart=always` starts the unit
+ * again every few seconds, each start exiting "already running" — forever.
  * @author Nguyễn Ngọc Trí Vĩ <vinnt@smartgift.vn>
  */
 
+const fs = require("fs");
 const path = require("path");
 const { spawn } = require("child_process");
 
+const UNIT = "agenthub";
+
+/**
+ * "systemd" when this process lives in our unit's cgroup, "launchd" when it
+ * is our LaunchAgent job, else null. Pure reads: no process is spawned.
+ */
+function owningUnit({
+  platform = process.platform,
+  cgroup = () => fs.readFileSync("/proc/self/cgroup", "utf8"),
+  env = process.env,
+} = {}) {
+  if (platform === "darwin") {
+    return env.XPC_SERVICE_NAME === `io.smartgift.${UNIT}` ? "launchd" : null;
+  }
+  try {
+    return cgroup()
+      .split("\n")
+      .some((l) => l.trim().endsWith(`/${UNIT}.service`))
+      ? "systemd"
+      : null;
+  } catch {
+    return null;
+  }
+}
+
 /**
  * @param {string} [root] repo root the helper should run from.
+ * @returns {"service"|"exit"|"helper"} how the restart was handed over.
  */
-function scheduleRestart(root = path.join(__dirname, "..", "..")) {
+function scheduleRestart(root = path.join(__dirname, "..", ".."), deps = {}) {
+  const {
+    unit = owningUnit(),
+    svc = () => require("../../scripts/service-manager"),
+    kill = () => process.kill(process.pid, "SIGTERM"),
+    spawnHelper = spawn,
+  } = deps;
+  // systemd: one non-blocking restart job (sudo -n for the system unit). If
+  // that cannot be asked, a plain exit still comes back — the unit is
+  // Restart=always — but never with a second server beside it.
+  if (unit === "systemd") {
+    let restarted = false;
+    try {
+      restarted = svc().restartUserService(UNIT) === "restarted";
+    } catch {
+      /* fall through to the exit */
+    }
+    if (restarted) return "service";
+    kill();
+    return "exit";
+  }
+  // launchd: KeepAlive restarts the job on exit; a `bootout` from inside the
+  // job would wait for this very process.
+  if (unit === "launchd") {
+    kill();
+    return "exit";
+  }
   const helperPath = path.join(root, "scripts", "restart-helper.js");
   const entry = process.argv[1] || path.join(root, "server", "index.js");
-  const helper = spawn(process.execPath, [helperPath, String(process.pid), entry], {
+  const helper = spawnHelper(process.execPath, [helperPath, String(process.pid), entry], {
     cwd: root,
     detached: true,
     stdio: "ignore",
     env: process.env,
   });
   helper.unref();
-  process.kill(process.pid, "SIGTERM");
+  kill();
+  return "helper";
 }
 
-module.exports = { scheduleRestart };
+module.exports = { scheduleRestart, owningUnit };

server/lib/tmux.js

@@ -226,16 +226,21 @@ function listOutput(acct = null) {
 }
 
 // A member's list is never waited for: a read answers from `listed` and,
-// once that is LIST_TTL_MS old, starts one async re-list. Until a first
+// once that is MEMBER_LIST_TTL_MS old, starts one async re-list. Until a first
 // answer — or once the last is MEMBER_LIST_MAX_AGE_MS old (nobody polled) —
 // or while their server fails, it throws: their panes are UNKNOWN. A session
 // this process creates or kills is written into the answer at once (`noteSession`).
 const MEMBER_LIST_MAX_AGE_MS = 15_000; // > the 10 s heal beat
+// A member's list is a `sudo` (fork + PAM session + 3 auth.log lines) — at a
+// 1 s TTL every open dashboard re-asked each member ~30 times a minute. A pane
+// this process starts or ends is corrected at once (`noteSession`); only one
+// that exits on its own reads live for up to these 5 s.
+const MEMBER_LIST_TTL_MS = 5_000;
 const refreshing = new Set(); // member logins with a list in flight
 function memberListOutput(acct) {
   const hit = listed.get(acct.login);
   const age = hit ? Date.now() - hit.at : Infinity;
-  if (age >= LIST_TTL_MS) refreshList(acct);
+  if (age >= MEMBER_LIST_TTL_MS) refreshList(acct);
   if (age >= MEMBER_LIST_MAX_AGE_MS) {
     throw Object.assign(new Error(`${acct.login}: tmux list pending`), { code: "ETMUXPENDING" });
   }

server/routes/lanes.js

@@ -128,6 +128,10 @@ function healRunId(lane, live = null) {
   if (require("../lib/linux-accounts").unseenMemberFolder(lane.cwd)) return lane;
   if (live) {
     if (live.has(lane.run_id)) return lane;
+    // Started a moment ago: a cached list may not show it yet (dashboard-runs.js).
+    const dashboardRuns = require("../lib/dashboard-runs");
+    const row = dashboardRuns.getRun(lane.run_id);
+    if (row && row.status === "running" && dashboardRuns.isYoungRun(row.started_at)) return lane;
   } else {
     const run = runs.getRun(lane.run_id);
     if (run && run.status === "running") return lane;
@@ -2060,6 +2064,22 @@ router.post("/:id/:action", sameOriginGuard, async (req, res) => {
   try {
     switch (action) {
       case "start": {
+        // A folder that is gone (or never was) would start the pane in the
+        // account's home instead, under this workspace's name. EACCES is not
+        // "gone": a member's folder may be unreadable to this process.
+        try {
+          if (!fs.statSync(lane.cwd).isDirectory())
+            throw Object.assign(new Error(), { code: "ENOTDIR" });
+        } catch (err) {
+          if (err.code === "ENOENT" || err.code === "ENOTDIR") {
+            return res.status(409).json({
+              error: {
+                code: "ENOCWD",
+                message: `the workspace folder does not exist: ${lane.cwd}`,
+              },
+            });
+          }
+        }
         // `"auto"`: the first free tab, picked under the lane lock below — a
         // client's own pick comes from a list that may already be stale.
         const auto = body.slot === "auto";