Merge branch 'perf/local-opt': faster local install and updates, lighter polling and ingest

docs/API.md

@@ -1006,6 +1006,9 @@ cannot collide on `index.lock`.
   the file resolves by removing it. `merge-commit` is `409 ECONFLICTS` while
   anything is unmerged, and it and `merge-abort` are `409 ENOMERGE` with no
   merge in progress. `merge-abort` needs `confirm: true` (`400 ECONFIRM`).
+- Any action on a host with no `git` binary (or none on the dashboard's PATH)
+  is `503 EGITMISSING`: a missing dependency, not a bad request; the message
+  carries the install command (like `ETMUXMISSING` for tmux).
 - `push` pushes `HEAD`, never a branch name from the request, so a workspace can
   only push the branch it is on.
 - A **path** in the body is never trusted as a path: `stage`, `unstage` and
@@ -3057,6 +3060,16 @@ POST   /api/settings/backup/run    202 — started (or already running); poll GE
 Settings → Data → **Daily backup** (`server/lib/db-backup.js`). Once a day, at `hour` (0–23, the server's local time, default 2), the database is copied to `dir` as `agenthub-YYYYMMDD-HHMMSS.db`: `VACUUM INTO` on a read-only connection in a worker thread, so the copy is a consistent snapshot (writes from another dashboard on the same database never restart it), compacted, and the event loop keeps serving while it runs (≈30 s for a 1.25 GB database). It is written as `<name>.partial` and renamed when complete. Beyond `keep` (1–365, default 7) the oldest are deleted — only regular files matching that exact name; nothing else in the folder, and no symlink, is ever touched. A day missed while the server was down runs at the next start; a failed run is retried the next day or by **Back up now**, never every minute. The three values persist to the settings file as `AGENTHUB_BACKUP_DIR` / `AGENTHUB_BACKUP_HOUR` / `AGENTHUB_BACKUP_KEEP`; one the environment set is `env_owned` (`409 EENVOWNED`). Writes are same-origin, and admin-only in server mode. `400 INVALID_PATH` for a relative, missing, non-directory or unwritable folder, `400 INVALID_INPUT` for an hour or count out of range, `400 ENOBACKUPDIR` for a run with no folder. The scheduler beat is `DASHBOARD_BACKUP_TICK_MS` (default 60000, `0` = off).
 
 Restoring is manual, on purpose: `agenthub stop`, delete `dashboard.db-wal` and `dashboard.db-shm`, copy a backup over `dashboard.db` (its path is on Settings → System), `agenthub start`.
+
+### Retention and Compact
+
+```
+GET    /api/settings/retention     { days, max_days, env_owned: { days }, last_run, space: { auto_vacuum, page_size, page_count, freelist_pages, free_bytes }, db_size, compacting, last_compact }
+PUT    /api/settings/retention     { days } → the same state; 0 turns it off (the default)
+POST   /api/settings/compact       { confirm: true } → 202 — started (or already running); poll GET while `compacting`
+```
+
+Settings → Data → **Session cleanup** (`server/lib/db-retention.js`). Automatic retention is OFF unless `days` (1–3650, `AGENTHUB_RETENTION_DAYS` in the settings file) is set; nothing is deleted by default. When set, the hourly maintenance (`DASHBOARD_DB_MAINTAIN_MS`) purges finished (completed/error/abandoned) sessions that started more than `days` ago — the same rule and the same deletion (`purgeSessionIds`: events, agents, token usage, the session) as `POST /api/settings/cleanup` with `purge_days` — 25 sessions per transaction with a yield between, at most 5000 per run. A new database is created with `auto_vacuum = INCREMENTAL`, an existing one up to 32 MB is converted by a VACUUM at startup; the same hourly run then returns free pages to the OS (`incremental_vacuum`, in 1024-page slices, at most 65536 pages per run) once more than 1024 are free. A larger, older database is converted only by **Compact**: a full VACUUM in a worker thread followed by a TRUNCATE checkpoint — it holds the write lock while it runs (tens of seconds on a GB-size file), so hooks wait meanwhile; it runs only on a click. In server mode both writes are an admin's (`/settings` is in `ADMIN_WRITE_PREFIXES`).
 ---
 
 ## WebSocket API

docs/DATABASE.md

@@ -915,7 +915,7 @@ worker's own writable connection. SQLite's automatic checkpoint runs inside whic
 per commit under load. A failed checkpoint switches the automatic one back on. A second
 process on the same file (another dashboard, the CLI) keeps its own setting.
 
-**Upkeep.** `maintainDatabase()` (exported by `server/db.js`) runs hourly from `server/index.js` (`DASHBOARD_DB_MAINTAIN_MS`, `0` = off): a `wal_checkpoint(PASSIVE)` — it never waits on a reader, such as the other dashboard — after which the next write shrinks the WAL to `journal_size_limit`, and `PRAGMA optimize`. Nothing runs `VACUUM`: it rewrites the whole file under an exclusive lock, so space freed by a cleanup is reused for new rows instead of returned to the disk.
+**Upkeep.** `maintainDatabase()` (exported by `server/db.js`) runs hourly from `server/index.js` (`DASHBOARD_DB_MAINTAIN_MS`, `0` = off): a `wal_checkpoint(PASSIVE)` — it never waits on a reader, such as the other dashboard — after which the next write shrinks the WAL to `journal_size_limit`, and `PRAGMA optimize`. Then `runScheduledMaintenance` (`server/lib/db-retention.js`) applies the OPT-IN retention (`AGENTHUB_RETENTION_DAYS`, off by default: finished sessions older than N days, purged 25 per transaction like the manual cleanup) and, when the database is in `auto_vacuum = INCREMENTAL` and more than 1024 pages are free, a bounded `incremental_vacuum` (1024-page slices, ≤ 65536 pages per run) that returns them to the disk. A new database is created INCREMENTAL; an existing one up to 32 MB is converted by one `VACUUM` at startup (`convertAutoVacuum`), a larger one only by Settings → Data → **Compact** (`POST /api/settings/compact`: a full `VACUUM` in a worker thread, write lock held while it runs). No automatic path runs a full `VACUUM` on a large file. See docs/API.md § Retention and Compact.
 
 **Growth.** `events.data` is the hook payload TRIMMED by `trim_event_data(?)` (`server/lib/event-payload.js`, a SQL function registered on the connection in `server/db.js`, used by every event `INSERT` and by the duplicate guard): Edit/MultiEdit/NotebookEdit/Write drop `tool_response.originalFile`/`oldString`/`newString` (Write also `content`), Read drops `tool_response.file.content`/`base64`, and any string inside `tool_input`/`tool_response` over 8 KB keeps its first 6 KB and last 2 KB; `structuredPatch` and every field outside those two are kept, and `_trimmed` lists what was dropped. The full text stays in the session transcript and AgentHub's snapshot of it under `<data dir>/transcripts/`, which the Conversation tab reads. Rows stored before this are trimmed once in the background (`sweepEventPayloads`, 200 rows per transaction, `user_version` 4 after marker 3); measured on a 1.2 GB install it took `events.data` from 913 MB to ~400 MB. Freed pages are reused, not returned to the disk (no `VACUUM`). No ROW is pruned automatically. The only deletions are the user's: Settings → clear data / cleanup, a workspace's purge, `scripts/clear-data.js`. Clear data runs in ONE transaction (foreign keys off around it, so workflows are deleted explicitly), cleanup deletes in 500-id chunks inside one transaction. `GET /api/settings/export` streams `events` in keyset pages of 500 instead of loading the table.
 

docs/DEPLOYMENT.md

@@ -256,9 +256,9 @@ DASHBOARD_DB_PATH=/var/lib/agenthub/dashboard.db
 DASHBOARD_WORKFLOW_POLL_MS=12000        # Workflow-tool run journals
 DASHBOARD_CODEX_POLL_MS=15000           # Codex rollout journals → codex lane running/idle + Codex sessions/tokens
 CODEX_HOME=                             # where Codex keeps sessions/ (default ~/.codex)
-DASHBOARD_ANTIGRAVITY_POLL_MS=15000     # agy conversation files → Antigravity sessions/tokens
+DASHBOARD_ANTIGRAVITY_POLL_MS=15000     # agy conversation files → Antigravity sessions/tokens (60 s while idle)
 AGENTHUB_GEMINI_DIR=                    # where agy keeps its data (default ~/.gemini)
-DASHBOARD_CURSOR_POLL_MS=15000          # Cursor agent transcripts → Cursor sessions (no tokens)
+DASHBOARD_CURSOR_POLL_MS=15000          # Cursor agent transcripts → Cursor sessions (no tokens; 60 s while idle)
 AGENTHUB_CURSOR_DIR=                    # where Cursor keeps its data (default ~/.cursor)
 AGENTHUB_AGY_USAGE=                     # 0 never runs `agy --print /usage` for the Usage panel's Antigravity quota (default on; running agy may update it)
 

docs/HOOKS.md

@@ -491,7 +491,8 @@ graph TB
 >    entry whose PID is no longer alive** and POSTs the hook payload to one
 >    port per **unique SQLite data directory** (lowest port wins when two
 >    instances share `~/.claude/agenthub`). The other instance re-broadcasts
->    the workspaces those hooks moved within `DASHBOARD_LANE_SYNC_MS` (2 s), so its
+>    the workspaces those hooks moved within `DASHBOARD_LANE_SYNC_MS` (2 s; a server
+>    alone on its data dir only looks for such a peer every 30 s), so its
 >    browsers see a workspace's status without a refresh.
 > 3. If neither yields a target, the handler falls back to `4820`.
 >
@@ -589,6 +590,7 @@ Claude Code waits only for the handler process: Node start-up, reading stdin, se
 
 - **Fire-and-forget POST** — no wait for the dashboard's reply.
 - **Incremental transcript reads** — `TranscriptCache` reads only the bytes appended since the last hook.
+- **Only new transcript entries are written** — a per-transcript mark (`transcriptMarks` in `server/routes/hooks.js`, set once the hook's transaction commits) skips the token, compaction, API-error and turn-duration writes when the transcript did not change, and walks only the entries after the last one handled when it grew.
 - **One POST per database, not per server** — servers sharing a database get each event once (`resolveHookIngestPorts`).
 - **Heavy reads off the event loop** — aggregates run in a worker (`server/lib/read-pool.js`), so a slow page never delays ingest.
 

docs/OVERVIEW.md

@@ -115,7 +115,7 @@ Internal build — all rights reserved.
 
 | Need | Version | Required for |
 |---|---|---|
-| **Node.js** | >= 20 (>= 22.5 for a plugin install) | Everything. Both suites are verified green on **24** and **26** |
+| **Node.js** | >= 22.5 (`node:sqlite`; Node 20 is end-of-life) | Everything. Both suites are verified green on **24** and **26** |
 | **npm** | 9+ | Ships with Node |
 | **git** | any | Cloning, and every workspace operation (worktrees, branch/status reads) |
 | **tmux** | 3.x | The browser run console, `agenthub workspace shell`, typing `/reload-plugins` into a live pane |

docs/OVERVIEW.vi.md

@@ -81,7 +81,7 @@ Internal build — all rights reserved.
 
 | Cần | Phiên bản | Dùng cho |
 |---|---|---|
-| **Node.js** | >= 20 (>= 22.5 nếu cài qua plugin) | Tất cả. Cả hai test suite verify xanh trên Node **24** và **26** |
+| **Node.js** | >= 22.5 (`node:sqlite`; Node 20 đã hết hạn hỗ trợ) | Tất cả. Cả hai test suite verify xanh trên Node **24** và **26** |
 | **npm** | 9+ | Đi kèm Node |
 | **git** | bất kỳ | Clone repo, và mọi thao tác workspace (worktree, đọc branch/status) |
 | **tmux** | 3.x | Run console trên trình duyệt, `agenthub workspace shell`, gõ `/reload-plugins` vào pane đang sống |

docs/PLUGINS.md

@@ -61,13 +61,13 @@ claude --plugin-dir plugins/agenthub-analytics
 ## Prerequisites
 
 - [Claude Code](https://docs.anthropic.com/en/docs/claude-code) installed and authenticated
-- Node **>= 22.5** when installing via the `agenthub` plugin. A plugin install has no
-  native `better-sqlite3`, so the server stores data through `node:sqlite`, which
-  landed in 22.5. The bootstrap refuses with one line on anything older instead
-  of letting the server crash. (A checkout install still works on Node >= 20.)
+- Node **>= 22.5**. A plugin install has no native `better-sqlite3`, so the
+  server stores data through `node:sqlite`, which landed in 22.5. The bootstrap
+  refuses on anything older instead of letting the server crash, and says so in
+  the session (below). A checkout needs the same (`engines` in `package.json`).
 - The Agent Monitor dashboard reachable at `http://localhost:4820` — the `agenthub`
   plugin starts it for you; a checkout starts it with `npm start` (see
-  [SETUP.md](../SETUP.md))
+  [INSTALL.md](../INSTALL.md))
 
 Skills and commands are invoked as `/agenthub-<plugin>:<name>`. Agents are dispatched automatically by Claude Code (or named explicitly).
 
@@ -124,12 +124,12 @@ waits on an install:
 
 1. **Fast path** — recorded state matches this plugin build and a server is live → exit. "Live" means a discovery entry whose pid answers AND whose `startedAt` is after this boot (60 s margin): an entry left over from before a reboot names a pid the OS may have handed to an unrelated process, and without systemd (macOS) nothing else clears it.
 2. **Superseded check** — if Claude Code records a DIFFERENT build of this plugin as installed, this one logs a line and does nothing else (see [A retired build never acts](#a-retired-build-never-acts)).
-3. **Node gate** — refuse below 22.5 with one line (see Prerequisites).
+3. **Node gate** — refuse below 22.5 with one line (see Prerequisites). Also checked by the foreground pass itself (it costs nothing), which then spawns no worker and tells the session at once.
 4. **Lock** — atomic `mkdir` lock holding the PID; reclaimed when the owner is dead or the lock is older than 10 minutes, so two sessions cannot race the install or spawn two servers.
-5. **Install** — `npm install --omit=dev --ignore-scripts` into the runtime dir, run as the npm bundled with the node running the bootstrap (`npmCommand`: its `npm-cli.js` beside `process.execPath`, no shell, that node's directory first on PATH; `npm` on PATH only when that node ships none) — the update chain runs with the dashboard server's environment, whose PATH need not reach `npm.cmd` on Windows. The UI build below and the doctor's npm row use the same npm (never into the plugin cache, which is garbage-collected and replaced on every update). `--ignore-scripts` keeps the root `postinstall` from pulling the whole Vite client toolchain.
+5. **Install** — only when the dependencies changed (`depsHash` covers what npm installs — the dependency lists and the lockfile's package tree — never the release's own `version`, so a plain version bump reinstalls nothing): `npm install --omit=dev --ignore-scripts` into the runtime dir, output in `runtime/npm-install.log`, run as the npm bundled with the node running the bootstrap (`npmCommand`: its `npm-cli.js` beside `process.execPath`, no shell, that node's directory first on PATH; `npm` on PATH only when that node ships none) — the update chain runs with the dashboard server's environment, whose PATH need not reach `npm.cmd` on Windows. The UI build below and the doctor's npm row use the same npm (never into the plugin cache, which is garbage-collected and replaced on every update). `--ignore-scripts` keeps the root `postinstall` from pulling the whole Vite client toolchain.
 6. **Legacy hook cleanup** — see above.
 7. **CLI** — write the `~/.local/bin/agenthub` launcher, never clobbering a `agenthub` the bootstrap did not write. It pins the node that ran the bootstrap and falls back to `node` on `PATH` when that binary is gone (a Homebrew Cellar or nvm path deleted by an upgrade).
-8. **UI build** — `client/` is copied into the runtime dir and built there (`npm ci --include=dev --ignore-scripts`, falling back to `npm install` with the same flags, then `tsc -b` and `vite build` run by the bootstrap's own `node` — the client's `build` script without `npm run`), landing in `runtime/client-dist` — the same directory the server serves from. Nothing in it looks `node` up on PATH: on Windows an install script or a `.bin/*.cmd` shim runs through cmd.exe, and the server's PATH may have no `node` (esbuild's postinstall failed there with "'node' is not recognized" and the update kept the old UI); esbuild, the only dependency with an install script, finds its binary without it. `--include=dev` is required, not tidy: the entire build toolchain (`vite`, `typescript`, `@vitejs/plugin-react`, `tailwindcss`, `@types/*`) is a devDependency, and npm reads an inherited `NODE_ENV=production` as `--omit=dev`. Every bootstrap descended from the running server carries that env — the self-update chain, and any `SessionStart` fired by a `claude` inside a tmux pane the server started — so without the flag the install drops `vite` and the build dies on missing types. Skipped when a bundle for the current version already exists; a failure here does not fail the bootstrap (the API still works, and `/agenthub-open` can retry) but does mean `/run` 404s until it's fixed — and when a bundle from the PREVIOUS version is still on disk the failure is quieter still: the dashboard keeps serving the old UI against the new API, visible only as the Settings page's `UI build vX (rebuild client to match)` line.
+8. **UI build** — `client/` is copied into the runtime dir and built there (`npm ci --include=dev --ignore-scripts`, falling back to `npm install` with the same flags — skipped while `client/package.json` and its lockfile are unchanged, because the installed tree in `runtime/client-src/node_modules` is kept with a `.agenthub-deps` stamp; a failed build drops the stamp so the next one installs from clean — then `vite build` run by the bootstrap's own `node`: the vite half of the client's `build` script without `npm run`. The `tsc -b` half is the release's job, not every user's: it emits nothing the bundle needs and cost ~25 s and ~900 MB per update), landing in `runtime/client-dist` — the same directory the server serves from. Nothing in it looks `node` up on PATH: on Windows an install script or a `.bin/*.cmd` shim runs through cmd.exe, and the server's PATH may have no `node` (esbuild's postinstall failed there with "'node' is not recognized" and the update kept the old UI); esbuild, the only dependency with an install script, finds its binary without it. `--include=dev` is required, not tidy: the entire build toolchain (`vite`, `typescript`, `@vitejs/plugin-react`, `tailwindcss`, `@types/*`) is a devDependency, and npm reads an inherited `NODE_ENV=production` as `--omit=dev`. Every bootstrap descended from the running server carries that env — the self-update chain, and any `SessionStart` fired by a `claude` inside a tmux pane the server started — so without the flag the install drops `vite` and the build dies on missing types. Skipped when a bundle for the current version already exists (`clientBuild` in `state.json`, recorded only after a build that did not fail, so the next session retries a failed one); a failure here does not fail the bootstrap (the API still works, and `/agenthub-open` can retry) but does mean `/run` 404s until it's fixed — and when a bundle from the PREVIOUS version is still on disk the failure is quieter still: the dashboard keeps serving the old UI against the new API, visible only as the Settings page's `UI build vX (rebuild client to match)` line.
 9. **Server** — when the user has turned **Start AgentHub automatically** on
    (Settings → System; OFF by default), install/refresh the
    `agenthub.service` unit on Linux — on macOS with a GUI login, the
@@ -149,10 +149,21 @@ waits on an install:
 The first run takes a few minutes — most of it is the UI build, which is what
 makes client-only routes (`http://localhost:4820/run`, and every other page)
 work the moment `claude` starts, with no manual `/agenthub-open` step. Progress
-goes to `~/.claude/agenthub/runtime/bootstrap.log`; the UI build's own
-output goes to `client-build.log` next to it; the server's own output goes to
+goes to `~/.claude/agenthub/runtime/bootstrap.log`; npm's own output goes to
+`npm-install.log` and the UI build's to `client-build.log` next to it; the server's own output goes to
 `server.log`.
 
+**A failure is reported once, in the session.** The worker's output is
+discarded, so a Node that is too old, a failed npm install or a failed UI build
+is recorded in `runtime/last-failure.json` (kind, message, log file). The NEXT
+foreground pass — its stdout is what Claude Code reads — prints it as a
+SessionStart `{"systemMessage": "AgentHub: … — details in <log>. Run
+/agenthub-doctor …"}` and marks it reported, so the same failure is not repeated
+every session; a different failure is reported again, and a bootstrap that
+succeeds removes the file. The steady state prints nothing, and the report is a
+single small read (plus one write, only when there is something to show) — the
+fast path still spawns nothing.
+
 ### A retired build never acts
 
 `claude plugin update` unpacks the new version into a **new** cache directory

docs/README.md

@@ -9,8 +9,7 @@ Comprehensive documentation for the Agent Dashboard project.
 - [Architecture Overview](../ARCHITECTURE.md) - System design and technical reference
 - [I18N Architecture](./I18N.md) - Internationalization architecture and usage guide
 - [CLI Reference](./CLI.md) - The `agenthub` terminal CLI: every command, discovery, safety model
-- [Setup Guide](../SETUP.md) - Installation and configuration
-- [Installation](../INSTALL.md) - Detailed installation instructions
+- [Installation & setup](../INSTALL.md) - Installation, configuration and troubleshooting
 
 ---
 
@@ -20,7 +19,7 @@ Comprehensive documentation for the Agent Dashboard project.
 
 ```mermaid
 graph TB
-    Start[Start Here] --> Setup[SETUP.md<br/>Installation & Config]
+    Start[Start Here] --> Setup[INSTALL.md<br/>Installation & Config]
     Start --> Architecture[ARCHITECTURE.md<br/>System Design]
     
     Setup --> Client[Client README<br/>React UI docs]
@@ -62,7 +61,7 @@ graph TB
 
 ```mermaid
 graph LR
-    A[New to Project] --> B[Read SETUP.md]
+    A[New to Project] --> B[Read INSTALL.md]
     B --> C[Install Dependencies]
     C --> D[Run npm run dev]
     D --> E[Open localhost:5173]
@@ -75,7 +74,7 @@ graph LR
 
 1. Install the system packages — `node` (20+), `git`, and `tmux` (the run
    console has no fallback without it): [INSTALL.md → Step 0](../INSTALL.md#step-0--system-packages)
-2. Read [SETUP.md](../SETUP.md)
+2. Read the rest of [INSTALL.md](../INSTALL.md)
 3. Run `npm run setup`
 4. Run `npm run dev`
 5. Open browser to `http://localhost:5173`
@@ -287,7 +286,7 @@ graph LR
 **Documentation:**
 - [Hook System Guide](./HOOKS.md)
 - [Hook Processing](../server/README.md#hook-processing)
-- [Installation](../SETUP.md#install-hooks)
+- [Installation](../INSTALL.md#hook-auto-installation)
 
 ---
 
@@ -441,4 +440,4 @@ This documentation covers:
 - ✅ **Internationalization** - Language resources, switching flow, locale formatting, rollout checklist
 - ✅ **Development guides** - Setup, testing, contributing
 
-**Start with:** [SETUP.md](../SETUP.md) for installation, then explore specific areas based on your role.
+**Start with:** [INSTALL.md](../INSTALL.md) for installation, then explore specific areas based on your role.