Merge branch 'perf/local-opt': faster local install and updates, lighter polling and ingest
This commit is contained in:
+19
-7
@@ -310,7 +310,7 @@ graph TD
|
||||
|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `server/index.js` | Express app setup, middleware, route mounting, static file serving in production, HTTP server creation. Static middleware sets explicit `Cache-Control` headers — `immutable` for `/assets/*`, `no-cache, must-revalidate` for `index.html` / `sw.js` / `manifest.json`, a short revalidation window otherwise — so a rebuild always replaces the in-browser bundle without a hard refresh. Runs a periodic maintenance sweep — cadence derived from `DASHBOARD_STALE_MINUTES` (¼ of the threshold, clamped to 60 s – 5 min, default ~45 min) — that abandons stale sessions with transcript cache eviction and scans active sessions for new compaction entries by reading `sessions.transcript_path` directly (an O(active sessions) lookup; the previous `SELECT DISTINCT json_extract(events.data,'$.transcript_path')` scan grew with the events table and is gone). **Error detection watchdog** runs every 15 seconds: finds active sessions with no recent hook events (>10 s stale), re-reads their transcript files looking for API errors (401 auth, rate limits, quota exhaustion), derives transcript paths from session `cwd` for imported sessions, and marks sessions/agents as `error` when API errors are found — catches cases where the CLI doesn't fire a hook after API errors. The same watchdog also performs **user-interrupt (Esc) recovery**: `Esc` fires no hook, so a cancelled turn would otherwise leave the main agent stuck `working`. It detects this two ways — (a) the transcript's `[Request interrupted by user]` marker, surfaced as `result.pendingInterrupt` from `TranscriptCache` (computed from transcript ordering alone, immune to server/transcript clock skew), and (b) an idle-working fallback for an Esc pressed *before any output* (which writes no marker): when the main agent has been `working` with `current_tool` null and neither a hook event nor the transcript mtime has advanced for `DASHBOARD_WORKING_IDLE_SECONDS` (default 120) — and moves the session to **Waiting** (agent → `waiting`, `awaiting_input_since` stamped) with an `Interrupted` event. Triggers legacy session import (with active-session detection for recently-modified JSONL files) and compaction backfill on startup, plus a boot **liveness reap** — immediately for rows from a previous run and again ~5 s later for rows the startup sync just imported (see the `routes/hooks.js` row) — so sessions that died while the dashboard was down never render as Waiting. Also starts the **Remote Data Sources** sync poller (`startRemoteSourceSync`) that pulls each enabled remote source on an interval — `DASHBOARD_REMOTE_SYNC_MS` (default 60000; 0 disables) — reusing `server/lib/remote-sync.js`. **Graceful shutdown** (SIGTERM/SIGINT) tears down in order: drop realtime clients first (`closeWebSocket` terminates WS clients so their sockets release), then `httpServer.close()` to stop new connections, then `httpServer.closeAllConnections()` to drop lingering keep-alive sockets so `close()` fires promptly, and only **then** close SQLite — inside the `close()` callback, after the HTTP server has drained. Closing the DB before drain made in-flight requests throw `The database connection is not open`; leaving WS/keep-alive sockets open stalled the shutdown until the 5 s force-exit backstop (the "waiting for graceful termination" hang under `node --watch`). A second signal forces an immediate exit |
|
||||
| `server/openapi.js` | OpenAPI 3.0.3 document generator for the backend API (metadata, schemas, endpoint paths), merging supplementary fragments from `server/openapi-extra/` in `createOpenApiSpec()`. Feeds the raw spec endpoint (`/api/openapi.json`), Swagger UI (`/api/docs`), **ReDoc** (`/api/redoc`, served via `server/lib/redoc.js` with a self-hosted bundle — never a CDN), and the committed `openapi.yaml` regenerated by `npm run openapi:yaml` |
|
||||
| `server/lib/redoc.js` | Serves the **ReDoc** API reference (`/api/redoc`) as a self-hosted three-panel rendering of the OpenAPI spec, with the ReDoc bundle served locally from `/api/redoc/redoc.standalone.js` (bundled via the `redoc` dependency, never a CDN) so the reference works fully offline / air-gapped |
|
||||
| `server/lib/redoc.js` | Serves the **ReDoc** API reference (`/api/redoc`) as a self-hosted three-panel rendering of the OpenAPI spec, with the ReDoc bundle served locally from `/api/redoc/redoc.standalone.js` (vendored under `server/vendor/redoc/` — the `redoc` npm package is not a dependency — never a CDN) so the reference works fully offline / air-gapped |
|
||||
| `server/openapi-extra/` | Supplementary OpenAPI path/schema fragments merged into the spec by `createOpenApiSpec()` — covers `cc-config.js`, `push.js`, `run.js`, and `misc.js` route groups |
|
||||
| `server/db.js` | SQLite connection with WAL mode, schema migration (CREATE TABLE IF NOT EXISTS + ALTER TABLE for column additions), all prepared statements as a reusable `stmts` object. Tries `better-sqlite3` first, falls back to `node:sqlite` via `compat-sqlite.js`. Migrations use literal defaults for ALTER TABLE since SQLite does not support expressions like `strftime()` in column defaults added via ALTER TABLE |
|
||||
| `server/compat-sqlite.js` | Compatibility wrapper that gives Node.js built-in `node:sqlite` (`DatabaseSync`) the same API as `better-sqlite3` — pragma, transaction, prepare. Used as automatic fallback when the native module is unavailable (Node 22+) Maps every better-sqlite3 API the server uses — `function` (the `trim_event_data` SQL function `db.js` registers at boot), `stmt.pluck()`, the `readonly`/`fileMustExist` open options — and `db.js` exports the chosen class as `Database` so other SQLite files (agy's conversations) open with the same driver. A plugin install has NO better-sqlite3 build, so this is production there; `server/__tests__/compat-sqlite.test.js` boots `db.js` with better-sqlite3 blocked because the rest of the suite never runs on it. |
|
||||
@@ -1066,7 +1066,7 @@ flowchart TD
|
||||
- Uses Node.js `http` module directly — no dependencies.
|
||||
- Resolution order is **env override → discovery file → default**:
|
||||
- `CLAUDE_DASHBOARD_PORT` forces a single target (no fan-out, no discovery).
|
||||
- Otherwise `server/lib/server-info.js` reads `~/.claude/.agenthub.json`, prunes dead-PID entries, and returns one live `port` per unique `dataDir` (lowest port wins when two instances share `~/.claude/agenthub`). The handler POSTs to each returned port in parallel. The other instance learns of a lane the hooks moved through `broadcastForeignLaneChanges` (`server/routes/lanes.js`): every `DASHBOARD_LANE_SYNC_MS` (2 s) it reads `PRAGMA data_version`, which moves only on another connection's commit, and re-broadcasts each lane whose `updated_at` changed (and `removed` for a vanished one) to its own browsers. Sessions and events are not re-broadcast this way — a page on that instance shows them on its next fetch.
|
||||
- Otherwise `server/lib/server-info.js` reads `~/.claude/.agenthub.json`, prunes dead-PID entries, and returns one live `port` per unique `dataDir` (lowest port wins when two instances share `~/.claude/agenthub`). The handler POSTs to each returned port in parallel. The other instance learns of a lane the hooks moved through `broadcastForeignLaneChanges` (`server/routes/lanes.js`): every `DASHBOARD_LANE_SYNC_MS` (2 s) — while `peersSharingDataDir()` finds another live dashboard on its data dir, which it checks every 30 s — it reads `PRAGMA data_version`, which moves only on another connection's commit, and re-broadcasts each lane whose `updated_at` changed (and `removed` for a vanished one) to its own browsers. Sessions and events are not re-broadcast this way — a page on that instance shows them on its next fetch.
|
||||
- If neither yields anything, the handler falls back to `4820`.
|
||||
- Per-target promises never reject — a dead listener can't starve the others, and the handler can wait on `Promise.all` for clean exit timing.
|
||||
|
||||
@@ -1223,7 +1223,12 @@ common "nothing changed" tick is just a handful of `stat` calls), funnels them
|
||||
through the same `parseSessionFile` + `importSession` pipeline as every other
|
||||
path, and broadcasts `session_created` for newly imported sessions /
|
||||
`session_updated` for grown ones — the same events hooks emit, so the UI
|
||||
refreshes live. All timers and watchers are `unref`'d and best-effort; nothing
|
||||
refreshes live. Two kinds of transcript are not re-parsed: one whose session
|
||||
hooks are feeding **live** (status `active`, newest event a hook's, a hook
|
||||
within `DASHBOARD_SYNC_LIVE_HOOK_MS`, default 5 min) is left until it goes
|
||||
quiet or ends — then caught up once — and one whose session a hook created
|
||||
(no `imported` flag, which `importSession` skips anyway) only has its durable
|
||||
snapshot refreshed, never a parse. All timers and watchers are `unref`'d and best-effort; nothing
|
||||
here blocks shutdown or can take down the server.
|
||||
|
||||
### Upload request sequence
|
||||
@@ -1579,7 +1584,11 @@ re-point if that changes.
|
||||
`DASHBOARD_CODEX_POLL_MS` beat, a first pass over every journal in the
|
||||
background at startup). Each file's read offset and parse state live in the
|
||||
`codex_rollouts` table, so an unchanged file is never re-read and a grown one is
|
||||
read from where it stopped. `token_count` events are summed per `turn_context`
|
||||
read from where it stopped (a pass loads every row's size/mtime in one query).
|
||||
The first pass and one every five minutes walk every journal; the poll's other
|
||||
passes stat only the two newest day folders plus the journals the table saw
|
||||
written in the last hour (or not yet read to the end), so a thread resumed
|
||||
after an hour's quiet in an older folder is picked up by the next full pass. `token_count` events are summed per `turn_context`
|
||||
model — `input_tokens` includes `cached_input_tokens` (stored as input − cached
|
||||
plus cache_read) and `output_tokens` includes reasoning. A **fork** copies its
|
||||
parent's whole history into its own file, token events included, so every total
|
||||
@@ -1654,7 +1663,9 @@ the binary's own proto descriptors.
|
||||
|
||||
**Sessions and tokens** (`server/lib/antigravity-usage.js`,
|
||||
`startAntigravityUsagePoll`, every `DASHBOARD_ANTIGRAVITY_POLL_MS`, default
|
||||
15000, `0` disables). agy writes one SQLite file per conversation,
|
||||
15000, `0` disables; after a pass that read nothing, wrote nothing and left no
|
||||
fresh session to settle it slows to once a minute until one finds work —
|
||||
`server/lib/backoff-poll.js`). agy writes one SQLite file per conversation,
|
||||
`~/.gemini/antigravity-cli/conversations/<conversation id>.db`. Each is opened
|
||||
read-only whenever its mtime (or its non-empty `-wal`'s) changes;
|
||||
`antigravity_conversations` holds the mtime read, so a session the user deleted
|
||||
@@ -1743,7 +1754,8 @@ is ever inferred.
|
||||
## Cursor Sessions
|
||||
|
||||
`server/lib/cursor-usage.js` (`startCursorUsagePoll`, every
|
||||
`DASHBOARD_CURSOR_POLL_MS`, default 15000, `0` disables) reads the agent
|
||||
`DASHBOARD_CURSOR_POLL_MS`, default 15000, `0` disables; idle passes slow it to
|
||||
once a minute like the Antigravity poll) reads the agent
|
||||
transcripts Cursor writes — the CLI (`cursor-agent`) and the editor's agent both —
|
||||
at `~/.cursor/projects/<project slug>/agent-transcripts/<id>/<id>.jsonl`
|
||||
(`AGENTHUB_CURSOR_DIR` overrides `~/.cursor`). Each becomes a `sessions` row with
|
||||
@@ -2225,7 +2237,7 @@ sequenceDiagram
|
||||
| Component | Responsibility |
|
||||
| --- | --- |
|
||||
| **`server/lib/update-check.js`** | `getUpdatesStatus(root?, { skipFetch?, latestRelease? })` + `applyUpdate(root?)`. Runs every git call via `execFile` (no shell, 10s–120s timeouts) and reads the newest release over `fetch` with an `AbortSignal.timeout` — `fetchLatestRelease` NEVER throws, resolving to `{ok:false, reason}` (`no_remote`, `remote_not_http`, `no_releases`, `unreachable`, `bad_payload`, `http_<status>`) so a status poll degrades instead of erroring. `releaseApiUrl` derives the endpoint from a repository URL (https, scp-style and `ssh://` forms) and returns null for anything not owner/repo shaped, which is what keeps local-path remotes off the network entirely. `fetchLatestRelease` finds that URL via the `origin` remote; `fetchReleaseFromUrl` takes one directly, which is how the **git-free plugin path** works: `pluginMarketplaceUrl(mkt, plugin)` reads `known_marketplaces.json` → the marketplace's `.claude-plugin/marketplace.json` → the entry's `source.url`, returning null (never a guessed host) for a `github` `owner/repo` shorthand and never throwing. **Order matters:** `detectPluginInstall` runs FIRST in `getUpdatesStatus`, ahead of the `.git` probe — a plugin cache unpacked from an archive has no `.git`, and probing first is the 1.5.1 bug that made `update_available` permanently false. `compareVersions` orders numeric triples and strips a leading `v`, since this repo carries both `v1.5.0` and `1.4.6`. `isShallowClone` gates every commit-distance calculation; `detectPluginInstall` reads the marketplace and plugin name out of the cache path. **Branch- and fork-aware** as before: resolves `<remote>/master`/`/main`/`/HEAD`, reads the current branch and its tracked upstream, and shapes `manual_command` per situation — the `claude plugin ...` commands for `plugin_managed`; `git fetch <remote> --tags && git merge --ff-only <tag>` for `release_available`; `git pull --ff-only` when the branch tracks the canonical ref; `git fetch <remote> && git merge --ff-only <ref>` for forks; `git fetch <remote>` only on a feature branch or detached HEAD. Non-git installs, missing remotes, fetch failures, and unresolvable upstream refs are returned as soft payloads — never throws. |
|
||||
| **`server/update-scheduler.js`** | Ticks the lib every `DASHBOARD_UPDATE_CHECK_INTERVAL_MS` (default 300 000, floor 60 000). First tick is scheduled 8s after server start with `.unref()` so it doesn't block shutdown. Broadcasts only when the fingerprint `{update_available, remote_sha, commits_behind, fetch_error, manual_command}` changes — `manual_command` is included so situation transitions (e.g. user switches branches, or adds an `upstream` remote) trigger a re-broadcast even when the SHA and commit count are unchanged. Emits a framed message to stdout on "up-to-date → behind" transitions, and only suggests "restart the dashboard" when the printed command actually rewrites the working tree. `DASHBOARD_UPDATE_CHECK=0\|false\|off` disables the scheduler entirely. |
|
||||
| **`server/update-scheduler.js`** | Ticks the lib every `DASHBOARD_UPDATE_CHECK_INTERVAL_MS` (default 3 600 000 = 1 h, floor 60 000). A tick with no `/ws` browser connected (`getConnectionCount`) is skipped, and the first browser to connect afterwards (`onConnection` in `server/websocket.js`) gets one check 3 s later if the last one is older than the interval; the manual `POST /api/updates/check` is unaffected. First tick is scheduled 8s after server start with `.unref()` so it doesn't block shutdown. Broadcasts only when the fingerprint `{update_available, remote_sha, commits_behind, fetch_error, manual_command}` changes — `manual_command` is included so situation transitions (e.g. user switches branches, or adds an `upstream` remote) trigger a re-broadcast even when the SHA and commit count are unchanged. Emits a framed message to stdout on "up-to-date → behind" transitions, and only suggests "restart the dashboard" when the printed command actually rewrites the working tree. `DASHBOARD_UPDATE_CHECK=0\|false\|off` disables the scheduler entirely. |
|
||||
| **`server/routes/updates.js`** | Three endpoints: `GET /status` (read-only check), `POST /check` (check + broadcast), `POST /apply`. `/apply` has **two modes**, chosen by how this copy was installed, never by the request body: **checkout** — fast-forward onto the release tag, `npm run setup`, `npm run build` in production (both through the bootstrap's `npmCommand` with `NODE_ENV` dropped — `npmInvocation` in `server/lib/update-check.js` — since the server's `NODE_ENV=production` makes npm omit the client's devDependency build toolchain, and a bare `npm.cmd` is EINVAL under `execFile` on Windows), then `scheduleRestart()`; **plugin** — `applyUpdate`'s `plugin_managed` refusal is caught and `schedulePluginUpdate()` runs instead, answering `200 {applied:true, mode:"plugin"}`. The plugin branch fires only when there is genuinely something to pull (`update_available` plus a marketplace and plugin name parsed from the cache path) — a `plugin_managed` refusal with nothing newer published stays a 409 rather than spawning a chain that would stop the server for nothing. Otherwise 409 `UPDATE_NOT_APPLICABLE` with `reason` `"up_to_date"`, `"not_fast_forwardable"`, or `"plugin_managed"`. Both modes write the response BEFORE anything stops the process. Mounted under `/api/*`, so all three ride the default loopback bind as their trust boundary and are gated by `DASHBOARD_TOKEN` when one is set. |
|
||||
| **`server/lib/plugin-update.js`** + **`scripts/plugin-update-helper.js`** | The plugin-cache update chain. The lib only spawns the helper detached (cwd = the runtime dir, **not** the plugin root, which is about to be superseded) and clears any previous verdict first so a stale success can't be read back as this run's result. All work is in the helper because its last step SIGTERMs the server that asked for the update — the same split as `self-restart.js`, one step further. The helper re-validates the marketplace and plugin names against `/^[A-Za-z0-9][\w.-]*$/` (they reach an argv across a process boundary), resolves `claude` on PATH or at `~/.local/bin/claude` (a hook-spawned server does not inherit the login shell's PATH), picks the newest cache directory that isn't the old root and does contain `scripts/plugin-bootstrap.js`, and records **every** outcome — including an unexpected throw — to `plugin-update.json` under `~/.claude/agenthub/runtime/`, outside the cache so it survives the swap. Steps: `validate`, `resolve_claude`, `marketplace_update`, `plugin_update`, `already_current` (`ok:true`; nothing was pulled and the old server was left running), `bootstrap`, `done`. `readLastPluginUpdate()` surfaces it as `last_plugin_update` on the status payload — without it a failed chain is indistinguishable from a slow one. |
|
||||
| **`UpdateNotifier.tsx`** | Modal. Hydrates from `api.updates.status()` on mount and mirrors the payload back into the local `eventBus` so the Sidebar can listen without a second git fetch. Subscribes to `update_status` WS frames for ongoing sync. Keeps the dismissal key in `localStorage` (`agent-monitor-update-dismissed-sha`) and in React state — the **release tag** when one was read, else the upstream SHA, so a newer release re-opens the prompt while a dismissed one stays dismissed; a window event `dashboard:reset-update-dismissal` from the Sidebar clears both. ESC / backdrop click dismisses. |
|
||||
|
||||
Reference in New Issue
Block a user