perf: v2.5.10 — ready for 10+ concurrent users, leaner locally

Measured on a copy of a real 1.2 GB / 253K-event database, 10 simulated
browsers + 10 hooks/s for 90 s: event-loop delay p99 1239 ms → 32 ms (max
2.0 s → 94 ms), 266 failed requests → 0, 74 → 111 requests/s served, server
RSS 633 → 346 MB. Entry bundle 975 KB raw → 394 KB (106 KB brotli).

Server
- /api/analytics: the events-table scans run in the read worker
  (scoped-stats eventAggregates, read-pool job analyticsEvents); runRead takes
  maxAgeMs — analytics 10 s, /api/workflows 30 s — keyed by the args
- /api/workflows/skill-usage: +event_type and a tool_name range instead of
  LIKE, so it uses idx_events_tool (784 ms → a few ms)
- idx_events_session_type_tool (session_id, event_type, tool_name,
  created_at) replaces idx_events_session_type: covering reads for session
  stats, the hook dedupes and the import resume point (1.2 s → 20 ms); 64 MB
  cache_size and 256 MB mmap_size on both connections
- WAL: the server sets wal_autocheckpoint = 0 and checkpoints PASSIVE every
  DASHBOARD_WAL_CHECKPOINT_MS (5 s) from the read worker's own writable
  connection; a failed checkpoint restores the automatic one (hook COMMIT p95
  109 ms → 15 ms at 100 hooks/s)
- tmux: one list-sessions answers a whole GET /api/lanes and the heal beat
  (liveSessions), shared for 1 s and cleared by this process's new/kill —
  GET /api/run's sync spawns had blocked 60 s of a 90 s run; the heal beat
  skips tmux when nothing runs; attach options chained into 2 spawns
- hooks: TurnDuration/APIError deduped against one read per hook instead of a
  query per transcript entry (23 ms/hook on a 763-turn session); one lane
  scan per hook; hourly upserts skip rows that did not grow; frames are sent
  after the transaction commits (and after the pane binding); new_event
  carries its row id; a split pane's (slot 1–9) hooks no longer rebind the
  lane's session_id
- watchdog liveness probe runs ps asynchronously; transcript newline scan via
  indexOf; startup subagent backfill fingerprints the files it already checked
- pty attach: backpressure (pause past 1 MB unsent, resume under 256 KB)
- static: hashed assets served from their pre-compressed .br/.gz sibling

Client
- hidden tabs neither refetch on WS events nor poll (useLiveRefresh, panes,
  lane cards, git header, changes panel, settings); no per-tab hourly
  /updates/check (the server already checks)
- refetch storms throttled (workspace tree/session list 3 s, /api/run only on
  a run-key change, notification pane lookup only when shown); WS reconnect
  jitter; terminal retry backoff 1.5 → 30 s, stops on 1008
- heavier throttles for aggregates (15 s) and detail views; Monitor keepIfSame;
  GET in-flight dedupe in request()
- lazy Dashboard/History, xterm (with the first terminal), lane modals, Sagi
  (only while enabled); only the active locale loads; Workspace chunk
  prefetched on a /workspace deep link
- Sagi: no work while disabled, pupil via CSS variables, memo avatar
- build: .br/.gz siblings, sourcemap hidden; service worker no longer caches
  /assets/ (cache v3 drops the old one)

Tests: scripts/test-server.js drops the GIT_DIR-family variables a hook
exports, so a suite run from a linked worktree's pre-commit can no longer
commit into the real repository or set core.bare.

Docs: ARCHITECTURE, DATABASE, API, DEPLOYMENT, client README,
docs/plans/perf/README.md (audit, checklist, results).
This commit is contained in:
2026-10-07 13:24:15 +07:00
parent 8fdda66893
commit ca821008e2
75 changed files with 1711 additions and 699 deletions
+7 -3
View File
@@ -333,7 +333,7 @@ graph TD
| `lib/webhooks.js` | Universal webhook delivery engine driven by the provider registry. `buildRequest()` resolves the URL, formats the provider-native payload, and assembles headers (provider auth headers + generic-family custom headers + optional HMAC-SHA256 signature via `X-Webhook-Signature` / `X-Webhook-Timestamp`). `dispatchAlert()` fans a fired alert out to every enabled, in-scope target (optional per-rule scoping via `rule_ids`); each `deliver()` POSTs with an `AbortController` timeout and bounded retry/backoff (retries transport errors / 429 / 5xx, never other 4xx) and records the attempt-chain outcome in `webhook_deliveries` (pruned to the newest 2000 rows). Delivery is detached and fully fail-safe — it never throws into the alert path. Enabled targets are cached like alert rules; tunables (`WEBHOOK_TIMEOUT_MS`, `WEBHOOK_MAX_ATTEMPTS`, `WEBHOOK_RETRY_BASE_MS`) are env-overridable. `sendTest()` awaits a synthetic delivery for the test endpoint |
| `routes/workflows.js` | Aggregate workflow visualization data (agent orchestration graphs, tool transition flows, collaboration networks, workflow pattern detection, model delegation, error propagation, concurrency timelines, session complexity metrics, compaction impact). Accepts `?status=active\|completed` query parameter to filter all data by session status and `?sections=a,b` to compute only those parts. The `GET /` bundle itself lives in `lib/workflow-stats.js` and runs in the read worker (`lib/read-pool.js`). `GET /api/workflows/skill-usage` reports per-skill and per-plugin call counts, counting `PreToolUse` rows only so a call is not double-counted against its `PostToolUse` twin. Per-session drill-in endpoint with agent tree, tool timeline, and event details |
| `lib/workflow-stats.js` | The eleven `GET /api/workflows` aggregations as pure reads over a passed-in connection (`computeWorkflowData(conn, status, sections)`), so they run in the read worker. `stats.topFlow` is the head of `toolFlow.transitions`: tool pairs come from one `LEAD() OVER (PARTITION BY session_id ORDER BY id)` pass — never a correlated `MIN(id) … id > ?` per event, which is quadratic in a session's size |
| `lib/read-pool.js` | Heavy read-only jobs off the event loop: one worker thread with its own `query_only` connection (same driver order as `db.js`), identical in-flight calls shared, a job past `AGENTHUB_READ_TIMEOUT_MS` (60 s) refused `EREADTIMEOUT` and its worker replaced, inline when `AGENTHUB_READ_WORKER=0` or a worker cannot start. Add a job to `JOBS` to move another aggregate there |
| `lib/read-pool.js` | Heavy read-only jobs off the event loop: one worker thread with its own `query_only` connection (same driver order as `db.js`), identical in-flight calls shared, a caller's `maxAgeMs` reusing a result that young (`/api/workflows` 30 s, `/api/analytics`'s events scans 10 s — args are the key, so a member's scoped result is never shared), a job past `AGENTHUB_READ_TIMEOUT_MS` (60 s) refused `EREADTIMEOUT` and its worker replaced, inline when `AGENTHUB_READ_WORKER=0` or a worker cannot start. Add a job to `JOBS` to move another aggregate there. The server's WAL checkpoint (`walCheckpoint`, every `DASHBOARD_WAL_CHECKPOINT_MS`, default 5 s) also runs here, on a second writable connection of the worker: the main connection sets `wal_autocheckpoint = 0`, so no hook's `COMMIT` pays for a checkpoint; a failed checkpoint turns the automatic one back on |
| `lib/transcript-cache.js` | Stat-based JSONL transcript cache with incremental byte-offset reads. Shared between `hooks.js` (token extraction on every event) and the periodic compaction scanner (`index.js`). Extracts tokens, compaction entries, API errors (`isApiErrorMessage` + raw error responses), turn durations (`system` subtype `turn_duration`), thinking block counts, usage extras (service_tier, speed, inference_geo), user-interrupt markers (the transcript `[Request interrupted by user]` entry / `interruptedMessageId` field — surfaced as `pendingInterrupt`, computed from transcript ordering: latest interrupt vs latest real turn activity, both on Claude Code's clock), and the latest session title — `custom-title` (`/rename`, `claude -n`, picker Ctrl+R) and `ai-title` (auto / plan-accept), append-only so the last value wins, carried through both full and incremental reads — plus the session's **first user prompt** (`firstUserMessage`: tool-result, meta/caveat, slash-command plumbing, compact-summary, and interrupt entries skipped; whitespace-collapsed, capped at 500 chars; first value wins across incremental reads), used as a fallback descriptor for placeholder-named sessions/agents. Uses `(path, mtime, size)` cache key — unchanged files return cached results instantly, grown files only parse new bytes, shrunk files (compaction) trigger full re-read. Each cache entry stores **only** `{mtimeMs, size, bytesRead, result}` — the previous shape that duplicated every growable array at both the top level and inside `result` is gone, halving steady-state memory per entry. Per-entry growable arrays (`turnDurations`, `errors`, `compaction.entries`, `usageExtras.*`) are bounded to `TRANSCRIPT_CACHE_MAX_ARRAY_LEN` (default `1000`, tail-kept) — older items remain in the `events` table thanks to hook dedup, so the cap only affects the in-memory view. Trimming runs both during parse (when an array reaches `2 * MAX_ARRAY_LEN`, amortized O(N)) and at finalize, so even a fresh full-file parse on a multi-day session cannot accumulate an unbounded transient before returning. **Chunked sync byte-stream reader** (`_streamRange`, 4 MiB chunks split on `0x0A` bytes — safe across UTF-8 multibyte sequences — with a growable per-line byte buffer capped at 64 MiB) replaces the previous `readFileSync("utf8")` so transcripts larger than V8's max JS string length (~512 MiB on 64-bit Node 20) parse without aborting Node with `FATAL ERROR: v8::ToLocalChecked Empty MaybeLocal`. Both full and incremental reads share the same line-level state machine (`_initParseState` / `_consumeLine` / `_finalizeState`). LRU eviction caps at 200 entries. Entries evicted on SessionEnd and abandoned session cleanup |
| `lib/session-liveness.js` | Process-liveness probe for the watchdog's dead-session reap. `probeLiveCwds()` enumerates running `claude` CLI processes (`ps -Ao pid=,args=`, then `lsof -a -p <pids> -d cwd -Fn` on macOS or `/proc/<pid>/cwd` on Linux) and returns the set of their working directories; `isClaudeCommand()` matches the bare binary and `node`/`bun`-launched shims while rejecting lookalikes (`claude-mem`, `Claude.app`). Fail-safe by contract: returns `available: false` (callers must change nothing) on Windows, inside containers (reuses `isInsideContainer` from `scripts/install-hooks.js` — host processes are invisible there, so an empty list would lie), on `ps`/`lsof` failure, or when `DASHBOARD_LIVENESS_PROBE=0` |
| `bin/agenthub.js` | Dependency-free umbrella CLI (`agenthub <command>`) exposing the full dashboard surface in the terminal: monitoring (health / stats / kanban / tail via short-interval event polling), data browsing (sessions, per-session detail with an indented agent tree + cost + recent events, agents, events), insights (analytics, workflow intelligence, dynamic Workflow-tool runs, per-model cost), alerts + rules + webhook test probes, pricing CRUD, imports (rescan / scan-path), and administration (doctor, info, export, cleanup, reinstall-hooks, clear-data). Linked globally by `npm run setup` via a fail-soft `npm link` (`link-cli` script). Discovers the live server through `server/lib/server-info.js` (`~/.claude/.agenthub.json`, PID-liveness-checked) with `CLAUDE_DASHBOARD_PORT` / `DASHBOARD_PORT` env overrides and a 4820 fallback; renders a full terminal UI — box-drawn width-fitted tables with right-aligned numeric columns, status icons, inline bar charts (stats / analytics / cost), `├─`/`└─` agent trees, and a TTY spinner for `start` — whose ANSI styling degrades to plain text when piped (`--no-color` / `NO_COLOR` / `FORCE_COLOR` / `AGENTHUB_COLOR` respected); the one destructive command (`clear-data`) refuses to run without `--yes`. Server lifecycle: `agenthub status` shows a ●/○ up/down indicator and `agenthub start` boots a detached production server (waits for /api/health, logs to `data/agenthub-server.log`). `agenthub repl` (aliases `shell` / `i`) opens an interactive shell — a readline prompt with tab-completion (commands / subcommands / flags), persisted arrow-key history (`data/.agenthub_repl_history`), and a live server-status prompt; each entered line runs as a short-lived child `agenthub` process (via `runCommand` dispatch), so an offline refusal or a blocking `tail` cannot take the shell down, and piped input runs each line in order then exits at EOF. When the server is down, read-only commands (sessions / session / agents / events / kanban / stats / pricing list / alerts list / rules / export / doctor) fall back to direct SQLite reads of `data/dashboard.db` under a ⚠ Offline-mode banner — the connection is opened without SQLite readonly mode so a live WAL stays visible, and is SELECT-only by construction — while server-only commands (tail, analytics, workflows, runs, cost, mutations) refuse with the specific reason |
@@ -911,10 +911,10 @@ erDiagram
| `idx_agents_status` | agents | `status` | Kanban board column queries |
| `idx_events_session_created` | events | `session_id, created_at` | A session's newest event / time order (watchdog every 15 s, every lane payload) |
| `idx_events_tool` | events | `tool_name` | Per-tool counts (analytics, workflows) and the events filter's tool list without a scan of every event row |
| `idx_events_session` | events | `session_id, id` | A session's events in insert order: every hook's duplicate guard reads the latest one. Not a prefix of `idx_events_session_type` (that one is not id-ordered); dropping it once made every hook sort its whole session |
| `idx_events_session` | events | `session_id, id` | A session's events in insert order: every hook's duplicate guard reads the latest one. Not a prefix of `idx_events_session_type_tool` (that one is not id-ordered); dropping it once made every hook sort its whole session |
| `idx_events_type` | events | `event_type` | Filter events by type |
| `idx_events_created` | events | `created_at DESC` | Activity feed ordering |
| `idx_events_session_type` | events | `session_id, event_type` | Per-session event-type filters |
| `idx_events_session_type_tool` | events | `session_id, event_type, tool_name, created_at` | Per-session event-type filters, and a covering read for session stats (per-tool and per-type counts), the hook's TurnDuration/APIError dedupe and the import's newest-per-type resume point (1.2 s → 20 ms on a 29K-event session). Replaces `idx_events_session_type` (dropped at startup), so a hook writes no more index entries than before |
| `idx_events_agent_type` | events | `agent_id, event_type` | `importSubagentFromJsonl` reads the agent's existing `tool_use_id`s once through it (it used to run a `data LIKE` per tool event) |
| `idx_sessions_status` | sessions | `status` | Status filter on Sessions page and Kanban Sessions view |
| `idx_sessions_started` | sessions | `started_at DESC` | Default sort order |
@@ -966,6 +966,10 @@ All messages are JSON with this envelope:
}
```
Frames a hook produces are sent only after its transaction committed and its
pane binding (`bindRunSession`) ran — in order, in the same tick — so no frame
describes a rolled-back row. `new_event` carries the event's row `id` (additive).
The `remote_source.status` message (emitted by the **Remote Data Sources**
sync poller and the `/api/remote-sources` routes) carries
`{ id, status, error?, last_sync_at? }`, where `status` is one of