perf: v2.5.10 — ready for 10+ concurrent users, leaner locally
Measured on a copy of a real 1.2 GB / 253K-event database, 10 simulated browsers + 10 hooks/s for 90 s: event-loop delay p99 1239 ms → 32 ms (max 2.0 s → 94 ms), 266 failed requests → 0, 74 → 111 requests/s served, server RSS 633 → 346 MB. Entry bundle 975 KB raw → 394 KB (106 KB brotli). Server - /api/analytics: the events-table scans run in the read worker (scoped-stats eventAggregates, read-pool job analyticsEvents); runRead takes maxAgeMs — analytics 10 s, /api/workflows 30 s — keyed by the args - /api/workflows/skill-usage: +event_type and a tool_name range instead of LIKE, so it uses idx_events_tool (784 ms → a few ms) - idx_events_session_type_tool (session_id, event_type, tool_name, created_at) replaces idx_events_session_type: covering reads for session stats, the hook dedupes and the import resume point (1.2 s → 20 ms); 64 MB cache_size and 256 MB mmap_size on both connections - WAL: the server sets wal_autocheckpoint = 0 and checkpoints PASSIVE every DASHBOARD_WAL_CHECKPOINT_MS (5 s) from the read worker's own writable connection; a failed checkpoint restores the automatic one (hook COMMIT p95 109 ms → 15 ms at 100 hooks/s) - tmux: one list-sessions answers a whole GET /api/lanes and the heal beat (liveSessions), shared for 1 s and cleared by this process's new/kill — GET /api/run's sync spawns had blocked 60 s of a 90 s run; the heal beat skips tmux when nothing runs; attach options chained into 2 spawns - hooks: TurnDuration/APIError deduped against one read per hook instead of a query per transcript entry (23 ms/hook on a 763-turn session); one lane scan per hook; hourly upserts skip rows that did not grow; frames are sent after the transaction commits (and after the pane binding); new_event carries its row id; a split pane's (slot 1–9) hooks no longer rebind the lane's session_id - watchdog liveness probe runs ps asynchronously; transcript newline scan via indexOf; startup subagent backfill fingerprints the files it already checked - pty attach: backpressure (pause past 1 MB unsent, resume under 256 KB) - static: hashed assets served from their pre-compressed .br/.gz sibling Client - hidden tabs neither refetch on WS events nor poll (useLiveRefresh, panes, lane cards, git header, changes panel, settings); no per-tab hourly /updates/check (the server already checks) - refetch storms throttled (workspace tree/session list 3 s, /api/run only on a run-key change, notification pane lookup only when shown); WS reconnect jitter; terminal retry backoff 1.5 → 30 s, stops on 1008 - heavier throttles for aggregates (15 s) and detail views; Monitor keepIfSame; GET in-flight dedupe in request() - lazy Dashboard/History, xterm (with the first terminal), lane modals, Sagi (only while enabled); only the active locale loads; Workspace chunk prefetched on a /workspace deep link - Sagi: no work while disabled, pupil via CSS variables, memo avatar - build: .br/.gz siblings, sourcemap hidden; service worker no longer caches /assets/ (cache v3 drops the old one) Tests: scripts/test-server.js drops the GIT_DIR-family variables a hook exports, so a suite run from a linked worktree's pre-commit can no longer commit into the real repository or set core.bare. Docs: ARCHITECTURE, DATABASE, API, DEPLOYMENT, client README, docs/plans/perf/README.md (audit, checklist, results).
This commit is contained in:
+4
-2
@@ -1152,7 +1152,7 @@ head of `toolFlow.transitions` (one window pass over the events for both).
|
||||
The bundle scans every event, so it is computed in a worker thread on its own
|
||||
read-only connection (`server/lib/read-pool.js`) and never stalls another
|
||||
request, hook or websocket meanwhile. Identical requests in flight share one
|
||||
run. `AGENTHUB_READ_WORKER=0` computes it inline instead.
|
||||
run, and a result is reused for 30 s per distinct query. `AGENTHUB_READ_WORKER=0` computes it inline instead.
|
||||
|
||||
**Status codes:**
|
||||
- **200** — the bundle, or the requested sections of it
|
||||
@@ -1721,7 +1721,7 @@ sequenceDiagram
|
||||
GET /api/events
|
||||
```
|
||||
|
||||
Returns `{ events, limit, offset, total }`, newest first, filtered by `event_type`, `tool_name`, `agent_id`, `session_id` (each comma-separated), `q`, `from`/`to` (on `created_at`) and `sources`. Each row carries its labels, so a list page need not fetch every session and agent to name them: `session_name`, `session_cwd`, `session_lane_id`, `agent_name` and `agent_subagent_type` (each `null` when the session/agent row is missing). These columns are REST-only — the WebSocket `new_event` message is unchanged.
|
||||
Returns `{ events, limit, offset, total }`, newest first, filtered by `event_type`, `tool_name`, `agent_id`, `session_id` (each comma-separated), `q`, `from`/`to` (on `created_at`) and `sources`. Each row carries its labels, so a list page need not fetch every session and agent to name them: `session_name`, `session_cwd`, `session_lane_id`, `agent_name` and `agent_subagent_type` (each `null` when the session/agent row is missing). These columns are REST-only — the WebSocket `new_event` message does not carry them (it carries the row `id`).
|
||||
|
||||
---
|
||||
|
||||
@@ -2048,6 +2048,8 @@ GET /api/analytics?tz_offset=<minutes>&sources=<csv>&tool=<id>&from=<YYYY-MM-DD>
|
||||
|
||||
The Analytics bundle (`tokens`, `total_cost`, `tool_usage`, `daily_events`, `daily_sessions`, `agent_types`, `event_types`, `avg_events_per_session`, `total_subagents`, `overview`, `agents_by_status`, `sessions_by_status`). With `from`/`to` (same rules as above) the shape is unchanged and the numbers narrow: sessions and agents count by `started_at`, events by `created_at`, and `tokens` / `total_cost` come from the cost rows of the days in the range (`token_usage` has no date, so tokens follow the spend-day split). `daily_events` and `daily_sessions` keep their 365-day window.
|
||||
|
||||
The four numbers that scan `events` (`tool_usage`, `daily_events`, `event_types`, `avg_events_per_session`) are computed in the read worker (`server/lib/read-pool.js`) and reused for 10 s per distinct query (scope, range, `tz_offset`), so they may trail a new event by up to that long. A worker job past `AGENTHUB_READ_TIMEOUT_MS` answers **503** `EREADTIMEOUT`.
|
||||
|
||||
#### Spend Budgets
|
||||
|
||||
A budget is an alert rule of type `budget`, created and edited through `/api/alerts/rules`:
|
||||
|
||||
+21
-1
@@ -677,7 +677,7 @@ CREATE INDEX idx_agents_status ON agents(status);
|
||||
-- tens of seconds to sub-second.
|
||||
CREATE INDEX idx_events_agent_type ON events(agent_id, event_type);
|
||||
|
||||
-- A session's events in insert order. NOT a prefix of idx_events_session_type
|
||||
-- A session's events in insert order. NOT a prefix of idx_events_session_type_tool
|
||||
-- (not id-ordered): every hook's duplicate guard reads a session's latest event
|
||||
-- through it. Dropping it (ad6e801) made that read sort the whole session per
|
||||
-- hook and made a correlated "next tool event" query quadratic — minutes on a
|
||||
@@ -687,6 +687,14 @@ CREATE INDEX idx_events_session ON events(session_id, id);
|
||||
-- Per-tool counts (analytics, workflows) and the events filter's tool list
|
||||
-- read this alone instead of every event row (~300 ms -> ~20 ms at 233k rows).
|
||||
CREATE INDEX idx_events_tool ON events(tool_name);
|
||||
|
||||
-- A session's events by type and tool, covering: session stats (per-tool and
|
||||
-- per-type counts), the hook's TurnDuration/APIError dedupe (one read per hook)
|
||||
-- and the import's newest-per-type resume point read no event row
|
||||
-- (1.2 s -> 20 ms on a 29K-event session). Replaces idx_events_session_type
|
||||
-- (session_id, event_type), dropped at startup, so a hook writes no more index
|
||||
-- entries than before.
|
||||
CREATE INDEX idx_events_session_type_tool ON events(session_id, event_type, tool_name, created_at);
|
||||
```
|
||||
|
||||
### tool_executions Indexes
|
||||
@@ -872,8 +880,20 @@ db.pragma("foreign_keys = ON");
|
||||
db.pragma("busy_timeout = 5000"); // two dashboards may share the file
|
||||
db.pragma("synchronous = NORMAL"); // safe with WAL: a power cut loses only the last commits, never the file
|
||||
db.pragma("journal_size_limit = 67108864"); // a checkpoint truncates the WAL back to 64 MB
|
||||
db.pragma("cache_size = -65536"); // 64 MB page cache (default 2 MB)
|
||||
db.pragma("mmap_size = 268435456"); // memory-mapped reads, 256 MB
|
||||
```
|
||||
|
||||
The read worker (`server/lib/read-pool.js`) sets the same `cache_size`/`mmap_size`.
|
||||
|
||||
**Checkpoints.** The running server sets `wal_autocheckpoint = 0` on its main
|
||||
connection and checkpoints every `DASHBOARD_WAL_CHECKPOINT_MS` (default 5000, `0` =
|
||||
SQLite's automatic checkpoint as before) with `wal_checkpoint(PASSIVE)` from the read
|
||||
worker's own writable connection. SQLite's automatic checkpoint runs inside whichever
|
||||
`COMMIT` crosses 1000 WAL pages — a hook's, on the event loop — and measured up to 2.7 s
|
||||
per commit under load. A failed checkpoint switches the automatic one back on. A second
|
||||
process on the same file (another dashboard, the CLI) keeps its own setting.
|
||||
|
||||
**Upkeep.** `maintainDatabase()` (exported by `server/db.js`) runs hourly from `server/index.js` (`DASHBOARD_DB_MAINTAIN_MS`, `0` = off): a `wal_checkpoint(PASSIVE)` — it never waits on a reader, such as the other dashboard — after which the next write shrinks the WAL to `journal_size_limit`, and `PRAGMA optimize`. Nothing runs `VACUUM`: it rewrites the whole file under an exclusive lock, so space freed by a cleanup is reused for new rows instead of returned to the disk.
|
||||
|
||||
**Growth.** `events.data` is the hook payload TRIMMED by `trim_event_data(?)` (`server/lib/event-payload.js`, a SQL function registered on the connection in `server/db.js`, used by every event `INSERT` and by the duplicate guard): Edit/MultiEdit/NotebookEdit/Write drop `tool_response.originalFile`/`oldString`/`newString` (Write also `content`), Read drops `tool_response.file.content`/`base64`, and any string inside `tool_input`/`tool_response` over 8 KB keeps its first 6 KB and last 2 KB; `structuredPatch` and every field outside those two are kept, and `_trimmed` lists what was dropped. The full text stays in the session transcript and AgentHub's snapshot of it under `<data dir>/transcripts/`, which the Conversation tab reads. Rows stored before this are trimmed once in the background (`sweepEventPayloads`, 200 rows per transaction, `user_version` 4 after marker 3); measured on a 1.2 GB install it took `events.data` from 913 MB to ~400 MB. Freed pages are reused, not returned to the disk (no `VACUUM`). No ROW is pruned automatically. The only deletions are the user's: Settings → clear data / cleanup, a workspace's purge, `scripts/clear-data.js`. Clear data runs in ONE transaction (foreign keys off around it, so workflows are deleted explicitly), cleanup deletes in 500-id chunks inside one transaction. `GET /api/settings/export` streams `events` in keyset pages of 500 instead of loading the table.
|
||||
|
||||
+18
-9
@@ -263,8 +263,9 @@ AGENTHUB_CURSOR_DIR= # where Cursor keeps its data (default ~
|
||||
|
||||
# Pricing and saved settings
|
||||
AGENTHUB_PRICING_SYNC= # 0|false|off disables automatic pricing from LiteLLM (default on)
|
||||
AGENTHUB_READ_WORKER= # 0 runs heavy aggregates (GET /api/workflows) inline instead of in the read worker (default worker)
|
||||
AGENTHUB_READ_WORKER= # 0 runs heavy aggregates (GET /api/workflows, /api/analytics' event scans) inline instead of in the read worker (default worker)
|
||||
AGENTHUB_READ_TIMEOUT_MS=60000 # a read-worker job past this answers 503 EREADTIMEOUT and gets a fresh worker
|
||||
DASHBOARD_WAL_CHECKPOINT_MS=5000 # WAL checkpoint from the read worker instead of inside a hook's COMMIT (0 = SQLite's automatic one)
|
||||
AGENTHUB_SETTINGS_FILE= # where dashboard-saved settings persist (default ~/.config/agenthub/settings.env);
|
||||
# load order: real env > the checkout's .env > this file
|
||||
|
||||
@@ -726,14 +727,22 @@ node --optimize-for-size server/index.js
|
||||
|
||||
### SQLite Tuning
|
||||
|
||||
```javascript
|
||||
// server/db.js - Add these pragmas
|
||||
db.pragma('journal_mode = WAL'); // Write-Ahead Logging
|
||||
db.pragma('synchronous = NORMAL'); // Faster writes
|
||||
db.pragma('cache_size = -64000'); // 64MB cache
|
||||
db.pragma('temp_store = MEMORY'); // Temp tables in memory
|
||||
db.pragma('mmap_size = 30000000000'); // Memory-mapped I/O
|
||||
db.pragma('page_size = 4096'); // Optimal page size
|
||||
`server/db.js` already sets WAL, `synchronous = NORMAL`, a 64 MB `cache_size` and a
|
||||
256 MB `mmap_size`, and the server checkpoints the WAL from its read worker
|
||||
(`DASHBOARD_WAL_CHECKPOINT_MS`) instead of inside a hook's `COMMIT` — see
|
||||
[DATABASE.md § SQLite Pragmas](DATABASE.md#sqlite-pragmas). Nothing to add by hand.
|
||||
|
||||
### Compression
|
||||
|
||||
The client build writes a `.br` and a `.gz` beside every hashed asset, and the
|
||||
server sends the one the browser accepts (`Content-Encoding`, `Vary: Accept-Encoding`)
|
||||
— a first Workspace load is ~75 % smaller with no compression CPU at request time.
|
||||
API responses are not compressed by AgentHub; behind nginx, add:
|
||||
|
||||
```nginx
|
||||
gzip on;
|
||||
gzip_types application/json;
|
||||
gzip_min_length 1024;
|
||||
```
|
||||
|
||||
### Nginx Tuning
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# Performance pass — 10+ concurrent users, lean local mode
|
||||
|
||||
Audit 2026-10-07 against a copy of the real DB (1.2 GB, 253K events). Baseline,
|
||||
10 simulated browsers + 10 hooks/s for 90 s: event-loop delay p99 1239 ms, request
|
||||
p95 ~10 s, 266 failed. Bench tools: `/tmp/ahperf/tools/` (`clients.js`, `hookload.js`,
|
||||
`bench.js`). Target: ELD p99 < 100 ms, request p95 < 300 ms, 0 failed.
|
||||
|
||||
Dev (4830) and prod (4820) keep sharing one `dashboard.db`.
|
||||
|
||||
## Server
|
||||
- [x] S1 `/api/workflows/skill-usage`: `+event_type` + range instead of LIKE (784 ms → ~5 ms)
|
||||
- [x] S2 `/api/analytics` into `read-pool.js` JOBS + short memo keyed by scope/args
|
||||
- [x] S3 `/api/workflows` result memo (30 s)
|
||||
- [x] S4 `idx_events_session_type_tool` replaces `idx_events_session_type` (partial index skipped: the 30 s workflows memo covers it); `cache_size`/`mmap_size`
|
||||
- [x] S5 WAL: `wal_autocheckpoint=0` on the main connection, PASSIVE checkpoint off-thread, fallback
|
||||
- [x] S6 tmux: 1 s `list-sessions` memo (cleared by this process's new/kill), one `liveSessions()` per `GET /api/lanes`, attach options chained (4 spawns → 2), async `ps` for the watchdog, heal tick skipped when nothing runs
|
||||
- [x] S7 hooks: hoist TurnDuration/APIError statements + watermark; one `listLanes()` per hook; hourly upsert only on change
|
||||
- [x] S8 hooks.js `session_id` rebind skipped for slot 1–9 panes
|
||||
- [x] S9 startup subagent backfill: do not re-parse unstampable files every boot
|
||||
- [x] S10 pty-attach backpressure
|
||||
- [x] S11 serve pre-compressed `.br`/`.gz` assets
|
||||
- [x] S12 broadcasts only after the hook transaction commits
|
||||
- [x] S13 `new_event` carries `id`
|
||||
- [~] S14 codex-usage memo — DROPPED: an in-memory "already read" map survives Clear data and blocks the re-import (caught by codex-usage tests); the pass costs ~5 ms per 15 s
|
||||
- [x] S15 transcript newline scan via `indexOf`
|
||||
|
||||
## Client
|
||||
- [x] C1 no WS-triggered refetch in hidden tabs; hidden-tab pollers paused; drop hourly `/updates/check`
|
||||
- [x] C2 throttle WorkspaceTree / WorkspaceSessionList / `/api/run` refetchers; lazy notification lookup
|
||||
- [x] C3 WS reconnect jitter; terminal retry backoff
|
||||
- [x] C4 heavier throttles for aggregates / detail refetches; Monitor `keepIfSame`; GET in-flight dedupe
|
||||
- [x] C5 build: pre-compress assets, `sourcemap: "hidden"`
|
||||
- [x] C6 load only the active locale; lazy routes/Sagi/modals/xterm
|
||||
- [x] C7 Sagi re-render fixes; service-worker asset cache
|
||||
|
||||
## Verify
|
||||
- [x] `npm run test:server` (2556/2556), `npm run test:client` (1084/1084), `tsc -b`, header check; headless Chrome smoke (en + vi, 4 pages, terminal attach)
|
||||
- [x] re-run the 10-client bench (server side, same old-client simulation) — see Result
|
||||
|
||||
## Result (server, `clients.js 10 90 10`, same simulated old client)
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| event-loop delay p99 / max | 1239 ms / 2.0 s | 31.8 ms / 94 ms |
|
||||
| failed requests | 266 | 0 |
|
||||
| requests served | 74/s | 111/s |
|
||||
| hook p95 / p99 (10/s) | — | 7.6 / 28.8 ms |
|
||||
| RSS | 633 MB | 346 MB |
|
||||
|
||||
Biggest single cause after the first round: 2,247 synchronous `tmux list-sessions`
|
||||
from `GET /api/run` = 60 s of a 90 s run blocked (fork of a 600 MB process); the 1 s
|
||||
memo removed it. The client fixes (C1–C4) cut the request rate itself on top.
|
||||
Reference in New Issue
Block a user