perf: v2.5.10 — ready for 10+ concurrent users, leaner locally
Measured on a copy of a real 1.2 GB / 253K-event database, 10 simulated browsers + 10 hooks/s for 90 s: event-loop delay p99 1239 ms → 32 ms (max 2.0 s → 94 ms), 266 failed requests → 0, 74 → 111 requests/s served, server RSS 633 → 346 MB. Entry bundle 975 KB raw → 394 KB (106 KB brotli). Server - /api/analytics: the events-table scans run in the read worker (scoped-stats eventAggregates, read-pool job analyticsEvents); runRead takes maxAgeMs — analytics 10 s, /api/workflows 30 s — keyed by the args - /api/workflows/skill-usage: +event_type and a tool_name range instead of LIKE, so it uses idx_events_tool (784 ms → a few ms) - idx_events_session_type_tool (session_id, event_type, tool_name, created_at) replaces idx_events_session_type: covering reads for session stats, the hook dedupes and the import resume point (1.2 s → 20 ms); 64 MB cache_size and 256 MB mmap_size on both connections - WAL: the server sets wal_autocheckpoint = 0 and checkpoints PASSIVE every DASHBOARD_WAL_CHECKPOINT_MS (5 s) from the read worker's own writable connection; a failed checkpoint restores the automatic one (hook COMMIT p95 109 ms → 15 ms at 100 hooks/s) - tmux: one list-sessions answers a whole GET /api/lanes and the heal beat (liveSessions), shared for 1 s and cleared by this process's new/kill — GET /api/run's sync spawns had blocked 60 s of a 90 s run; the heal beat skips tmux when nothing runs; attach options chained into 2 spawns - hooks: TurnDuration/APIError deduped against one read per hook instead of a query per transcript entry (23 ms/hook on a 763-turn session); one lane scan per hook; hourly upserts skip rows that did not grow; frames are sent after the transaction commits (and after the pane binding); new_event carries its row id; a split pane's (slot 1–9) hooks no longer rebind the lane's session_id - watchdog liveness probe runs ps asynchronously; transcript newline scan via indexOf; startup subagent backfill fingerprints the files it already checked - pty attach: backpressure (pause past 1 MB unsent, resume under 256 KB) - static: hashed assets served from their pre-compressed .br/.gz sibling Client - hidden tabs neither refetch on WS events nor poll (useLiveRefresh, panes, lane cards, git header, changes panel, settings); no per-tab hourly /updates/check (the server already checks) - refetch storms throttled (workspace tree/session list 3 s, /api/run only on a run-key change, notification pane lookup only when shown); WS reconnect jitter; terminal retry backoff 1.5 → 30 s, stops on 1008 - heavier throttles for aggregates (15 s) and detail views; Monitor keepIfSame; GET in-flight dedupe in request() - lazy Dashboard/History, xterm (with the first terminal), lane modals, Sagi (only while enabled); only the active locale loads; Workspace chunk prefetched on a /workspace deep link - Sagi: no work while disabled, pupil via CSS variables, memo avatar - build: .br/.gz siblings, sourcemap hidden; service worker no longer caches /assets/ (cache v3 drops the old one) Tests: scripts/test-server.js drops the GIT_DIR-family variables a hook exports, so a suite run from a linked worktree's pre-commit can no longer commit into the real repository or set core.bare. Docs: ARCHITECTURE, DATABASE, API, DEPLOYMENT, client README, docs/plans/perf/README.md (audit, checklist, results).
This commit is contained in:
+26
-28
@@ -151,6 +151,11 @@ db.pragma("synchronous = NORMAL");
|
||||
// (an import, a cleanup) left a WAL of hundreds of MB that never shrank while
|
||||
// any dashboard kept the database open. See also `maintainDatabase`.
|
||||
db.pragma("journal_size_limit = 67108864");
|
||||
// A 64 MB page cache (default 2 MB) and memory-mapped reads: the aggregates
|
||||
// re-read the same index pages, 5–45 % faster on a large DB. Both are per
|
||||
// connection and cost address space, not a fixed allocation.
|
||||
db.pragma("cache_size = -65536");
|
||||
db.pragma("mmap_size = 268435456");
|
||||
// Every event insert stores its payload through `trim_event_data(?)`
|
||||
// (`server/lib/event-payload.js`): whole-file mirrors and long output stay in
|
||||
// the transcript, not in `events.data`.
|
||||
@@ -322,7 +327,7 @@ db.exec(`
|
||||
-- read walked the rows' overflow chains (created_at sits after data).
|
||||
CREATE INDEX IF NOT EXISTS idx_events_session_created ON events(session_id, created_at);
|
||||
-- (session_id, id): a session's events in insert order, NOT a prefix of
|
||||
-- idx_events_session_type (that one is not id-ordered). Every hook's
|
||||
-- idx_events_session_type_tool (that one is not id-ordered). Every hook's
|
||||
-- duplicate guard reads a session's latest event through it; dropped, that
|
||||
-- read sorted the whole session per hook.
|
||||
CREATE INDEX IF NOT EXISTS idx_events_session ON events(session_id, id);
|
||||
@@ -335,7 +340,14 @@ db.exec(`
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_started ON sessions(started_at DESC);
|
||||
|
||||
-- Composite indexes for frequent query patterns (columns that exist at table creation time)
|
||||
CREATE INDEX IF NOT EXISTS idx_events_session_type ON events(session_id, event_type);
|
||||
-- (session_id, event_type, tool_name, created_at) covers a session's per-type,
|
||||
-- per-tool and newest-per-type reads (session stats, the import's resume
|
||||
-- point) without touching the rows: 1.2 s → 20 ms on a 29K-event session. It
|
||||
-- replaces the (session_id, event_type) index it extends, so a hook still
|
||||
-- writes the same number of index entries.
|
||||
CREATE INDEX IF NOT EXISTS idx_events_session_type_tool
|
||||
ON events(session_id, event_type, tool_name, created_at);
|
||||
DROP INDEX IF EXISTS idx_events_session_type;
|
||||
-- Subagent JSONL import dedups each tool event with
|
||||
-- "WHERE agent_id = ? AND event_type = ? AND data LIKE '%tool_use_id%'".
|
||||
-- Without an agent_id index that is a full events-table scan per tool event;
|
||||
@@ -1963,6 +1975,16 @@ const stmts = {
|
||||
web_search_requests = MAX(web_search_requests, excluded.web_search_requests),
|
||||
web_fetch_requests = MAX(web_fetch_requests, excluded.web_fetch_requests),
|
||||
code_execution_requests = MAX(code_execution_requests, excluded.code_execution_requests)
|
||||
-- Every hook re-sends every hour of the transcript; most rows did not grow.
|
||||
-- Skipping those leaves the values identical (every SET is a MAX) and
|
||||
-- writes no page, which keeps the WAL — and its checkpoints — small.
|
||||
WHERE excluded.input_tokens > input_tokens OR excluded.output_tokens > output_tokens
|
||||
OR excluded.cache_read_tokens > cache_read_tokens
|
||||
OR excluded.cache_write_tokens > cache_write_tokens
|
||||
OR excluded.cache_write_1h_tokens > cache_write_1h_tokens
|
||||
OR excluded.web_search_requests > web_search_requests
|
||||
OR excluded.web_fetch_requests > web_fetch_requests
|
||||
OR excluded.code_execution_requests > code_execution_requests
|
||||
`),
|
||||
getTokenTotals: db.prepare(`
|
||||
SELECT
|
||||
@@ -2035,22 +2057,8 @@ const stmts = {
|
||||
matchPricing: db.prepare(
|
||||
"SELECT * FROM model_pricing WHERE ? LIKE model_pattern ORDER BY LENGTH(model_pattern) DESC LIMIT 1"
|
||||
),
|
||||
toolUsageCounts: db.prepare(`
|
||||
SELECT tool_name, COUNT(*) as count
|
||||
FROM events
|
||||
WHERE tool_name IS NOT NULL
|
||||
GROUP BY tool_name
|
||||
ORDER BY count DESC
|
||||
LIMIT 20
|
||||
`),
|
||||
// Accept a timezone modifier (e.g. '-420 minutes') so GROUP BY uses local dates
|
||||
dailyEventCounts: db.prepare(`
|
||||
SELECT DATE(created_at, ?) as date, COUNT(*) as count
|
||||
FROM events
|
||||
WHERE created_at >= DATE('now', '-365 days')
|
||||
GROUP BY 1
|
||||
ORDER BY date ASC
|
||||
`),
|
||||
// Accept a timezone modifier (e.g. '-420 minutes') so GROUP BY uses local dates.
|
||||
// The events-table analytics live in scoped-stats.js's eventAggregates (read worker).
|
||||
dailySessionCounts: db.prepare(`
|
||||
SELECT DATE(started_at, ?) as date, COUNT(*) as count
|
||||
FROM sessions
|
||||
@@ -2066,16 +2074,6 @@ const stmts = {
|
||||
ORDER BY count DESC
|
||||
`),
|
||||
totalSubagentCount: db.prepare("SELECT COUNT(*) as count FROM agents WHERE type = 'subagent'"),
|
||||
eventTypeCounts: db.prepare(`
|
||||
SELECT event_type, COUNT(*) as count
|
||||
FROM events
|
||||
GROUP BY event_type
|
||||
ORDER BY count DESC
|
||||
`),
|
||||
avgEventsPerSession: db.prepare(`
|
||||
SELECT ROUND(CAST(COUNT(*) AS REAL) / MAX(1, (SELECT COUNT(*) FROM sessions)), 1) as avg
|
||||
FROM events
|
||||
`),
|
||||
|
||||
// Per-session aggregations powering the SessionOverview panel.
|
||||
sessionEventCount: db.prepare("SELECT COUNT(*) as count FROM events WHERE session_id = ?"),
|
||||
|
||||
Reference in New Issue
Block a user