# Analytics parity with the upstream agent monitor (hoangsonww) Baseline compared: upstream `hoangsonww's upstream agent monitor` at `523170e` (= tag v2.2.6, 2026-10-04) plus its open PRs, against AgentHub `main` at `151dc8a`. Agreed with the user on 2026-10-05: phases 1–6 in full, plus Cursor CLI, a small MCP server, a statusline script and Grafana boards. NOT doing: the deep workflow charts and the web Kanban (dropped on purpose in `4a0f995`), SSE (the websocket covers it), extra UI locales. Each phase ships on its own, with tests, docs (`update-project-docs`) and the CLAUDE.md marker. Nothing is committed without the user's ask. ## Phase 1 — data hygiene ✅ - `events.data` trimmed at every insert through the SQL function `trim_event_data(?)` (`server/lib/event-payload.js`, registered in `server/db.js`): Edit/MultiEdit/NotebookEdit/Write drop `originalFile`, `oldString`/`newString` (Write also `content`); Read drops `file.content` and `file.base64`; every string in `tool_input`/`tool_response` over 8 KB keeps a 6 KB head + 2 KB tail; `structuredPatch` and everything outside those two fields stay whole; `_trimmed` names what was dropped. The full text stays in the transcript and its snapshot under `/transcripts/`. - One-time sweep of stored rows (`sweepEventPayloads`, `user_version` 4, after marker 3), 200 rows per write transaction. No VACUUM (user's choice). Measured on the user's DB: 18,518 rows, 595 MB → 86 MB. - The session list prices every column `/api/pricing/cost/:id` does (`tokensForSessions` in `server/routes/sessions.js`). ## Phase 2 — cost by tool ✅ - `?tool=` on `/api/pricing/cost`, `/api/analytics`, `/api/stats`, `/api/workflows` (+ `/skill-usage`): `parseTool`/`parseScope` in `server/lib/source-filter.js` (id-shaped, else 400), `scoped-stats.js` now takes a `{where, params}` predicate over `sessions`. - `by_tool` in the cost response; Analytics gets an AI-tool select (applies to every sub-tab, Workflow included) and a **Cost by AI tool** card. - `/api/analytics`'s `total_cost` now equals `/api/pricing/cost` (it had ignored the compaction baselines and used UTC start dates). - Codex service tier read from `thread_settings_applied`: fast/priority → `speed = fast` priced with LiteLLM's `*_priority` rates (synced into the fast columns for non-Claude models), flex → `service_tier = flex` at half price. - Prometheus: `agenthub_cost_usd_total{tool,model}`. - DEFERRED (with the reason in `pricing-sync.js`): long-context (>200k/>272k) rates. No current Claude model has one, Codex here runs a 258k window (0 of 17,701 turns over 272k), so a `token_usage` primary-key rebuild would change no number today. ## Phase 3 — budgets, burn rate, forecast ✅ (budgets as alert rules) - `budgets` table: name, period (day/week/month), limit USD, scope (all / tool / workspace), thresholds (%), enabled. `/api/budgets` CRUD + `GET /api/budgets/status` (spend this period, %, projected end-of-period). - Evaluated on the alert beat; a crossed threshold fires once per period through the existing alerts feed → webhooks → push. - Alert rule type `cost_threshold` (USD per session / per day). - `POST /api/alerts/rules/preview`: dry-run a rule over recorded history. - UI: Settings → Budgets page; burn rate + month-end forecast tiles on Analytics → Cost; budget chip in the status bar when ≥ a threshold. - **As built:** a budget is an alert rule of type `budget` (`server/lib/budgets.js`; config `period` / `limit_usd` / `thresholds` / `tool` / `lane` / `tz_offset`), so there is no `budgets` table, no `/api/budgets` CRUD and no Settings → Budgets page: budgets are created and edited with the other rules in Settings → Notifications → Alert rules (editable in place since v2.3), and `GET /api/budgets/status` is the one budgets route. `cost_threshold`, the preview, the Spend pace + Budgets cards and the status-bar chip are as planned. ## Phase 4 — analytics UX ✅ - **Date range.** `parseRange` (`server/lib/source-filter.js`): `from`/`to` are local `YYYY-MM-DD`, inclusive, read with `tz_offset`, either end open; a bad date (or year 9999, or `from` > `to`) is `400 INVALID_INPUT`. `GET /api/pricing/cost` keeps only the cost rows of those spend days for `total_cost` / `breakdown` / `by_tool` (the existing spend-day split); `daily_costs` stays the whole series so the 30-day charts and the spend pace keep their windows. `GET /api/analytics` counts sessions/agents by `started_at` and events by `created_at` (an optional `range` on the `scoped-stats.js` functions), and takes tokens and cost from the same cost rows (`token_usage` has no date); `daily_events`/`daily_sessions` keep 365 days. `GET /api/workflows` (+ `/skill-usage`): sessions started in the range. The bodies are now `computeAnalytics` / `computeCost` (exported). Client: a range select on the Analytics toolbar beside the AI-tool select — 7d / 30d / 90d / All (default) / Custom with two `` — in `?range=` (+ `from`/`to`), `client/src/lib/dateRange.ts`; it also reaches the Workflow sub-tab. - **Query Explorer.** `GET /api/query` (`server/routes/query.js`): entity sessions / events / agents, `q` (literal substring), `tool`, `status`, `from`/`to`, `cwd` (folder subtree), `sources`; `sort`/`dir` from a per-entity allowlist (anything else 400 — no request text reaches SQL), ties by id; pages of ≤ 500; `format=csv|json` exports the whole filter capped at 50,000 rows (`X-Total-Count`, `X-Truncated`). CSV: RFC 4180, CRLF, BOM, `'` before a cell starting with = + - @ tab CR. Page `/query` (`client/src/pages/QueryExplorer.tsx`), sidebar entry **Query Explorer** (the sidebar has no analytics item of its own; it sits under Sessions). Column headers show the raw field names, which are also the CSV header. - **Scheduled reports.** Tables `report_definitions` / `report_runs`; `server/lib/reports.js` + `/api/reports` (CRUD, `POST /:id/run`, runs, `GET /:id/runs/:runId/:format` with the path built from ids + the stored stamp). A run covers the `window_days` full local days BEFORE the run day, computed with `computeAnalytics` + `computeCost`, plus the current budget status; files `/reports//.{html,json}`, HTML fully escaped. Scheduler: unref'd interval in `startBackgroundServices` (`DASHBOARD_REPORTS_MS`, default 60 s, 0 = off), each slot claimed by one conditional UPDATE (two dashboards on one DB run it once); a slot missed while down runs once on the next start; create/edit/re-enable claims the slot already past. Weekly reports have a `weekday` (default Monday) — needed for "weekly", not in the original list. Writes same-origin. Definitions AND runs survive Clear Data (like alert rules and snapshots). UI: Settings → **Reports** (`components/settings/ReportSettings.tsx`). - Not done: OpenAPI entries for the new routes (snapshots/budgets have none either); report runs are never pruned (one file per run per format). - Upstream's PR branches (#286/#172 `feat/query-explorer`, #174 `feat/scheduled-reports`) were not in the reference clone; built from the plan above. - Tests: `analytics-range`, `query-explorer`, `reports` in `server/__tests__/`; client `dateRange`, `QueryExplorer`, `ReportSettings`, and a `?range=` case in `screens.snapshot.test.tsx`. ## Phase 5 — sessions ✅ - Replay: a **Replay** tab on `/sessions/:id` beside Agents / Conversation / Timeline (`client/src/components/EventReplay.tsx`). It reads the session's events through `GET /api/events?session_id=` in pages of 500 (the newest 5,000 at most, with a note when there are more), oldest first, on the tab's first visit. A range slider over the event SEQUENCE (not the wall clock — an hour-long pause does not stall it), play/pause at 0.5×/1×/2×/5× (one event per 500 ms at 1×), step back/forward, jump start/end, keys ←/→/Home/End/Space, a mini-map of lifecycle markers (SessionStart, UserPromptSubmit, Compaction, SubagentStop, APIError, Stop) that jump the cursor, a state panel derived up to the cursor (status, tool in flight, prompts, errors, compactions, elapsed, tool-call counts) and the selected event through `EventDetail`. The pure derivations live in `client/src/lib/replay.ts` (unit-tested). - Snapshots: `POST /api/snapshots {session_id, redact?: {file_paths?}}` freezes `{session, agents, events, stats, cost}` into an immutable `session_snapshots` row (a trigger refuses UPDATE) under a 24-byte base64url token; `GET /api/snapshots` (no payload), `GET /api/snapshots/:token`, `DELETE /api/snapshots/:id`; writes behind `sameOriginGuard`. Redaction runs at capture: path-valued keys → `[redacted]`, the session cwd / home dir in any other string → `` / `~`. Snapshots survive Clear Data, like alert rules. UI: a **Snapshot** action in the session header (confirm with a "redact file paths" box, default on; copies the link), the read-only page `/snapshot/:token` (`client/src/pages/SnapshotView.tsx`: SessionOverview fed the frozen stats via its new `stats` prop, the agents, Replay and Timeline — the Timeline list now `client/src/components/EventTimeline.tsx`, shared with the analysis page), and Settings → **Snapshots** (`/settings/snapshots`, `components/settings/SnapshotSettings.tsx`: list, open, copy link, delete with confirm, the same on a right-click menu). - Upstream's PR branches (#265 `feat/session-replay`, #173 `feat/session-snapshots`) were not in the reference clone; built from the plan above. ## Phase 6 — ingest breadth ✅ - **Remote Codex.** A remote source has an opt-in `sync_codex` (+ optional `remote_codex_home`, default `~/.codex`, `wsl:~/.codex` beside a `wsl:` Claude home) — two additive columns on `remote_sources`, a checkbox in the source form, `--codex`/`--codex-home` on `agenthub remote-sources add`. A sync mirrors `sessions/` with mtimes kept (`scp -r -p`, or tar from WSL) plus `session_index.jsonl` into `/remote-sources//codex/`, then runs `ingestCodexUsage({home, source, activeMs})` — the same journal reader as local Codex (`server/lib/codex-usage.js`), so sessions arrive as `tool = 'codex'` rows tagged with the source id, with tokens and cost. A pass only ever writes rows of its own source; the local settle never touches a remote row; the remote activity window is `DASHBOARD_REMOTE_ACTIVE_WINDOW_MS`. With Codex on, a machine may have only one CLI: one half failing is `ok` + `last_error`. Differs from upstream (which always tries both): opt-in, and no separate Codex import pipeline — AgentHub's journal reader is reused. - **Push ingest.** `POST /api/hooks/ingest-batch` — body `{source, events: [{hook_type, data, tool?}]}`, each event through the SAME function as `POST /api/hooks/event` (`ingestHook` in `server/routes/hooks.js`). Off (503) until Settings → Data → **Push ingest** creates a token: 32 random bytes, shown once, stored only as its SHA-256 (`AGENTHUB_PUSH_TOKEN_SHA256` in the settings file), checked in constant time; rotate/revoke; `GET` never returns it. Sessions tagged `push:`; a pushed hook may only extend a session its own source created (`EFOREIGN`); `transcript_path`s are dropped (they name the sender's files); no lane/pane binding; 500 events / 1 MB. The UI shows a forwarder script keeping the token in a curl header file, not argv. Differs from upstream: upstream's wire format is a collector's own session/ token/turn records; AgentHub takes plain hook payloads, so a pushed session has events and live status but no token cost (that needs transcripts). - **Live Codex overlay — not built.** Upstream's `include_transient` (`codex-process-overlay.js`) scans `ps` for an interactive `codex` TUI that has not written a rollout yet, i.e. one started but never prompted. In AgentHub a Codex pane started from the dashboard already shows (its tab, and **New session** in the sidebar from its run), and anything that has done work has a journal the 15 s poll reads (or a SessionStart hook with Codex hooks on). The only row it would add is an idle Codex in an outside terminal with no prompt and no cost — not worth a per-tick process scan. - **Privacy** (`server/lib/privacy.js`, Settings → Data → **Privacy**, `GET/PUT /api/settings/privacy`, both off by default, persisted to the settings file, env-pinnable): (a) **Redact secrets** (`AGENTHUB_REDACT_SECRETS=1`) — chained in front of the trim in `trim_event_data` (so a key is redacted before a trim could cut it) plus a new `redact_text(?)` on every events `summary` insert; known key shapes (Anthropic, OpenAI, GitHub, GitLab, Slack, AWS, Google, npm, JWT), bearer/basic headers, PEM private keys, `password=`/`*_TOKEN:`-style values, and any string under a secret-named JSON key → `[redacted]`, with `_redacted: ` on the payload. New data only; the transcript is untouched; session NAMES (first prompt) are not redacted. (b) **Ignored folders** (`AGENTHUB_IGNORE_CWD`, `path.delimiter`-separated, path-boundary match): hooks dropped whole (`ignored: true`), and the history import and the Codex reader skip those sessions too. ## Phase 7 — extras chosen by the user - Cursor CLI: session + token ingest from `~/.cursor`, Cursor pricing rows. - MCP server (stdio, read-only, ~15 tools: stats, cost, sessions, session detail, analytics, workflows, budgets, alerts). - `statusline/` script for Claude Code (model, branch, context %, session $, today $ from the dashboard). - `monitoring/` Grafana dashboards + Prometheus rules for `/api/metrics`. ### Phase 7 status — MCP, statusline, monitoring ✅; Cursor ✅ (no cost) - **MCP** (`mcp/server.js`, `agenthub mcp`, docs/MCP.md): JSON-RPC over stdio written directly (no SDK), 15 GET-only tools (health, info, stats, total / session / lane cost, sessions, session, session stats, analytics, workflows, alerts, lanes, pricing, budget status). `AGENTHUB_URL`, else the CLI's discovery, now shared as `server/lib/dashboard-url.js`. A missing route — 404 or the production SPA page served with 200 — reads "Not available on this dashboard", which is what `get_budget_status` says until Phase 3 lands. NOT auto-registered by the plugin: `claude mcp add agenthub -- agenthub mcp`. - **Statusline** (`statusline/statusline.js`, `agenthub statusline install|uninstall [--force]`, docs/STATUSLINE.md): model · branch · context bar · session $ · today $ (`/api/pricing/cost`, 1 s timeout, 30 s cache in the OS temp dir, failures cached). Install refuses a foreign `statusLine` without `--force` and backs settings.json up first; a plugin install registers the `~/.local/bin/agenthub` launcher, not the cache dir. - **Monitoring** (`monitoring/`, `docker-compose.monitoring.yml`, docs/MONITORING.md): scrape config, `agenthub:*` recording rules, boards Overview / Sessions & agents / Tokens & cost; the spend panels read Phase 2's `agenthub_cost_usd_total{tool,model}` and say so. - Tests: `mcp-server`, `mcp-server-live`, `statusline`, `monitoring` in `server/__tests__/`. - **Cursor** (`server/lib/cursor-usage.js`): agent transcripts under `~/.cursor/projects//agent-transcripts/` become `tool = 'cursor'` sessions (`cursor_transcripts` table, `DASHBOARD_CURSOR_POLL_MS`, `AGENTHUB_CURSOR_DIR`). NOT built: tokens, model and cost — the transcripts record none, and `state.vscdb` (which has them) also holds auth tokens, so it is never read; hence no Cursor pricing rows either. ## Ported onto main Phases 2–7 were built in a separate worktree on 138e706 and ported onto v2.3.1 on 2026-10-06 (phase 1 had shipped on its own), released as v2.4.0.