feat: analytics parity — tool cost, date ranges, budgets, query explorer, replay, snapshots (v2.4.0)
Analytics parity phases 2–7 (docs/plans/analytics-parity/), built on
138e706 and ported onto v2.3.1:
- Cost by AI tool (?tool= on cost/analytics/stats/workflows, by_tool,
Codex fast/priority and flex tiers, agenthub_cost_usd_total{tool,model})
- Date ranges (from/to) on cost/analytics/workflows; Analytics range select
with Today / This week (Monday) / This month (1st) / 7d / 30d / 90d / All /
Custom; workflow filters carried into workflow-stats.js (read worker)
- Budgets and cost alerts as alert rule types (alert_rules rebuilt without
its rule_type CHECK), GET /api/budgets/status, rule preview, spend pace,
status-bar budget chip; rules stay editable in place
- Query Explorer (/query, allowlisted sort, CSV/JSON export) and scheduled
reports (report_definitions/report_runs, Settings → Reports)
- Session Replay tab and immutable snapshots (session_snapshots, /snapshot/:token)
- Remote Codex sync, push ingest with a hashed token, privacy redaction and
ignored folders (Settings → Data), Cursor sessions (no cost)
- Read-only MCP server, statusline, Grafana/Prometheus files
Also: resizable sidebar, pet run/wave animations, the workspace home keeps a
half-typed prompt, and the pre-commit hook runs only the related tests for a
small commit.
This commit is contained in:
@@ -0,0 +1,234 @@
|
||||
# Analytics parity with the upstream agent monitor (hoangsonww)
|
||||
|
||||
Baseline compared: upstream `hoangsonww's upstream agent monitor` at
|
||||
`523170e` (= tag v2.2.6, 2026-10-04) plus its open PRs, against AgentHub `main`
|
||||
at `151dc8a`. Agreed with the user on 2026-10-05: phases 1–6 in full, plus
|
||||
Cursor CLI, a small MCP server, a statusline script and Grafana boards. NOT
|
||||
doing: the deep workflow charts and the web Kanban (dropped on purpose in
|
||||
`4a0f995`), SSE (the websocket covers it), extra UI locales.
|
||||
|
||||
Each phase ships on its own, with tests, docs (`update-project-docs`) and the
|
||||
CLAUDE.md marker. Nothing is committed without the user's ask.
|
||||
|
||||
## Phase 1 — data hygiene ✅
|
||||
- `events.data` trimmed at every insert through the SQL function
|
||||
`trim_event_data(?)` (`server/lib/event-payload.js`, registered in
|
||||
`server/db.js`): Edit/MultiEdit/NotebookEdit/Write drop `originalFile`,
|
||||
`oldString`/`newString` (Write also `content`); Read drops `file.content` and
|
||||
`file.base64`; every string in `tool_input`/`tool_response` over 8 KB keeps a
|
||||
6 KB head + 2 KB tail; `structuredPatch` and everything outside those two
|
||||
fields stay whole; `_trimmed` names what was dropped. The full text stays in
|
||||
the transcript and its snapshot under `<data dir>/transcripts/`.
|
||||
- One-time sweep of stored rows (`sweepEventPayloads`, `user_version` 4, after
|
||||
marker 3), 200 rows per write transaction. No VACUUM (user's choice).
|
||||
Measured on the user's DB: 18,518 rows, 595 MB → 86 MB.
|
||||
- The session list prices every column `/api/pricing/cost/:id` does
|
||||
(`tokensForSessions` in `server/routes/sessions.js`).
|
||||
|
||||
## Phase 2 — cost by tool ✅
|
||||
- `?tool=` on `/api/pricing/cost`, `/api/analytics`, `/api/stats`,
|
||||
`/api/workflows` (+ `/skill-usage`): `parseTool`/`parseScope` in
|
||||
`server/lib/source-filter.js` (id-shaped, else 400), `scoped-stats.js` now
|
||||
takes a `{where, params}` predicate over `sessions`.
|
||||
- `by_tool` in the cost response; Analytics gets an AI-tool select (applies to
|
||||
every sub-tab, Workflow included) and a **Cost by AI tool** card.
|
||||
- `/api/analytics`'s `total_cost` now equals `/api/pricing/cost` (it had
|
||||
ignored the compaction baselines and used UTC start dates).
|
||||
- Codex service tier read from `thread_settings_applied`: fast/priority →
|
||||
`speed = fast` priced with LiteLLM's `*_priority` rates (synced into the fast
|
||||
columns for non-Claude models), flex → `service_tier = flex` at half price.
|
||||
- Prometheus: `agenthub_cost_usd_total{tool,model}`.
|
||||
- DEFERRED (with the reason in `pricing-sync.js`): long-context (>200k/>272k)
|
||||
rates. No current Claude model has one, Codex here runs a 258k window
|
||||
(0 of 17,701 turns over 272k), so a `token_usage` primary-key rebuild would
|
||||
change no number today.
|
||||
|
||||
## Phase 3 — budgets, burn rate, forecast ✅ (budgets as alert rules)
|
||||
- `budgets` table: name, period (day/week/month), limit USD, scope
|
||||
(all / tool / workspace), thresholds (%), enabled. `/api/budgets` CRUD +
|
||||
`GET /api/budgets/status` (spend this period, %, projected end-of-period).
|
||||
- Evaluated on the alert beat; a crossed threshold fires once per period
|
||||
through the existing alerts feed → webhooks → push.
|
||||
- Alert rule type `cost_threshold` (USD per session / per day).
|
||||
- `POST /api/alerts/rules/preview`: dry-run a rule over recorded history.
|
||||
- UI: Settings → Budgets page; burn rate + month-end forecast tiles on
|
||||
Analytics → Cost; budget chip in the status bar when ≥ a threshold.
|
||||
- **As built:** a budget is an alert rule of type `budget` (`server/lib/budgets.js`;
|
||||
config `period` / `limit_usd` / `thresholds` / `tool` / `lane` / `tz_offset`),
|
||||
so there is no `budgets` table, no `/api/budgets` CRUD and no Settings →
|
||||
Budgets page: budgets are created and edited with the other rules in
|
||||
Settings → Notifications → Alert rules (editable in place since v2.3), and
|
||||
`GET /api/budgets/status` is the one budgets route. `cost_threshold`, the
|
||||
preview, the Spend pace + Budgets cards and the status-bar chip are as planned.
|
||||
|
||||
## Phase 4 — analytics UX ✅
|
||||
- **Date range.** `parseRange` (`server/lib/source-filter.js`): `from`/`to` are
|
||||
local `YYYY-MM-DD`, inclusive, read with `tz_offset`, either end open; a bad
|
||||
date (or year 9999, or `from` > `to`) is `400 INVALID_INPUT`.
|
||||
`GET /api/pricing/cost` keeps only the cost rows of those spend days for
|
||||
`total_cost` / `breakdown` / `by_tool` (the existing spend-day split);
|
||||
`daily_costs` stays the whole series so the 30-day charts and the spend pace
|
||||
keep their windows. `GET /api/analytics` counts sessions/agents by
|
||||
`started_at` and events by `created_at` (an optional `range` on the
|
||||
`scoped-stats.js` functions), and takes tokens and cost from the same cost
|
||||
rows (`token_usage` has no date); `daily_events`/`daily_sessions` keep 365
|
||||
days. `GET /api/workflows` (+ `/skill-usage`): sessions started in the range.
|
||||
The bodies are now `computeAnalytics` / `computeCost` (exported). Client:
|
||||
a range select on the Analytics toolbar beside the AI-tool select — 7d /
|
||||
30d / 90d / All (default) / Custom with two `<input type="date">` — in
|
||||
`?range=` (+ `from`/`to`), `client/src/lib/dateRange.ts`; it also reaches
|
||||
the Workflow sub-tab.
|
||||
- **Query Explorer.** `GET /api/query` (`server/routes/query.js`): entity
|
||||
sessions / events / agents, `q` (literal substring), `tool`, `status`,
|
||||
`from`/`to`, `cwd` (folder subtree), `sources`; `sort`/`dir` from a
|
||||
per-entity allowlist (anything else 400 — no request text reaches SQL),
|
||||
ties by id; pages of ≤ 500; `format=csv|json` exports the whole filter
|
||||
capped at 50,000 rows (`X-Total-Count`, `X-Truncated`). CSV: RFC 4180,
|
||||
CRLF, BOM, `'` before a cell starting with = + - @ tab CR. Page `/query`
|
||||
(`client/src/pages/QueryExplorer.tsx`), sidebar entry **Query Explorer**
|
||||
(the sidebar has no analytics item of its own; it sits under Sessions).
|
||||
Column headers show the raw field names, which are also the CSV header.
|
||||
- **Scheduled reports.** Tables `report_definitions` / `report_runs`;
|
||||
`server/lib/reports.js` + `/api/reports` (CRUD, `POST /:id/run`, runs,
|
||||
`GET /:id/runs/:runId/:format` with the path built from ids + the stored
|
||||
stamp). A run covers the `window_days` full local days BEFORE the run day,
|
||||
computed with `computeAnalytics` + `computeCost`, plus the current budget
|
||||
status; files `<data dir>/reports/<id>/<stamp>.{html,json}`, HTML fully
|
||||
escaped. Scheduler: unref'd interval in `startBackgroundServices`
|
||||
(`DASHBOARD_REPORTS_MS`, default 60 s, 0 = off), each slot claimed by one
|
||||
conditional UPDATE (two dashboards on one DB run it once); a slot missed
|
||||
while down runs once on the next start; create/edit/re-enable claims the
|
||||
slot already past. Weekly reports have a `weekday` (default Monday) —
|
||||
needed for "weekly", not in the original list. Writes same-origin.
|
||||
Definitions AND runs survive Clear Data (like alert rules and snapshots).
|
||||
UI: Settings → **Reports** (`components/settings/ReportSettings.tsx`).
|
||||
- Not done: OpenAPI entries for the new routes (snapshots/budgets have none
|
||||
either); report runs are never pruned (one file per run per format).
|
||||
- Upstream's PR branches (#286/#172 `feat/query-explorer`, #174
|
||||
`feat/scheduled-reports`) were not in the reference clone; built from the
|
||||
plan above.
|
||||
- Tests: `analytics-range`, `query-explorer`, `reports` in
|
||||
`server/__tests__/`; client `dateRange`, `QueryExplorer`, `ReportSettings`,
|
||||
and a `?range=` case in `screens.snapshot.test.tsx`.
|
||||
|
||||
## Phase 5 — sessions ✅
|
||||
- Replay: a **Replay** tab on `/sessions/:id` beside Agents / Conversation /
|
||||
Timeline (`client/src/components/EventReplay.tsx`). It reads the session's
|
||||
events through `GET /api/events?session_id=` in pages of 500 (the newest
|
||||
5,000 at most, with a note when there are more), oldest first, on the tab's
|
||||
first visit. A range slider over the event SEQUENCE (not the wall clock — an
|
||||
hour-long pause does not stall it), play/pause at 0.5×/1×/2×/5× (one event
|
||||
per 500 ms at 1×), step back/forward, jump start/end, keys ←/→/Home/End/Space,
|
||||
a mini-map of lifecycle markers (SessionStart, UserPromptSubmit, Compaction,
|
||||
SubagentStop, APIError, Stop) that jump the cursor, a state panel derived up
|
||||
to the cursor (status, tool in flight, prompts, errors, compactions, elapsed,
|
||||
tool-call counts) and the selected event through `EventDetail`. The pure
|
||||
derivations live in `client/src/lib/replay.ts` (unit-tested).
|
||||
- Snapshots: `POST /api/snapshots {session_id, redact?: {file_paths?}}` freezes
|
||||
`{session, agents, events, stats, cost}` into an immutable
|
||||
`session_snapshots` row (a trigger refuses UPDATE) under a 24-byte base64url
|
||||
token; `GET /api/snapshots` (no payload), `GET /api/snapshots/:token`,
|
||||
`DELETE /api/snapshots/:id`; writes behind `sameOriginGuard`. Redaction runs
|
||||
at capture: path-valued keys → `[redacted]`, the session cwd / home dir in
|
||||
any other string → `<cwd>` / `~`. Snapshots survive Clear Data, like alert
|
||||
rules. UI: a **Snapshot** action in the session header (confirm with a
|
||||
"redact file paths" box, default on; copies the link), the read-only page
|
||||
`/snapshot/:token` (`client/src/pages/SnapshotView.tsx`: SessionOverview fed
|
||||
the frozen stats via its new `stats` prop, the agents, Replay and Timeline —
|
||||
the Timeline list now `client/src/components/EventTimeline.tsx`, shared with
|
||||
the analysis page), and Settings → **Snapshots** (`/settings/snapshots`,
|
||||
`components/settings/SnapshotSettings.tsx`: list, open, copy link, delete
|
||||
with confirm, the same on a right-click menu).
|
||||
- Upstream's PR branches (#265 `feat/session-replay`, #173
|
||||
`feat/session-snapshots`) were not in the reference clone; built from the
|
||||
plan above.
|
||||
|
||||
## Phase 6 — ingest breadth ✅
|
||||
- **Remote Codex.** A remote source has an opt-in `sync_codex` (+ optional
|
||||
`remote_codex_home`, default `~/.codex`, `wsl:~/.codex` beside a `wsl:` Claude
|
||||
home) — two additive columns on `remote_sources`, a checkbox in the source
|
||||
form, `--codex`/`--codex-home` on `agenthub remote-sources add`. A sync mirrors
|
||||
`sessions/` with mtimes kept (`scp -r -p`, or tar from WSL) plus
|
||||
`session_index.jsonl` into `<data dir>/remote-sources/<id>/codex/`, then runs
|
||||
`ingestCodexUsage({home, source, activeMs})` — the same journal reader as local
|
||||
Codex (`server/lib/codex-usage.js`), so sessions arrive as `tool = 'codex'`
|
||||
rows tagged with the source id, with tokens and cost. A pass only ever writes
|
||||
rows of its own source; the local settle never touches a remote row; the
|
||||
remote activity window is `DASHBOARD_REMOTE_ACTIVE_WINDOW_MS`. With Codex on, a
|
||||
machine may have only one CLI: one half failing is `ok` + `last_error`.
|
||||
Differs from upstream (which always tries both): opt-in, and no separate
|
||||
Codex import pipeline — AgentHub's journal reader is reused.
|
||||
- **Push ingest.** `POST /api/hooks/ingest-batch` — body `{source, events:
|
||||
[{hook_type, data, tool?}]}`, each event through the SAME function as
|
||||
`POST /api/hooks/event` (`ingestHook` in `server/routes/hooks.js`). Off (503)
|
||||
until Settings → Data → **Push ingest** creates a token: 32 random bytes,
|
||||
shown once, stored only as its SHA-256 (`AGENTHUB_PUSH_TOKEN_SHA256` in the
|
||||
settings file), checked in constant time; rotate/revoke; `GET` never returns
|
||||
it. Sessions tagged `push:<source>`; a pushed hook may only extend a session
|
||||
its own source created (`EFOREIGN`); `transcript_path`s are dropped (they
|
||||
name the sender's files); no lane/pane binding; 500 events / 1 MB. The UI
|
||||
shows a forwarder script keeping the token in a curl header file, not argv.
|
||||
Differs from upstream: upstream's wire format is a collector's own session/
|
||||
token/turn records; AgentHub takes plain hook payloads, so a pushed session
|
||||
has events and live status but no token cost (that needs transcripts).
|
||||
- **Live Codex overlay — not built.** Upstream's `include_transient`
|
||||
(`codex-process-overlay.js`) scans `ps` for an interactive `codex` TUI that
|
||||
has not written a rollout yet, i.e. one started but never prompted. In
|
||||
AgentHub a Codex pane started from the dashboard already shows (its tab, and
|
||||
**New session** in the sidebar from its run), and anything that has done work
|
||||
has a journal the 15 s poll reads (or a SessionStart hook with Codex hooks
|
||||
on). The only row it would add is an idle Codex in an outside terminal with
|
||||
no prompt and no cost — not worth a per-tick process scan.
|
||||
- **Privacy** (`server/lib/privacy.js`, Settings → Data → **Privacy**,
|
||||
`GET/PUT /api/settings/privacy`, both off by default, persisted to the
|
||||
settings file, env-pinnable): (a) **Redact secrets** (`AGENTHUB_REDACT_SECRETS=1`)
|
||||
— chained in front of the trim in `trim_event_data` (so a key is redacted
|
||||
before a trim could cut it) plus a new `redact_text(?)` on every events
|
||||
`summary` insert; known key shapes (Anthropic, OpenAI, GitHub, GitLab, Slack,
|
||||
AWS, Google, npm, JWT), bearer/basic headers, PEM private keys,
|
||||
`password=`/`*_TOKEN:`-style values, and any string under a secret-named JSON
|
||||
key → `[redacted]`, with `_redacted: <n>` on the payload. New data only; the
|
||||
transcript is untouched; session NAMES (first prompt) are not redacted.
|
||||
(b) **Ignored folders** (`AGENTHUB_IGNORE_CWD`, `path.delimiter`-separated,
|
||||
path-boundary match): hooks dropped whole (`ignored: true`), and the history
|
||||
import and the Codex reader skip those sessions too.
|
||||
|
||||
## Phase 7 — extras chosen by the user
|
||||
- Cursor CLI: session + token ingest from `~/.cursor`, Cursor pricing rows.
|
||||
- MCP server (stdio, read-only, ~15 tools: stats, cost, sessions, session
|
||||
detail, analytics, workflows, budgets, alerts).
|
||||
- `statusline/` script for Claude Code (model, branch, context %, session $,
|
||||
today $ from the dashboard).
|
||||
- `monitoring/` Grafana dashboards + Prometheus rules for `/api/metrics`.
|
||||
|
||||
### Phase 7 status — MCP, statusline, monitoring ✅; Cursor ✅ (no cost)
|
||||
- **MCP** (`mcp/server.js`, `agenthub mcp`, docs/MCP.md): JSON-RPC over stdio
|
||||
written directly (no SDK), 15 GET-only tools (health, info, stats, total /
|
||||
session / lane cost, sessions, session, session stats, analytics, workflows,
|
||||
alerts, lanes, pricing, budget status). `AGENTHUB_URL`, else the CLI's
|
||||
discovery, now shared as `server/lib/dashboard-url.js`. A missing route —
|
||||
404 or the production SPA page served with 200 — reads "Not available on
|
||||
this dashboard", which is what `get_budget_status` says until Phase 3 lands.
|
||||
NOT auto-registered by the plugin: `claude mcp add agenthub -- agenthub mcp`.
|
||||
- **Statusline** (`statusline/statusline.js`, `agenthub statusline
|
||||
install|uninstall [--force]`, docs/STATUSLINE.md): model · branch · context
|
||||
bar · session $ · today $ (`/api/pricing/cost`, 1 s timeout, 30 s cache in
|
||||
the OS temp dir, failures cached). Install refuses a foreign `statusLine`
|
||||
without `--force` and backs settings.json up first; a plugin install
|
||||
registers the `~/.local/bin/agenthub` launcher, not the cache dir.
|
||||
- **Monitoring** (`monitoring/`, `docker-compose.monitoring.yml`,
|
||||
docs/MONITORING.md): scrape config, `agenthub:*` recording rules, boards
|
||||
Overview / Sessions & agents / Tokens & cost; the spend panels read Phase 2's
|
||||
`agenthub_cost_usd_total{tool,model}` and say so.
|
||||
- Tests: `mcp-server`, `mcp-server-live`, `statusline`, `monitoring` in
|
||||
`server/__tests__/`.
|
||||
- **Cursor** (`server/lib/cursor-usage.js`): agent transcripts under
|
||||
`~/.cursor/projects/<slug>/agent-transcripts/` become `tool = 'cursor'`
|
||||
sessions (`cursor_transcripts` table, `DASHBOARD_CURSOR_POLL_MS`,
|
||||
`AGENTHUB_CURSOR_DIR`). NOT built: tokens, model and cost — the transcripts
|
||||
record none, and `state.vscdb` (which has them) also holds auth tokens, so
|
||||
it is never read; hence no Cursor pricing rows either.
|
||||
|
||||
## Ported onto main
|
||||
Phases 2–7 were built in a separate worktree on 138e706 and ported onto v2.3.1
|
||||
on 2026-10-06 (phase 1 had shipped on its own), released as v2.4.0.
|
||||
Reference in New Issue
Block a user