nntrivi2001
4aae61297f
Analytics parity phases 2–7 (docs/plans/analytics-parity/), built on
138e706 and ported onto v2.3.1:
- Cost by AI tool (?tool= on cost/analytics/stats/workflows, by_tool,
Codex fast/priority and flex tiers, agenthub_cost_usd_total{tool,model})
- Date ranges (from/to) on cost/analytics/workflows; Analytics range select
with Today / This week (Monday) / This month (1st) / 7d / 30d / 90d / All /
Custom; workflow filters carried into workflow-stats.js (read worker)
- Budgets and cost alerts as alert rule types (alert_rules rebuilt without
its rule_type CHECK), GET /api/budgets/status, rule preview, spend pace,
status-bar budget chip; rules stay editable in place
- Query Explorer (/query, allowlisted sort, CSV/JSON export) and scheduled
reports (report_definitions/report_runs, Settings → Reports)
- Session Replay tab and immutable snapshots (session_snapshots, /snapshot/:token)
- Remote Codex sync, push ingest with a hashed token, privacy redaction and
ignored folders (Settings → Data), Cursor sessions (no cost)
- Read-only MCP server, statusline, Grafana/Prometheus files
Also: resizable sidebar, pet run/wave animations, the workspace home keeps a
half-typed prompt, and the pre-commit hook runs only the related tests for a
small commit.
61 lines
3.4 KiB
Markdown
61 lines
3.4 KiB
Markdown
# Prometheus and Grafana for AgentHub
|
|
|
|
AgentHub exposes its live counters at `GET /api/metrics` in the Prometheus text
|
|
format (see [API.md § Prometheus exposition](API.md#prometheus-exposition)).
|
|
`monitoring/` holds a scrape config, recording rules and three Grafana boards
|
|
built on those series, and `docker-compose.monitoring.yml` runs both tools with
|
|
everything provisioned.
|
|
|
|
## Files
|
|
|
|
| Path | What |
|
|
|---|---|
|
|
| `monitoring/prometheus/prometheus.yml` | Scrape config: job `agenthub`, `metrics_path: /api/metrics`, target `host.docker.internal:4820` |
|
|
| `monitoring/prometheus/agenthub-rules.yml` | Recording rules `agenthub:*` (sessions, agents, event and token rates, cost) |
|
|
| `monitoring/grafana/dashboards/agenthub-overview.json` | **Overview**: up, version, active sessions, working agents, tokens and spend over 24 h, sessions/agents by status, events/min, token burn, RSS, uptime, websocket clients, remote sources |
|
|
| `monitoring/grafana/dashboards/agenthub-sessions-agents.json` | **Sessions & agents**: counts by status, errored sessions, agents waiting on you, sessions finished per hour |
|
|
| `monitoring/grafana/dashboards/agenthub-tokens-cost.json` | **Tokens & cost**: tokens per hour by kind, burn rate, cache-read share, spend all-time / 24 h, spend per hour by tool, spend by tool, top models by spend |
|
|
| `monitoring/grafana/provisioning/` | The Prometheus datasource and the dashboard provider for the compose stack |
|
|
| `docker-compose.monitoring.yml` | Prometheus + Grafana, both on loopback only |
|
|
|
|
The boards choose their datasource through a `ds` variable, so they also import
|
|
as they are into an existing Grafana (Dashboards → New → Import).
|
|
|
|
## Metrics used
|
|
|
|
`agenthub_up`, `agenthub_build_info{version}`, `agenthub_process_uptime_seconds`,
|
|
`agenthub_process_resident_memory_bytes`, `agenthub_sessions{status}`,
|
|
`agenthub_agents{status}`, `agenthub_events_total`, `agenthub_websocket_clients`,
|
|
`agenthub_remote_sources{enabled}`, `agenthub_tokens_total{kind}`.
|
|
|
|
The spend panels and the `agenthub:cost_usd:*` rules read
|
|
**`agenthub_cost_usd_total{tool,model}`** (estimated USD, only ever rising
|
|
except when pricing is recomputed). A dashboard that does not export it yet
|
|
leaves those panels empty; each one says so in its description.
|
|
`server/__tests__/monitoring.test.js` fails if a board or rule queries any other
|
|
metric `server/routes/metrics.js` does not export.
|
|
|
|
## Run it with Docker
|
|
|
|
```bash
|
|
# 1. The dashboard on the host must accept the container's Host header
|
|
DASHBOARD_ALLOWED_HOSTS=host.docker.internal agenthub start # or npm start
|
|
# 2. Prometheus + Grafana
|
|
GF_SECURITY_ADMIN_PASSWORD='choose-one' docker compose -f docker-compose.monitoring.yml up -d
|
|
# 3. Grafana: http://127.0.0.1:3000 (admin / the password above), folder "AgentHub"
|
|
# Prometheus: http://127.0.0.1:9090
|
|
```
|
|
|
|
Without step 1 every scrape gets `403 EBADHOST` (the DNS-rebinding guard).
|
|
When the dashboard sets `DASHBOARD_TOKEN`, put the token in
|
|
`monitoring/prometheus/dashboard-token` and uncomment the `authorization` block in
|
|
`prometheus.yml`. Stop with `docker compose -f docker-compose.monitoring.yml down`
|
|
(`-v` also drops the stored series).
|
|
|
|
## Without Docker
|
|
|
|
Point an existing Prometheus at the dashboard — copy the `agenthub` job from
|
|
`prometheus.yml` with the target changed to `127.0.0.1:4820` (no
|
|
`DASHBOARD_ALLOWED_HOSTS` needed for loopback), add `agenthub-rules.yml` to its
|
|
`rule_files`, and import the three JSON boards into Grafana.
|