# Prometheus and Grafana for AgentHub

AgentHub exposes its live counters at `GET /api/metrics` in the Prometheus text
format (see [API.md § Prometheus exposition](API.md#prometheus-exposition)).
`monitoring/` holds a scrape config, recording rules and three Grafana boards
built on those series, and `docker-compose.monitoring.yml` runs both tools with
everything provisioned.

## Files

| Path | What |
|---|---|
| `monitoring/prometheus/prometheus.yml` | Scrape config: job `agenthub`, `metrics_path: /api/metrics`, target `host.docker.internal:4820` |
| `monitoring/prometheus/agenthub-rules.yml` | Recording rules `agenthub:*` (sessions, agents, event and token rates, cost) |
| `monitoring/grafana/dashboards/agenthub-overview.json` | **Overview**: up, version, active sessions, working agents, tokens and spend over 24 h, sessions/agents by status, events/min, token burn, RSS, uptime, websocket clients, remote sources |
| `monitoring/grafana/dashboards/agenthub-sessions-agents.json` | **Sessions & agents**: counts by status, errored sessions, agents waiting on you, sessions finished per hour |
| `monitoring/grafana/dashboards/agenthub-tokens-cost.json` | **Tokens & cost**: tokens per hour by kind, burn rate, cache-read share, spend all-time / 24 h, spend per hour by tool, spend by tool, top models by spend |
| `monitoring/grafana/provisioning/` | The Prometheus datasource and the dashboard provider for the compose stack |
| `docker-compose.monitoring.yml` | Prometheus + Grafana, both on loopback only |

The boards choose their datasource through a `ds` variable, so they also import
as they are into an existing Grafana (Dashboards → New → Import).

## Metrics used

`agenthub_up`, `agenthub_build_info{version}`, `agenthub_process_uptime_seconds`,
`agenthub_process_resident_memory_bytes`, `agenthub_event_loop_delay_seconds{quantile}`,
`agenthub_sessions{status}`,
`agenthub_agents{status}`, `agenthub_events_total`, `agenthub_websocket_clients`,
`agenthub_remote_sources{enabled}`, `agenthub_tokens_total{kind}`.

The spend panels and the `agenthub:cost_usd:*` rules read
**`agenthub_cost_usd_total{tool,model}`** (estimated USD, only ever rising
except when pricing is recomputed). A dashboard that does not export it yet
leaves those panels empty; each one says so in its description.
`server/__tests__/monitoring.test.js` fails if a board or rule queries any other
metric `server/routes/metrics.js` does not export.

## Run it with Docker

```bash
# 1. The dashboard on the host must accept the container's Host header
DASHBOARD_ALLOWED_HOSTS=host.docker.internal agenthub start   # or npm start
# 2. Prometheus + Grafana
GF_SECURITY_ADMIN_PASSWORD='choose-one' docker compose -f docker-compose.monitoring.yml up -d
# 3. Grafana: http://127.0.0.1:3000 (admin / the password above), folder "AgentHub"
#    Prometheus: http://127.0.0.1:9090
```

Without step 1 every scrape gets `403 EBADHOST` (the DNS-rebinding guard).
When the dashboard sets `DASHBOARD_TOKEN`, put the token in
`monitoring/prometheus/dashboard-token` and uncomment the `authorization` block in
`prometheus.yml`. Stop with `docker compose -f docker-compose.monitoring.yml down`
(`-v` also drops the stored series).

## Without Docker

Point an existing Prometheus at the dashboard — copy the `agenthub` job from
`prometheus.yml` with the target changed to `127.0.0.1:4820` (no
`DASHBOARD_ALLOWED_HOSTS` needed for loopback), add `agenthub-rules.yml` to its
`rule_files`, and import the three JSON boards into Grafana.