nntrivi2001
62425b2f58
Internal SmartGift build of a Claude Code monitoring dashboard. Lanes: a durable unit of parallel agent work, one per working directory, tracked across session restarts. Managed lanes are git worktrees the dashboard provisions and can reset or remove behind a three-check destroy guard and a counted preflight; adopted lanes are directories you already own and are never destroyable. Pipelines: a lane moves through pipeline stages. A stage the agent declares with evidence renders green; a stage inferred from the tool-event stream renders dashed amber and never counts as done. Detection is forward-only within a 30-minute window, and never writes the declared stage. Workspace: one page at /run with a lane grid, the selected lane's pipeline, and a full Claude console behind a disclosure.
273 lines
12 KiB
Markdown
273 lines
12 KiB
Markdown
# CCAM monitoring stack (Prometheus + Grafana)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A turnkey [Prometheus](https://prometheus.io/) + [Grafana](https://grafana.com/)
|
|
stack that scrapes the dashboard's [`GET /api/metrics`](../docs/API.md#metrics)
|
|
endpoint and renders **four auto-provisioned Grafana dashboards** (default home:
|
|
**CCAM — Overview**). Use it to watch live
|
|
sessions, agent states, event throughput, and token burn from the same
|
|
observability stack as the rest of your infra.
|
|
|
|
**No Homebrew, apt, or global installs required.** There is no official `prometheus` or
|
|
`grafana` server package on npm (only client libraries and UI components) — the
|
|
stack uses npm's own install lifecycle instead: `npm run monitoring:install`
|
|
runs `postinstall` in this folder and pulls official release binaries into
|
|
`monitoring/.bin/` (same pattern as Playwright browsers or Electron).
|
|
|
|
### Supported platforms (npm path)
|
|
|
|
| OS | Architectures |
|
|
| --- | --- |
|
|
| macOS | Apple Silicon (`arm64`), Intel (`x64`) |
|
|
| Linux | `arm64`, `amd64` (`x64`) |
|
|
| Windows | `x64` |
|
|
|
|
Node.js 20+ is the only prerequisite. On Windows, run the npm commands from
|
|
PowerShell or Command Prompt in the repo root (same as the rest of CCAM).
|
|
|
|
## Grafana login (local default)
|
|
|
|
Open <http://localhost:3000> after `monitoring:up` or `monitoring:docker:up`.
|
|
The admin account is **auto-created on first start** — no manual signup:
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| Username | `admin` |
|
|
| Password | `admin` |
|
|
|
|
Credentials are defined in [`grafana.defaults.env`](./grafana.defaults.env) and
|
|
applied by both the npm and Docker paths. **CCAM — Overview** is the default
|
|
home dashboard (`GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_UID=ccam-overview`) with
|
|
PromQL queries against your live `/api/metrics` scrape — no sample or synthetic
|
|
data. The Prometheus datasource is pre-provisioned, so you land straight in the
|
|
UI after login.
|
|
|
|
> If login fails after an earlier Grafana run, reset local state:
|
|
> `rm -rf monitoring/.data/grafana` (npm) or recreate the `grafana-data` Docker
|
|
> volume, then start the stack again.
|
|
|
|
```
|
|
monitoring/
|
|
├── package.json # npm install downloads binaries (postinstall)
|
|
├── grafana.defaults.env # default admin / admin credentials
|
|
├── scripts/ # lifecycle helpers (setup / up / down)
|
|
├── prometheus/
|
|
│ ├── prometheus-native.yml # scrape config for npm-managed stack
|
|
│ ├── prometheus.yml # scrape config for Docker stack
|
|
│ ├── prometheus-docker.yml # all-Docker stack (docker-compose.full.yml)
|
|
│ ├── ccam-rules.yml # recording rules (derived from live scrapes)
|
|
│ └── consoles/index.html # Prometheus CCAM console (pre-built graphs)
|
|
├── grafana/
|
|
│ ├── provisioning/… # Docker datasource + dashboard provider
|
|
│ ├── provisioning-native/… # npm-managed datasource template
|
|
│ └── dashboards/ # ccam-overview + 3 focused boards
|
|
└── docker-compose.yml # optional Docker path
|
|
```
|
|
|
|
## Quick start (npm — no Docker)
|
|
|
|
1. **Start the dashboard** on loopback (default `npm start` on port 4820). No
|
|
`DASHBOARD_ALLOWED_HOSTS` override is needed — Prometheus scrapes
|
|
`127.0.0.1:4820`, which the server's Host guard already accepts.
|
|
|
|
```bash
|
|
npm start
|
|
```
|
|
|
|
2. **One-time install** — uses npm's `postinstall` to download official
|
|
Prometheus + Grafana OSS binaries into `monitoring/.bin/` (~150 MB total):
|
|
|
|
```bash
|
|
npm run monitoring:install
|
|
```
|
|
|
|
(`monitoring:setup` is an alias for the same command.)
|
|
|
|
3. **Bring up the stack:**
|
|
|
|
```bash
|
|
npm run monitoring:up
|
|
```
|
|
|
|
4. **Open Grafana** at <http://localhost:3000> (login `admin` / `admin`). The
|
|
**CCAM — Overview** dashboard is already there — no import step. For
|
|
Prometheus, open the pre-built **CCAM console** at
|
|
<http://localhost:9090/consoles/index.html> (live graphs + tables), or
|
|
<http://localhost:9090> → **Consoles** → `index.html` (`Status → Targets`:
|
|
`ccam` should be **UP**).
|
|
|
|
Stop with `npm run monitoring:down`. For a foreground session with logs on the
|
|
terminal, use `npm run monitoring:start` instead (Ctrl+C stops both).
|
|
|
|
## Quick start (Docker / Podman)
|
|
|
|
Use this when you prefer containers or when the dashboard itself runs in Docker.
|
|
|
|
1. **Start the dashboard** so the container can scrape it. The server's
|
|
DNS-rebinding guard only accepts loopback `Host` headers, and Prometheus (in
|
|
Docker) reaches the host as `host.docker.internal` — so allow that Host:
|
|
|
|
```bash
|
|
DASHBOARD_ALLOWED_HOSTS=host.docker.internal npm start
|
|
```
|
|
|
|
When the dashboard runs **inside Docker** (`docker compose up` at the repo
|
|
root), set the same variable on the `agent-monitor` service:
|
|
|
|
```bash
|
|
DASHBOARD_ALLOWED_HOSTS=host.docker.internal docker compose up -d
|
|
```
|
|
|
|
> Without this you'll see the Prometheus target stuck **DOWN** with
|
|
> `403 EBADHOST`. If you also set `DASHBOARD_TOKEN`, see [Auth](#auth) below.
|
|
|
|
2. **Bring up the monitoring stack:**
|
|
|
|
```bash
|
|
npm run monitoring:docker:up
|
|
```
|
|
|
|
3. **Open Grafana** at <http://localhost:3000> (login `admin` / `admin`).
|
|
|
|
Tear down with `npm run monitoring:docker:down`.
|
|
|
|
### All-in-one Docker (app + monitoring)
|
|
|
|
Runs the dashboard, Prometheus, and Grafana on one network — no
|
|
`host.docker.internal` wiring required:
|
|
|
|
```bash
|
|
npm run docker:full:up
|
|
npm run monitoring:verify
|
|
```
|
|
|
|
Stop with `npm run docker:full:down`.
|
|
|
|
## Deployment options
|
|
|
|
| Dashboard | Monitoring | Commands |
|
|
| --- | --- | --- |
|
|
| **npm** (`npm start`) | **npm** | `monitoring:install` → `monitoring:up` |
|
|
| **npm** | **Docker** | `DASHBOARD_ALLOWED_HOSTS=host.docker.internal npm start` → `monitoring:docker:up` |
|
|
| **Docker** (`docker:up`) | **Docker** | `DASHBOARD_ALLOWED_HOSTS=host.docker.internal docker compose up -d` → `monitoring:docker:up` |
|
|
| **Docker** | **npm** | `docker:up` + `monitoring:up` (scrapes host `127.0.0.1:4820`) |
|
|
| **Docker full stack** | **included** | `docker:full:up` |
|
|
|
|
Verify any running stack: `npm run monitoring:verify`.
|
|
|
|
## npm scripts
|
|
|
|
| Command | Description |
|
|
| --- | --- |
|
|
| `npm run monitoring:install` | `npm install` in `monitoring/` — downloads binaries via `postinstall` |
|
|
| `npm run monitoring:setup` | Alias for `monitoring:install` |
|
|
| `npm run monitoring:up` | Start Prometheus (:9090) + Grafana (:3000) in the background |
|
|
| `npm run monitoring:down` | Stop the npm-managed stack |
|
|
| `npm run monitoring:start` | Foreground start (Ctrl+C stops both) |
|
|
| `npm run monitoring:docker:up` | `docker compose` Prometheus + Grafana |
|
|
| `npm run monitoring:docker:down` | Tear down the Docker monitoring stack |
|
|
| `npm run monitoring:verify` | Health-check dashboard + Prometheus + Grafana + scrape target |
|
|
| `npm run docker:up` | Start the dashboard container only |
|
|
| `npm run docker:down` | Stop the dashboard container |
|
|
| `npm run docker:full:up` | Dashboard + Prometheus + Grafana (all Docker) |
|
|
| `npm run docker:full:down` | Tear down the full Docker stack |
|
|
|
|
## Bundled Grafana dashboards
|
|
|
|
Four dashboards are auto-provisioned from `grafana/dashboards/` — all query live
|
|
`/api/metrics` data (no sample or synthetic series):
|
|
|
|
| Dashboard | UID | Focus |
|
|
| --- | --- | --- |
|
|
| **CCAM — Overview** | `ccam-overview` | Default home — fleet snapshot, totals, breakdowns, rates |
|
|
| **CCAM — Sessions & Agents** | `ccam-sessions-agents` | Session lifecycle, agent states, WebSocket clients |
|
|
| **CCAM — Tokens & Events** | `ccam-tokens-events` | Cumulative tokens/events, throughput rates, cache efficiency |
|
|
| **CCAM — Platform Health** | `ccam-platform` | Scrape/API uptime, process memory, remote sources, build info |
|
|
|
|
Each board links to the others in the header. After `monitoring:up`, open
|
|
http://localhost:3000/dashboards or land on **Overview** as the home dashboard.
|
|
|
|
## What's on the Overview dashboard
|
|
|
|
The bundled **CCAM — Overview** board uses only metrics from your database.
|
|
**Cumulative stat panels** (total sessions, events, tokens) show substantial
|
|
numbers on the first scrape; **rate panels** need a few minutes of scrape
|
|
history before lines appear.
|
|
|
|
| Section | Example queries |
|
|
| --- | --- |
|
|
| Live fleet | `ccam_sessions{status="active"}`, `ccam_agents{status="working"}`, `ccam_websocket_clients`, `sum(ccam_sessions)` |
|
|
| Database totals | `ccam_sessions{status="completed"}`, `ccam_events_total`, `ccam_tokens_total`, `sum(ccam_tokens_total)` |
|
|
| Breakdown | `ccam_sessions` (pie), `ccam_tokens_total` (bar gauge) |
|
|
| Over time | `ccam_sessions`, `ccam_events_total`, `ccam_tokens_total` |
|
|
| Rates | `rate(ccam_events_total[5m])`, `rate(ccam_tokens_total[5m])` |
|
|
| Process | `ccam_process_uptime_seconds`, `ccam_process_resident_memory_bytes`, `ccam_build_info` |
|
|
|
|
See [`docs/API.md` → Metrics](../docs/API.md#metrics) for the full metric list.
|
|
|
|
## Prometheus quick start
|
|
|
|
**Open the CCAM console first** — static HTML that queries Prometheus directly
|
|
(Prometheus 3.x compatible; no deprecated console template libraries):
|
|
|
|
**http://localhost:9090/consoles/index.html**
|
|
|
|
Also reachable from the Prometheus UI menu: **Consoles → index.html**.
|
|
|
|
The console runs real PromQL against your scraped metrics: session totals,
|
|
cumulative events/tokens, working agents, and drill-down links into the Graph UI.
|
|
If the page warns about a missing scrape target, start CCAM on port 4820 and
|
|
wait ~15s for the first poll.
|
|
|
|
### Graph tab bookmarks
|
|
|
|
Open [http://localhost:9090](http://localhost:9090) and paste any of these into
|
|
the **Graph** tab (all read live scraped data — never seeded):
|
|
|
|
| Query | What it shows |
|
|
| --- | --- |
|
|
| `sum(ccam_sessions)` | Total sessions across all statuses |
|
|
| `ccam_events_total` | Cumulative hook events in the database |
|
|
| `sum(ccam_tokens_total)` | All token kinds combined |
|
|
| `ccam:sessions:total` | Same as above via [recording rule](./prometheus/ccam-rules.yml) |
|
|
| `ccam:events:rate5m` | Events per second (5m window) |
|
|
|
|
Pre-filled graph links (bookmark these):
|
|
|
|
- [Total sessions](http://localhost:9090/graph?g0.expr=sum(ccam_sessions)&g0.tab=0)
|
|
- [Total events](http://localhost:9090/graph?g0.expr=ccam_events_total&g0.tab=0)
|
|
- [Token totals by kind](http://localhost:9090/graph?g0.expr=ccam_tokens_total&g0.tab=0)
|
|
|
|
## Configuration
|
|
|
|
- **npm-managed stack.** Scrape target lives in
|
|
[`prometheus/prometheus-native.yml`](./prometheus/prometheus-native.yml)
|
|
(default `127.0.0.1:4820`). Binary versions are pinned in
|
|
[`scripts/paths.js`](./scripts/paths.js).
|
|
- **Docker stack.** Host-native dashboard: [`prometheus/prometheus.yml`](./prometheus/prometheus.yml)
|
|
(`host.docker.internal:4820`). All-Docker stack: [`prometheus/prometheus-docker.yml`](./prometheus/prometheus-docker.yml)
|
|
(`agent-monitor:4820`, used by `docker-compose.full.yml`).
|
|
- **<a id="auth"></a>Auth (`DASHBOARD_TOKEN`).** If the server requires a token,
|
|
uncomment the `authorization` block in the relevant prometheus config and set
|
|
`credentials` to your `DASHBOARD_TOKEN`. (The `DASHBOARD_ALLOWED_HOSTS` step is
|
|
still required for Docker scrapes — the Host guard runs independently of the
|
|
token.)
|
|
- **Scrape interval** lives in the prometheus config (`global.scrape_interval`).
|
|
- **Grafana admin password** — defaults to `admin` / `admin` via
|
|
[`grafana.defaults.env`](./grafana.defaults.env) (Docker) and `grafanaAdminEnv()`
|
|
in [`scripts/paths.js`](./scripts/paths.js) (npm). Applied on **first start**
|
|
only; wipe `monitoring/.data/grafana` or the `grafana-data` volume to re-seed.
|
|
|
|
## Security note
|
|
|
|
`/api/metrics` exposes aggregate operational counts (session/agent tallies, event
|
|
and token totals, uptime) — no prompts, transcripts, costs, or secrets. It sits
|
|
behind the same loopback/Host guard and optional `DASHBOARD_TOKEN` as the rest of
|
|
the API; scraping is only possible once you explicitly allow the scraper's Host
|
|
(and token, if set). Keep Grafana/Prometheus on a trusted network or behind your
|
|
own reverse proxy.
|