nntrivi2001
57dc91585d
Internal SmartGift build of a Claude Code monitoring dashboard. Lanes: a durable unit of parallel agent work, one per working directory, tracked across session restarts. Managed lanes are git worktrees the dashboard provisions and can reset or remove behind a three-check destroy guard and a counted preflight; adopted lanes are directories you already own and are never destroyable. Pipelines: a lane moves through pipeline stages. A stage the agent declares with evidence renders green; a stage inferred from the tool-event stream renders dashed amber and never counts as done. Detection is forward-only within a 30-minute window, and never writes the declared stage. Workspace: one page at /run with a lane grid, the selected lane's pipeline, and a full Claude console behind a disclosure.
83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
---
|
||
description: >
|
||
Detect quality and efficiency regressions over time using Agent Monitor data —
|
||
rising error rate (APIError events), falling cache hit rate, growing compaction
|
||
frequency, and climbing cost-per-session. Splits history into an earlier
|
||
baseline window and a recent window and reports which metrics are getting
|
||
worse, by how much, and where. Use when checking whether things are degrading
|
||
or trending in the wrong direction.
|
||
---
|
||
|
||
# Regression Watch
|
||
|
||
Detect whether Claude Code sessions are getting worse over time across quality and
|
||
efficiency metrics, using Agent Monitor data.
|
||
|
||
## Input
|
||
|
||
The user provides: **$ARGUMENTS**
|
||
|
||
This may be:
|
||
- empty or "all" — check every regression metric (default)
|
||
- "errors" — error-rate regression only
|
||
- "cache" — cache hit-rate regression only
|
||
- "compaction" — compaction-frequency regression only
|
||
- "cost" — cost-per-session regression only
|
||
- A window like "last 30d" or "30 vs 90" — set the recent vs baseline window sizes
|
||
|
||
## Data Sources
|
||
|
||
| Endpoint | Returns |
|
||
|----------|---------|
|
||
| `GET /api/analytics` | `daily_events` (365d), `daily_sessions` (365d), `event_types`, `tokens` (total_input, total_output, total_cache_read, total_cache_write — baselines pre-summed), `avg_events_per_session` |
|
||
| `GET /api/events?session_id=X` | Event stream incl. `APIError`, `Compaction`, `PreToolUse`/`PostToolUse` — used to localize regressions to specific sessions |
|
||
| `GET /api/pricing/cost` | `{ total_cost, breakdown[...] }` — total cost to derive cost-per-session |
|
||
| `GET /api/pricing/cost/{sessionId}` | Per-session cost — used to compare recent vs baseline session cost |
|
||
| `GET /api/workflows/{sessionId}` | `compaction` (impact), `errorPropagation` (by depth), `effectiveness` — per-session quality signals |
|
||
| `GET /api/sessions?limit=N` | Sessions with `started_at`, `cost`, `metadata` — to bucket sessions into time windows |
|
||
|
||
## Report Sections
|
||
|
||
### 1. Windowing
|
||
Split history into a **baseline window** (older) and a **recent window** (newer).
|
||
Default: recent = last 30 days, baseline = the 30–90 day range before it. Use
|
||
`daily_events`/`daily_sessions` for series metrics and `GET /api/sessions?limit=N`
|
||
to assign sessions to each window by `started_at`.
|
||
|
||
### 2. Error Rate Regression
|
||
- Recent error rate = `APIError count / total events` in the recent window
|
||
(from `event_types` and `daily_events`, or per-session `GET /api/events`).
|
||
- Compare to the baseline rate. Flag if recent is higher.
|
||
- Report the absolute and relative change and which sessions contributed most
|
||
`APIError` events.
|
||
|
||
### 3. Cache Hit Rate Regression
|
||
- Cache hit rate = `total_cache_read / (total_cache_read + total_input)`.
|
||
- Compute for each window (per-window input/cache_read from session metadata or
|
||
the pricing breakdown). Flag a **falling** hit rate — that means more
|
||
uncached input tokens and higher cost.
|
||
|
||
### 4. Compaction Frequency Regression
|
||
- Compaction frequency = `Compaction events / session` per window (from
|
||
`event_types` / `daily_events`, confirmed via per-session
|
||
`GET /api/workflows/{id}` `compaction`). Flag a **rising** rate — context is
|
||
overflowing more often.
|
||
|
||
### 5. Cost-per-Session Regression
|
||
- Cost-per-session = window total cost / window session count, using
|
||
`GET /api/pricing/cost` overall and `GET /api/pricing/cost/{id}` for the
|
||
sessions in each window. Flag a **climbing** value.
|
||
|
||
### 6. Verdict
|
||
Roll up which metrics regressed, rank by relative worsening, and name the most
|
||
likely driver (e.g., cache hit rate fell → cost per session climbed).
|
||
|
||
## Output
|
||
|
||
- A Markdown table: metric | baseline | recent | Δ | direction (▲ worse / ▼ better) | verdict.
|
||
- Tag each regressed metric 🔴 (clear regression), 🟡 (mild/within noise), or 🟢 (improved).
|
||
- Currency in USD to 4 decimals; rates as percentages to 2 decimals.
|
||
- List the specific session IDs that contributed most to any regression.
|
||
- End with the single highest-priority regression to address and a concrete next step.
|
||
- Read-only: only report what the API returns; never fabricate baselines.
|