feat: Claude Code Monitor — lanes, pipelines and a merged workspace
Internal SmartGift build of a Claude Code monitoring dashboard. Lanes: a durable unit of parallel agent work, one per working directory, tracked across session restarts. Managed lanes are git worktrees the dashboard provisions and can reset or remove behind a three-check destroy guard and a counted preflight; adopted lanes are directories you already own and are never destroyable. Pipelines: a lane moves through pipeline stages. A stage the agent declares with evidence renders green; a stage inferred from the tool-event stream renders dashed amber and never counts as done. Detection is forward-only within a 30-minute window, and never writes the declared stage. Workspace: one page at /run with a lane grid, the selected lane's pipeline, and a full Claude console behind a disclosure.
This commit is contained in:
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"name": "ccam-quality",
|
||||
"description": "Reliability and error monitoring for Claude Code sessions — surfaces APIError events, hook delivery failures, tool-failure ratios (PreToolUse/PostToolUse gaps), and SLO tracking with error-budget reporting, all derived from the Claude Code Agent Monitor event stream at http://localhost:4820.",
|
||||
"version": "1.0.0",
|
||||
"author": {
|
||||
"name": "Nguyễn Ngọc Trí Vĩ",
|
||||
"url": "https://git.smartgift.io.vn/Smartgift-AI"
|
||||
},
|
||||
"homepage": "https://git.smartgift.io.vn/Smartgift-AI/Claude-Code-Monitor",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://git.smartgift.io.vn/Smartgift-AI/Claude-Code-Monitor.git"
|
||||
},
|
||||
"license": "MIT",
|
||||
"keywords": [
|
||||
"reliability",
|
||||
"errors",
|
||||
"slo",
|
||||
"quality",
|
||||
"monitoring",
|
||||
"error-budget",
|
||||
"claude-code"
|
||||
],
|
||||
"categories": ["reliability", "monitoring", "quality"]
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: reliability-engineer
|
||||
description: >
|
||||
Site-reliability-style agent that treats Claude Code usage as a service. It
|
||||
tracks an error budget, finds the tools and models that fail most, audits hook
|
||||
delivery health (PreToolUse vs PostToolUse gaps, missing Stop/SubagentStop),
|
||||
and reports SLO compliance — completion rate, tool success rate, and error
|
||||
rate — using the Agent Monitor event stream and analytics.
|
||||
model: sonnet
|
||||
tools:
|
||||
- Bash
|
||||
- Read
|
||||
- Grep
|
||||
---
|
||||
|
||||
# Reliability Engineer
|
||||
|
||||
You are a site reliability engineer for Claude Code. You treat each session as a
|
||||
service request and the fleet of sessions as a service. You query the Agent
|
||||
Monitor dashboard API at `http://localhost:4820` with
|
||||
`curl -s http://localhost:4820/api/...` and produce data-backed reliability
|
||||
reports: error budget, failing tools/models, hook health, and SLO compliance.
|
||||
|
||||
## Available Data Sources
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `/api/stats` | total_sessions, active_sessions, active_agents, total_agents, total_events, events_today, agents_by_status, sessions_by_status |
|
||||
| `/api/analytics` | `event_types` (counts per type incl. PreToolUse, PostToolUse, Stop, SubagentStop, APIError, Compaction), `tool_usage` (top 20), `daily_events` (365d), `daily_sessions` (365d), `sessions_by_status`, `agents_by_status`, `avg_events_per_session`, `total_subagents` |
|
||||
| `/api/events?session_id=X` | Event stream: `event_type`, `tool_name`, `summary`, `data`, `timestamp` — used to localize `APIError` and missing `PostToolUse` to specific sessions/tools |
|
||||
| `/api/events/facets` | Distinct facet values (event types, tools) for filtering |
|
||||
| `/api/sessions?limit=N` | Sessions with `status`, `model`, `started_at`, `ended_at` — completion accounting and per-model attribution |
|
||||
| `/api/alerts` , `/api/alerts/rules` | Fired alerts and configured alert rules — confirm whether reliability problems are already alerting |
|
||||
|
||||
## Key Reliability Metrics
|
||||
|
||||
- **Tool success rate** = `PostToolUse / PreToolUse` (from `event_types`). Should be ~1.0; a gap means tools that started but never reported completion (failed tools).
|
||||
- **Error rate** = `APIError / total_events`.
|
||||
- **Completion rate** = completed sessions / total sessions, from `sessions_by_status` (treat `active`/`running` as in-flight, not failures).
|
||||
- **Hook balance** = compare `Stop` + `SubagentStop` counts against session/subagent counts; missing terminators indicate dropped hook delivery.
|
||||
- **Error budget** = `1 − SLO_target`. If the SLO target is 99% success and the observed success rate is 97.3%, the budget is 1% and you are 2.7× over budget — report budget remaining as `(observed − target) / (1 − target)`.
|
||||
|
||||
## Analysis Framework
|
||||
|
||||
1. **Service overview** — pull `/api/stats` and `/api/analytics` for the current event-type distribution and session statuses.
|
||||
2. **Error budget** — compute error rate and tool success rate; compare to SLO targets (default 99% tool success, ≤1% error rate, ≥95% completion unless the user gives targets) and report budget remaining.
|
||||
3. **Failing tools/models** — rank tools by the PreToolUse→PostToolUse gap (largest gap = most failures); attribute `APIError` events to models via `/api/sessions` joined on `session_id`.
|
||||
4. **Hook health** — flag PreToolUse/PostToolUse imbalance, missing Stop/SubagentStop terminators, and stale ingestion (no recent events in `events_today`/`daily_events`).
|
||||
5. **Verdict + remediation** — OK / DEGRADED / FAILING with the single highest-impact fix.
|
||||
|
||||
## Output Standards
|
||||
|
||||
- Most important finding first; lead with the SLO verdict.
|
||||
- Cite real numbers from the API for every claim (exact counts, exact field names).
|
||||
- Rates as percentages to 2 decimals; any currency in USD to 4 decimals.
|
||||
- Use ▲ (worse) / ▼ (better) for deltas vs prior period.
|
||||
- Name the specific tools, models, and session IDs that drive each failure.
|
||||
- End with a prioritized action list (max 5 items).
|
||||
|
||||
## Constraints
|
||||
|
||||
- Read-only advisory role — never modify data.
|
||||
- Only use data returned by the API — never fabricate metrics or invent baselines.
|
||||
- If the dashboard is unreachable, tell the user to start it with `npm start` from the repo root.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
description: List the most recent APIError events with their session and a summary
|
||||
argument-hint: "[N]"
|
||||
---
|
||||
|
||||
List the most recent Claude Code errors from the Agent Monitor dashboard at
|
||||
`http://localhost:4820`. `$ARGUMENTS` is the number of errors to show (default 10).
|
||||
|
||||
1. Pull recent events and keep only API errors:
|
||||
```bash
|
||||
curl -s http://localhost:4820/api/events?limit=300
|
||||
```
|
||||
Filter the array to `event_type == "APIError"`. (If `$ARGUMENTS` is a number,
|
||||
show that many; otherwise show 10.) If none are found, also check
|
||||
`curl -s http://localhost:4820/api/analytics` `event_types.APIError` to confirm
|
||||
the true total and say "no recent APIError events (N total all-time)".
|
||||
|
||||
2. For each error, newest first, print one line:
|
||||
`timestamp · session_id · summary` — using the `timestamp`, `session_id`, and
|
||||
`summary` fields exactly as returned (trim long summaries to ~100 chars).
|
||||
|
||||
3. End with a one-line tally: total APIErrors shown and the most-affected
|
||||
`session_id`.
|
||||
|
||||
Output rules: cite only fields the API returned — never fabricate an error or a
|
||||
cause. Keep it to the list plus the tally; no extra prose. If `curl` cannot reach
|
||||
`http://localhost:4820`, tell the user to start the dashboard with `npm start`
|
||||
from the repo root.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
description: One-line reliability verdict (OK / DEGRADED / FAILING) for Claude Code usage
|
||||
---
|
||||
|
||||
Print a single reliability verdict for Claude Code usage from the Agent Monitor
|
||||
dashboard at `http://localhost:4820`. No arguments.
|
||||
|
||||
1. Fetch state:
|
||||
```bash
|
||||
curl -s http://localhost:4820/api/stats
|
||||
curl -s http://localhost:4820/api/analytics
|
||||
```
|
||||
From `stats`: `total_events`, `events_today`, `sessions_by_status`. From
|
||||
`analytics`: `event_types` (PreToolUse, PostToolUse, APIError).
|
||||
|
||||
2. Derive two reliability signals:
|
||||
- **Error rate** = `APIError / total_events` (percentage, 2 decimals).
|
||||
- **Tool-failure rate** = `(PreToolUse − PostToolUse) / PreToolUse` (percentage).
|
||||
|
||||
3. Pick the verdict from the worse of the two signals:
|
||||
- **OK** — error rate ≤ 1% and tool-failure rate ≤ 1%.
|
||||
- **DEGRADED** — either is in 1–5%.
|
||||
- **FAILING** — either exceeds 5%.
|
||||
|
||||
4. Print exactly one line:
|
||||
`Reliability: <OK|DEGRADED|FAILING> — errors X.XX%, tool failures Y.YY% (N events)`
|
||||
Prefix with ✅ (OK), ⚠️ (DEGRADED), or ❌ (FAILING).
|
||||
|
||||
Output rules: cite only fields the API returned — never fabricate. One line only;
|
||||
no extra prose. If `curl` cannot reach `http://localhost:4820`, print
|
||||
`Reliability: UNKNOWN — dashboard unreachable; start it with \`npm start\` from the repo root.`
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
description: Print a quick SLO snapshot — completion rate, tool success rate, and error rate
|
||||
---
|
||||
|
||||
Print a compact SLO snapshot for Claude Code usage from the Agent Monitor
|
||||
dashboard at `http://localhost:4820`. No arguments.
|
||||
|
||||
1. Fetch the raw counts:
|
||||
```bash
|
||||
curl -s http://localhost:4820/api/analytics
|
||||
curl -s http://localhost:4820/api/stats
|
||||
```
|
||||
From `analytics`: `event_types` (PreToolUse, PostToolUse, APIError) and
|
||||
`sessions_by_status`. From `stats`: `total_events`, `total_sessions`.
|
||||
|
||||
2. Compute three SLIs:
|
||||
- **Completion rate** = completed sessions / total sessions (from
|
||||
`sessions_by_status`; exclude still-`active`/`running` sessions from the
|
||||
denominator).
|
||||
- **Tool success rate** = `PostToolUse / PreToolUse` (cap at 100%).
|
||||
- **Error rate** = `APIError / total_events`.
|
||||
|
||||
3. Compare each to its default target (completion ≥ 95%, tool success ≥ 99%,
|
||||
error ≤ 1%) and print one line per SLI:
|
||||
`SLI .......... observed% (target X%) ✅ MET | ❌ BREACHED`
|
||||
|
||||
Output rules: rates as percentages to 2 decimals; cite only fields the API
|
||||
returned — never fabricate. End with one verdict line (e.g.,
|
||||
`SLOs: 3/3 met` or `SLOs: error rate BREACHED`). Keep it to the snapshot only; no
|
||||
extra prose. If `curl` cannot reach `http://localhost:4820`, tell the user to
|
||||
start the dashboard with `npm start` from the repo root.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
description: >
|
||||
Produce a detailed report on APIError events from Agent Monitor data — counts
|
||||
over time, which sessions and models are affected, and the likely root cause
|
||||
(rate limits, overload/529, or context-window pressure) inferred from each
|
||||
event's summary and data payload. Use when API errors spike or when you need to
|
||||
explain why requests are failing.
|
||||
---
|
||||
|
||||
# API Error Report
|
||||
|
||||
Drill into `APIError` events: how many, when, where, and most likely why.
|
||||
|
||||
## Input
|
||||
|
||||
The user provides: **$ARGUMENTS**
|
||||
|
||||
This may be:
|
||||
- empty or "all" — report on every APIError in the recent window (default)
|
||||
- a session ID — report APIErrors for that one session only
|
||||
- a window like "today" or "last 7d" — restrict the time range
|
||||
- a cause filter: "rate-limit", "overload", or "context"
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `GET /api/analytics` | `event_types` (total `APIError` count), `daily_events` (365d) — APIError volume and trend over time |
|
||||
| `GET /api/events?session_id=X` | Per-session event stream — each `APIError` carries `summary`, `data`, and `timestamp` used to classify the cause |
|
||||
| `GET /api/sessions?limit=N` | Sessions with `id`, `model`, `started_at` — attribute each error to a model and place it on the timeline |
|
||||
|
||||
## Report Sections
|
||||
|
||||
### 1. Volume & Trend
|
||||
From `GET /api/analytics`: total `APIError` count and its share of `total_events`. Use `daily_events` to chart APIErrors over the requested window and flag any day that spikes above the window mean.
|
||||
|
||||
### 2. Affected Sessions & Models
|
||||
For each session in scope, pull `GET /api/events?session_id=X` and collect `APIError` events. Group by `session_id` and, via `GET /api/sessions`, by `model`. Report the top affected sessions and which model accounts for the most errors.
|
||||
|
||||
### 3. Likely Cause Classification
|
||||
Inspect each error's `summary`/`data` and bucket it:
|
||||
- **Rate limit** — mentions 429, "rate limit", "quota", or retry-after.
|
||||
- **Overload** — mentions 529, "overloaded", or capacity.
|
||||
- **Context** — mentions context length, token limit, or "too long" (correlate with nearby `Compaction` events).
|
||||
- **Other** — anything else; quote the `summary`.
|
||||
Report the count and percentage in each bucket.
|
||||
|
||||
### 4. Timeline
|
||||
List the most recent APIErrors with `timestamp`, `session_id`, `model`, classified cause, and a one-line `summary` excerpt.
|
||||
|
||||
## Output
|
||||
|
||||
- A Markdown table per section (volume, by model, by cause).
|
||||
- Rates as percentages to 2 decimals; any currency in USD to 4 decimals.
|
||||
- Cite exact `session_id`, `model`, `timestamp`, and `summary` values — never invent a cause not supported by the payload; bucket as "Other" when unclear.
|
||||
- End with the dominant cause and a concrete mitigation (e.g., back off and retry on 529, reduce context to cut context errors, slow request rate on 429).
|
||||
- Read-only: only report what the API returns. If `curl` cannot reach `http://localhost:4820`, tell the user to start the dashboard with `npm start` from the repo root.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
description: >
|
||||
Scan recent Claude Code activity for errors and failure signals across all
|
||||
sessions using Agent Monitor data — APIError events and PreToolUse→PostToolUse
|
||||
gaps (tools that started but never completed) — then group failures by tool and
|
||||
model and rank them by frequency. Use when checking for errors or asking
|
||||
"what's failing right now".
|
||||
---
|
||||
|
||||
# Error Scan
|
||||
|
||||
Sweep recent events across sessions for error and failure signals, then rank them
|
||||
by how often they occur and which tool or model produced them.
|
||||
|
||||
## Input
|
||||
|
||||
The user provides: **$ARGUMENTS**
|
||||
|
||||
This may be:
|
||||
- empty or "all" — scan every failure signal (default)
|
||||
- "api" — APIError events only
|
||||
- "tools" — tool-failure gaps only
|
||||
- a number N — limit the scan to the most recent N sessions
|
||||
- a session ID — scan a single session
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `GET /api/analytics` | `event_types` (counts per type incl. PreToolUse, PostToolUse, APIError), `tool_usage` (top 20), `daily_events` (365d) — fleet-wide failure baseline |
|
||||
| `GET /api/events?session_id=X` | Per-session event stream: `event_type`, `tool_name`, `summary`, `data`, `timestamp` — locate `APIError` and unmatched `PreToolUse` |
|
||||
| `GET /api/sessions?limit=N` | Sessions with `id`, `status`, `model`, `started_at` — pick the recent window and attribute failures to a model |
|
||||
|
||||
## Report Sections
|
||||
|
||||
### 1. Scope
|
||||
Resolve `$ARGUMENTS` to a session set: pull `GET /api/sessions?limit=N` (default 50, ordered by `started_at`). Report how many sessions and what time span are covered.
|
||||
|
||||
### 2. Fleet Failure Counts
|
||||
From `GET /api/analytics` `event_types`, report total `APIError` count and the PreToolUse→PostToolUse gap: `gap = PreToolUse − PostToolUse` (unmatched tool starts = likely failures). State both as raw counts and as a share of `total_events`.
|
||||
|
||||
### 3. Group by Tool
|
||||
For each session in scope, pull `GET /api/events?session_id=X`. Match each `PreToolUse` to its following `PostToolUse` by `tool_name`; unmatched starts are failures. Aggregate failures and `APIError` events per `tool_name`. Rank tools by failure frequency (descending).
|
||||
|
||||
### 4. Group by Model
|
||||
Join failures to the owning session's `model` (from `GET /api/sessions`). Rank models by APIError count and tool-failure count.
|
||||
|
||||
### 5. Top Offenders
|
||||
List the single most failure-prone tool, the most error-prone model, and the session with the most failures, each with its exact count and one-line `summary` excerpt from a representative event.
|
||||
|
||||
## Output
|
||||
|
||||
- A ranked Markdown table: tool/model | APIError count | tool-failure (gap) count | total failures | share of events.
|
||||
- Rates as percentages to 2 decimals.
|
||||
- Cite exact `event_type`, `tool_name`, and `session_id` values — never fabricate counts.
|
||||
- End with the one failure pattern most worth investigating and a concrete next step.
|
||||
- Read-only: only report what the API returns. If `curl` cannot reach `http://localhost:4820`, tell the user to start the dashboard with `npm start` from the repo root.
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
description: >
|
||||
Audit hook delivery health from Agent Monitor data — balance PreToolUse vs
|
||||
PostToolUse (a gap means tools that started but never reported back), detect
|
||||
missing Stop/SubagentStop terminators (sessions/subagents that never closed),
|
||||
and check for stale ingestion (no recent events). Use when hooks look
|
||||
unreliable or events seem to be dropping.
|
||||
---
|
||||
|
||||
# Hook Failure Audit
|
||||
|
||||
Assess whether the hook pipeline is delivering events reliably, using the event
|
||||
counts and stream the dashboard already has. This is about *delivery* health
|
||||
(missing/dropped events), not about why a model errored.
|
||||
|
||||
## Input
|
||||
|
||||
The user provides: **$ARGUMENTS**
|
||||
|
||||
This may be:
|
||||
- empty or "all" — run every check (default)
|
||||
- "balance" — PreToolUse/PostToolUse balance only
|
||||
- "terminators" — missing Stop/SubagentStop only
|
||||
- "freshness" — stale-ingestion check only
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `GET /api/analytics` | `event_types` (counts per type: PreToolUse, PostToolUse, Stop, SubagentStop, SessionStart, SessionEnd), `daily_events` (365d), `total_subagents`, `sessions_by_status` — fleet-wide delivery balance |
|
||||
| `GET /api/stats` | `total_sessions`, `total_agents`, `total_events`, `events_today` — expected terminator counts and recency |
|
||||
| `GET /api/events?session_id=X` | Per-session stream — confirm which sessions are missing a `PostToolUse`, `Stop`, or `SubagentStop` |
|
||||
|
||||
## Report Sections
|
||||
|
||||
### 1. PreToolUse / PostToolUse Balance
|
||||
From `GET /api/analytics` `event_types`: `gap = PreToolUse − PostToolUse`. A positive gap means tools whose completion hook never arrived. Report the gap as a count and as a percentage of `PreToolUse`. A healthy pipeline keeps this near 0%.
|
||||
|
||||
### 2. Missing Terminators
|
||||
Compare `Stop` count against completed sessions (`sessions_by_status`) and `SubagentStop` against `total_subagents`/`total_agents` (from `/api/stats`). A shortfall means sessions or subagents that ran but never emitted a closing hook — likely dropped delivery or a crashed handler. Report expected vs observed for each.
|
||||
|
||||
### 3. Stale Ingestion
|
||||
Check `events_today` from `/api/stats` and the tail of `daily_events` from analytics. If recent days are empty while sessions exist, ingestion has stalled. Report the most recent day with events and how long ago that was.
|
||||
|
||||
### 4. Localize
|
||||
For the sessions with the largest gaps or missing terminators, pull `GET /api/events?session_id=X` and confirm which specific hook types are absent. List the offending session IDs.
|
||||
|
||||
## Output
|
||||
|
||||
- A check-by-check report with a PASS / WARN / FAIL marker each (✅ / ⚠️ / ❌) and the expected-vs-observed numbers.
|
||||
- Rates as percentages to 2 decimals.
|
||||
- Cite exact `event_type` counts and `session_id` values — never fabricate.
|
||||
- End with an overall verdict (e.g., "4/4 checks passed" or "hook delivery DEGRADED") and the single highest-impact remediation (e.g., reinstall hooks via the dashboard Settings, or restart the server with `npm start`).
|
||||
- Read-only: only report what the API returns. If `curl` cannot reach `http://localhost:4820`, tell the user to start the dashboard with `npm start` from the repo root.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
description: >
|
||||
Compare this period's reliability against the prior period using Agent Monitor
|
||||
data — error rate (APIError/total) and tool-failure rate (PreToolUse→PostToolUse
|
||||
gap) — flag any regression where reliability got worse, and optionally wire a
|
||||
persistent alert rule so the dashboard catches the next regression
|
||||
automatically. Use when checking whether reliability degraded.
|
||||
---
|
||||
|
||||
# Regression Alert
|
||||
|
||||
Detect whether Claude Code reliability is getting worse period-over-period, and
|
||||
optionally arm an alert so it never has to be checked by hand again. Scope is
|
||||
reliability/failures only — for cache/cost/compaction drift, use ccam-insights'
|
||||
`regression-watch` instead.
|
||||
|
||||
## Input
|
||||
|
||||
The user provides: **$ARGUMENTS**
|
||||
|
||||
This may be:
|
||||
- empty or "all" — check error rate and tool-failure rate (default)
|
||||
- "errors" — APIError-rate regression only
|
||||
- "tools" — tool-failure-rate regression only
|
||||
- a window like "7 vs 7" or "30 vs 30" — recent vs baseline window sizes (default: last 7 days vs the prior 7)
|
||||
- "arm" — after reporting, also create an alert rule via `POST /api/alerts/rules` (only on explicit request)
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `GET /api/analytics` | `daily_events` (365d), `daily_sessions` (365d), `event_types` — split into recent vs baseline windows to compute per-window failure rates |
|
||||
| `GET /api/events?session_id=X` | Per-session stream — localize a regression to the sessions driving it |
|
||||
| `GET /api/alerts/rules` | Existing alert rules — check whether a matching reliability rule already exists before arming a new one |
|
||||
| `POST /api/alerts/rules` | Create a new alert rule (only when the user says "arm") |
|
||||
|
||||
## Report Sections
|
||||
|
||||
### 1. Windowing
|
||||
Split history into a **recent window** (newer) and a **baseline window** (the equal-length period just before it). Default: recent = last 7 days, baseline = the prior 7. Use `daily_events`/`daily_sessions` to bucket counts by day.
|
||||
|
||||
### 2. Error-Rate Regression
|
||||
- Per window: `error rate = APIError count / total events`.
|
||||
- Compare recent vs baseline. Flag if recent is higher. Report absolute change (pp) and relative change (%), plus the recent sessions contributing the most `APIError` events.
|
||||
|
||||
### 3. Tool-Failure-Rate Regression
|
||||
- Per window: `tool-failure rate = (PreToolUse − PostToolUse) / PreToolUse`.
|
||||
- Compare recent vs baseline. Flag a rising rate as a reliability regression. Name the tools whose gap grew most.
|
||||
|
||||
### 4. Verdict
|
||||
Roll up which rates regressed, rank by relative worsening, and name the most likely driver.
|
||||
|
||||
### 5. Optional — Arm an Alert
|
||||
**Only if the user passed "arm".** First `GET /api/alerts/rules` to avoid duplicates. Then `POST /api/alerts/rules` with a rule that fires when the regressed metric crosses a threshold near the recent value (e.g., error rate > recent rate). Echo the created rule back; do not create webhooks or fire alerts.
|
||||
|
||||
## Output
|
||||
|
||||
- A Markdown table: metric | baseline | recent | Δ (pp) | Δ (%) | direction (▲ worse / ▼ better) | verdict.
|
||||
- Tag each metric 🔴 (clear regression), 🟡 (within noise), or 🟢 (improved).
|
||||
- Rates as percentages to 2 decimals; any currency in USD to 4 decimals.
|
||||
- List the specific session IDs that contributed most to any regression.
|
||||
- End with the single highest-priority regression and a concrete next step (and, if armed, the new rule's id/threshold).
|
||||
- Read-only **except** the explicit "arm" path, which is the only write. Never mutate alert rules otherwise. If `curl` cannot reach `http://localhost:4820`, tell the user to start the dashboard with `npm start` from the repo root.
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
description: >
|
||||
Define and check simple service-level objectives for Claude Code from Agent
|
||||
Monitor data — session completion rate, tool success rate
|
||||
(PostToolUse/PreToolUse), and error rate (APIError/total) — then compare each
|
||||
to its target and report the error budget remaining. Use when reporting
|
||||
reliability or when someone asks "are we meeting our SLOs?".
|
||||
---
|
||||
|
||||
# SLO Check
|
||||
|
||||
Turn raw event counts into a clear SLO scorecard with error-budget accounting.
|
||||
|
||||
## Input
|
||||
|
||||
The user provides: **$ARGUMENTS**
|
||||
|
||||
This may be:
|
||||
- empty — use the default SLO targets below over all available data
|
||||
- targets like "completion=95 success=99 error=1" — override the defaults (percentages)
|
||||
- a window like "last 7d" or "today" — restrict the measurement period
|
||||
|
||||
Default SLO targets: completion rate ≥ 95%, tool success rate ≥ 99%, error rate ≤ 1%.
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `GET /api/analytics` | `event_types` (PreToolUse, PostToolUse, APIError counts), `sessions_by_status`, `daily_events` (365d), `daily_sessions` (365d) — the raw numerators/denominators for every SLI |
|
||||
| `GET /api/stats` | `total_sessions`, `total_events`, `events_today`, `sessions_by_status` — fleet totals and recency |
|
||||
| `GET /api/events?session_id=X` | Per-session stream — drill into the sessions that breach an SLO |
|
||||
|
||||
## Report Sections
|
||||
|
||||
### 1. Service Level Indicators (SLIs)
|
||||
Compute each SLI from `GET /api/analytics` / `GET /api/stats`:
|
||||
- **Completion rate** = completed sessions / total sessions (from `sessions_by_status`; count `active`/`running` as in-flight, exclude them from the denominator if still open).
|
||||
- **Tool success rate** = `PostToolUse / PreToolUse` (capped at 100%).
|
||||
- **Error rate** = `APIError / total_events`.
|
||||
Within a window, derive the numerators/denominators from `daily_events` / `daily_sessions`.
|
||||
|
||||
### 2. SLO Scorecard
|
||||
For each SLI, compare to its target and mark MET ✅ or BREACHED ❌.
|
||||
|
||||
### 3. Error Budget
|
||||
For each objective, report the **error budget** and how much remains:
|
||||
- Budget = `1 − target` (e.g., 1% for a 99% target).
|
||||
- For "higher-is-better" SLOs (completion, success): remaining = `(observed − target) / (1 − target)`.
|
||||
- For "lower-is-better" SLOs (error rate): remaining = `(target − observed) / target`.
|
||||
- A negative result means the budget is exhausted — report how far over (e.g., "2.7× over budget").
|
||||
|
||||
### 4. Breach Drill-Down
|
||||
For any breached SLO, list the sessions contributing most to the breach (most failed tools or most APIErrors) via `GET /api/events?session_id=X`.
|
||||
|
||||
## Output
|
||||
|
||||
- A Markdown scorecard table: SLI | observed | target | status | error budget remaining.
|
||||
- Rates as percentages to 2 decimals; any currency in USD to 4 decimals.
|
||||
- Cite exact counts and `session_id` values — never fabricate numerators or denominators.
|
||||
- End with the SLO most at risk and the single action that would recover the most budget.
|
||||
- Read-only: only report what the API returns. If `curl` cannot reach `http://localhost:4820`, tell the user to start the dashboard with `npm start` from the repo root.
|
||||
Reference in New Issue
Block a user