feat: Claude Code Monitor — lanes, pipelines and a merged workspace
Internal SmartGift build of a Claude Code monitoring dashboard. Lanes: a durable unit of parallel agent work, one per working directory, tracked across session restarts. Managed lanes are git worktrees the dashboard provisions and can reset or remove behind a three-check destroy guard and a counted preflight; adopted lanes are directories you already own and are never destroyable. Pipelines: a lane moves through pipeline stages. A stage the agent declares with evidence renders green; a stage inferred from the tool-event stream renders dashed amber and never counts as done. Detection is forward-only within a 30-minute window, and never writes the declared stage. Workspace: one page at /run with a lane grid, the selected lane's pipeline, and a full Claude console behind a disclosure.
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
# Coralogix Integration
|
||||
|
||||
Full-stack observability for Claude Code Agent Monitor via [Coralogix](https://coralogix.com) — logs, metrics, traces, and SLO tracking through a single platform.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Kubernetes Cluster"
|
||||
APP["Agent Monitor Pods"]
|
||||
MCP["MCP Sidecar"]
|
||||
OTEL["OTel Collector<br/>(DaemonSet)"]
|
||||
end
|
||||
|
||||
APP -->|"metrics + logs"| OTEL
|
||||
MCP -->|"metrics + logs"| OTEL
|
||||
|
||||
OTEL -->|"OTLP (gRPC)"| CX["Coralogix Platform"]
|
||||
|
||||
subgraph "Coralogix"
|
||||
CX --> LOGS["Log Analytics<br/>DataPrime Queries"]
|
||||
CX --> MET["Metrics<br/>PromQL + Recording Rules"]
|
||||
CX --> TRACE["Distributed Tracing"]
|
||||
CX --> ALERT["Alert Engine"]
|
||||
CX --> DASH["Custom Dashboards"]
|
||||
CX --> SLO["SLO Management"]
|
||||
end
|
||||
|
||||
ALERT -->|"Critical"| PD["PagerDuty"]
|
||||
ALERT -->|"Warning"| SLACK["Slack"]
|
||||
|
||||
style OTEL fill:#4f46e5,color:#fff
|
||||
style CX fill:#1a1a2e,color:#fff
|
||||
style LOGS fill:#7c3aed,color:#fff
|
||||
style MET fill:#e6522c,color:#fff
|
||||
style TRACE fill:#059669,color:#fff
|
||||
style ALERT fill:#dc2626,color:#fff
|
||||
style DASH fill:#f46800,color:#fff
|
||||
style SLO fill:#0ea5e9,color:#fff
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `values.yaml` | Helm values for Coralogix OpenTelemetry Collector |
|
||||
| `alerts.yaml` | Alert definitions (mirrors Prometheus/Alertmanager rules) |
|
||||
| `dashboards.yaml` | Custom dashboard with 6 rows, 18 panels, SLO tracking |
|
||||
| `coralogix-terraform.tf` | Terraform-managed alerts, parsing rules, recording rules |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Add the Helm Repository
|
||||
|
||||
```bash
|
||||
helm repo add coralogix https://cgx.jfrog.io/artifactory/coralogix-charts-virtual
|
||||
helm repo update
|
||||
```
|
||||
|
||||
### 2. Create the API Key Secret
|
||||
|
||||
```bash
|
||||
kubectl create secret generic coralogix-keys \
|
||||
--namespace agent-monitor \
|
||||
--from-literal=PRIVATE_KEY=<YOUR_CORALOGIX_SEND_YOUR_DATA_KEY>
|
||||
```
|
||||
|
||||
### 3. Deploy the OTel Collector
|
||||
|
||||
```bash
|
||||
helm install coralogix-otel coralogix/opentelemetry \
|
||||
--namespace agent-monitor \
|
||||
-f deployments/monitoring/coralogix/values.yaml
|
||||
```
|
||||
|
||||
### 4. Import the Dashboard
|
||||
|
||||
Upload `dashboards.yaml` via the Coralogix UI:
|
||||
|
||||
**Dashboards → Custom Dashboards → Import**
|
||||
|
||||
### 5. (Optional) Terraform-managed Alerts
|
||||
|
||||
```bash
|
||||
cd deployments/monitoring/coralogix
|
||||
export CORALOGIX_API_KEY="<your-key>"
|
||||
export CORALOGIX_ENV="coralogix.com"
|
||||
terraform init
|
||||
terraform apply
|
||||
```
|
||||
|
||||
## What Gets Collected
|
||||
|
||||
| Signal | Source | Destination |
|
||||
|--------|--------|-------------|
|
||||
| **Logs** | Pod stdout/stderr (JSON structured) | Coralogix Log Analytics |
|
||||
| **Metrics** | Prometheus scrape (`/api/health`) | Coralogix Metrics |
|
||||
| **K8s Metrics** | kubelet, cAdvisor, host metrics | Coralogix Metrics |
|
||||
| **Traces** | OTLP from application (if instrumented) | Coralogix Tracing |
|
||||
|
||||
## Alert Parity
|
||||
|
||||
All 10 Prometheus/Alertmanager rules are replicated in Coralogix:
|
||||
|
||||
| Alert | Severity | Prometheus | Coralogix |
|
||||
|-------|----------|:----------:|:---------:|
|
||||
| Instance Down | Critical | ✓ | ✓ |
|
||||
| High Error Rate | Critical | ✓ | ✓ |
|
||||
| Pod Restart Loop | Critical | ✓ | ✓ |
|
||||
| PV Nearly Full | Critical | ✓ | ✓ |
|
||||
| High Latency | Warning | ✓ | ✓ |
|
||||
| WebSocket Spike | Warning | ✓ | ✓ |
|
||||
| High Memory | Warning | ✓ | ✓ |
|
||||
| High CPU | Warning | ✓ | ✓ |
|
||||
| HPA Maxed Out | Warning | ✓ | ✓ |
|
||||
| Slow DB Queries | Warning | ✓ | ✓ |
|
||||
|
||||
## Dashboard Panels
|
||||
|
||||
The custom dashboard provides 18 panels across 6 rows:
|
||||
|
||||
1. **Overview** — Active sessions, request rate, WebSocket connections
|
||||
2. **HTTP Performance** — Latency distribution, error rate, status codes
|
||||
3. **Application Logs** — Error log stream (DataPrime), log volume by severity, hook throughput
|
||||
4. **Infrastructure** — CPU, memory, pod status
|
||||
5. **Database & Storage** — SQLite query duration, PV usage, network I/O
|
||||
6. **SLO Tracking** — Availability SLO (99.9%), latency SLO (P95 < 500ms), error budget burn
|
||||
|
||||
## Coralogix Regions
|
||||
|
||||
Set `global.domain` in `values.yaml` to match your Coralogix region:
|
||||
|
||||
| Region | Domain |
|
||||
|--------|--------|
|
||||
| US1 | `coralogix.us` |
|
||||
| US2 | `cx498.coralogix.com` |
|
||||
| EU1 | `coralogix.com` |
|
||||
| EU2 | `eu2.coralogix.com` |
|
||||
| AP1 (India) | `coralogix.in` |
|
||||
| AP2 (Singapore) | `coralogix.sg` |
|
||||
@@ -0,0 +1,194 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Coralogix Alert Definitions for Claude Code Agent Monitor
|
||||
#
|
||||
# These alerts can be imported via the Coralogix Alerts API or Terraform
|
||||
# provider (coralogix/coralogix). They mirror the Prometheus/Alertmanager
|
||||
# rules in ../prometheus/rules/ for consistency across observability stacks.
|
||||
#
|
||||
# API import:
|
||||
# curl -X POST "https://api.coralogix.com/api/v1/external/alerts" \
|
||||
# -H "Authorization: Bearer $CORALOGIX_API_KEY" \
|
||||
# -H "Content-Type: application/json" \
|
||||
# -d @deployments/monitoring/coralogix/alerts.yaml
|
||||
#
|
||||
# Terraform:
|
||||
# See the coralogix_alert resources in coralogix-terraform.tf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
alerts:
|
||||
# ── Critical ───────────────────────────────────────────────────────────────
|
||||
|
||||
- name: "Agent Monitor Down"
|
||||
description: "No metrics received from agent-monitor pods for > 2 minutes"
|
||||
severity: critical
|
||||
type: metric
|
||||
condition:
|
||||
metric_name: "up"
|
||||
filter:
|
||||
job: "agent-monitor"
|
||||
threshold: 1
|
||||
comparison: less_than
|
||||
for_duration: "2m"
|
||||
of_last: "5m"
|
||||
notifications:
|
||||
- integration: pagerduty
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-critical"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
team: platform
|
||||
|
||||
- name: "High Error Rate"
|
||||
description: "5xx error rate exceeds 5% of total requests for 5 minutes"
|
||||
severity: critical
|
||||
type: ratio
|
||||
condition:
|
||||
numerator:
|
||||
query: 'http_requests_total{job="agent-monitor", status=~"5.."}'
|
||||
denominator:
|
||||
query: 'http_requests_total{job="agent-monitor"}'
|
||||
threshold: 0.05
|
||||
comparison: greater_than
|
||||
for_duration: "5m"
|
||||
notifications:
|
||||
- integration: pagerduty
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-critical"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "Pod Restart Loop"
|
||||
description: "Agent Monitor pod has restarted > 5 times in 15 minutes"
|
||||
severity: critical
|
||||
type: metric
|
||||
condition:
|
||||
query: 'increase(kube_pod_container_status_restarts_total{namespace=~"agent-monitor.*", container="agent-monitor"}[15m])'
|
||||
threshold: 5
|
||||
comparison: greater_than
|
||||
for_duration: "1m"
|
||||
notifications:
|
||||
- integration: pagerduty
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-critical"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "Persistent Volume Nearly Full"
|
||||
description: "SQLite persistent volume is > 90% full"
|
||||
severity: critical
|
||||
type: metric
|
||||
condition:
|
||||
query: '(kubelet_volume_stats_used_bytes{namespace=~"agent-monitor.*"} / kubelet_volume_stats_capacity_bytes{namespace=~"agent-monitor.*"}) * 100'
|
||||
threshold: 90
|
||||
comparison: greater_than
|
||||
for_duration: "5m"
|
||||
notifications:
|
||||
- integration: pagerduty
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-critical"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
# ── Warning ────────────────────────────────────────────────────────────────
|
||||
|
||||
- name: "High Latency"
|
||||
description: "P95 request latency exceeds 2 seconds for 5 minutes"
|
||||
severity: warning
|
||||
type: metric
|
||||
condition:
|
||||
query: 'histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="agent-monitor"}[5m]))'
|
||||
threshold: 2
|
||||
comparison: greater_than
|
||||
for_duration: "5m"
|
||||
notifications:
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-alerts"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "WebSocket Connection Spike"
|
||||
description: "Active WebSocket connections exceed 1000"
|
||||
severity: warning
|
||||
type: metric
|
||||
condition:
|
||||
metric_name: "websocket_connections_active"
|
||||
filter:
|
||||
job: "agent-monitor"
|
||||
threshold: 1000
|
||||
comparison: greater_than
|
||||
for_duration: "2m"
|
||||
notifications:
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-alerts"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "High Memory Usage"
|
||||
description: "Container memory usage exceeds 85% of limit"
|
||||
severity: warning
|
||||
type: metric
|
||||
condition:
|
||||
query: '(container_memory_working_set_bytes{namespace=~"agent-monitor.*", container="agent-monitor"} / container_spec_memory_limit_bytes{namespace=~"agent-monitor.*", container="agent-monitor"}) * 100'
|
||||
threshold: 85
|
||||
comparison: greater_than
|
||||
for_duration: "5m"
|
||||
notifications:
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-alerts"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "High CPU Usage"
|
||||
description: "Container CPU usage exceeds 80% for 10 minutes"
|
||||
severity: warning
|
||||
type: metric
|
||||
condition:
|
||||
query: '(rate(container_cpu_usage_seconds_total{namespace=~"agent-monitor.*", container="agent-monitor"}[5m]) / container_spec_cpu_quota{namespace=~"agent-monitor.*", container="agent-monitor"} * 100000)'
|
||||
threshold: 80
|
||||
comparison: greater_than
|
||||
for_duration: "10m"
|
||||
notifications:
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-alerts"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "HPA Maxed Out"
|
||||
description: "HPA replicas at max for 15 minutes — may need capacity increase"
|
||||
severity: warning
|
||||
type: metric
|
||||
condition:
|
||||
query: 'kube_horizontalpodautoscaler_status_current_replicas{namespace=~"agent-monitor.*"} == kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"agent-monitor.*"}'
|
||||
threshold: 1
|
||||
comparison: greater_than_or_equal
|
||||
for_duration: "15m"
|
||||
notifications:
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-alerts"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
- name: "Slow Database Queries"
|
||||
description: "SQLite query duration exceeds 1 second"
|
||||
severity: warning
|
||||
type: metric
|
||||
condition:
|
||||
metric_name: "sqlite_query_duration_seconds"
|
||||
filter:
|
||||
job: "agent-monitor"
|
||||
threshold: 1
|
||||
comparison: greater_than
|
||||
for_duration: "5m"
|
||||
notifications:
|
||||
- integration: slack
|
||||
channel: "#agent-monitor-alerts"
|
||||
labels:
|
||||
service: agent-monitor
|
||||
|
||||
# ── Notification integrations ────────────────────────────────────────────────
|
||||
# Configure these in Coralogix UI: Settings → Integrations → Outbound Webhooks
|
||||
#
|
||||
# Required integrations:
|
||||
# - pagerduty: PagerDuty Events API v2 routing key
|
||||
# - slack: Slack webhook for #agent-monitor-critical and #agent-monitor-alerts
|
||||
# - email: (optional) Email notification group
|
||||
@@ -0,0 +1,353 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Coralogix Terraform Integration for Claude Code Agent Monitor
|
||||
#
|
||||
# Provisions Coralogix resources via the official Terraform provider:
|
||||
# - Alert rules (mirroring Prometheus/Alertmanager rules)
|
||||
# - Log parsing rules for structured JSON ingestion
|
||||
# - Recording rules for pre-aggregated SLO metrics
|
||||
# - Dashboard provisioning
|
||||
#
|
||||
# Usage:
|
||||
# export CORALOGIX_API_KEY="<your-send-your-data-key>"
|
||||
# export CORALOGIX_ENV="<your-coralogix-domain>" # e.g. coralogix.com
|
||||
# terraform init
|
||||
# terraform plan
|
||||
# terraform apply
|
||||
#
|
||||
# Requires: hashicorp/terraform >= 1.5, coralogix/coralogix >= 1.10
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.5"
|
||||
required_providers {
|
||||
coralogix = {
|
||||
source = "coralogix/coralogix"
|
||||
version = "~> 1.10"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "coralogix" {
|
||||
# API key and environment are sourced from:
|
||||
# CORALOGIX_API_KEY – Send-Your-Data API key
|
||||
# CORALOGIX_ENV – Domain (e.g. coralogix.com, eu2.coralogix.com)
|
||||
}
|
||||
|
||||
# ── Variables ────────────────────────────────────────────────────────────────
|
||||
|
||||
variable "environment" {
|
||||
description = "Deployment environment (dev, staging, production)"
|
||||
type = string
|
||||
default = "production"
|
||||
validation {
|
||||
condition = contains(["dev", "staging", "production"], var.environment)
|
||||
error_message = "environment must be one of: dev, staging, production."
|
||||
}
|
||||
}
|
||||
|
||||
variable "notification_group_id" {
|
||||
description = "Coralogix notification group ID for alert routing"
|
||||
type = string
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "pagerduty_webhook_id" {
|
||||
description = "Coralogix outbound webhook ID for PagerDuty integration"
|
||||
type = string
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "slack_webhook_id" {
|
||||
description = "Coralogix outbound webhook ID for Slack integration"
|
||||
type = string
|
||||
default = ""
|
||||
}
|
||||
|
||||
locals {
|
||||
app_name = "agent-monitor"
|
||||
subsystem = "kubernetes"
|
||||
alert_prefix = "[Agent Monitor]"
|
||||
}
|
||||
|
||||
# ── Parsing Rules ────────────────────────────────────────────────────────────
|
||||
# Structured JSON log parsing for agent-monitor application logs
|
||||
|
||||
resource "coralogix_rules_group" "agent_monitor_parsing" {
|
||||
name = "${local.alert_prefix} Log Parsing"
|
||||
description = "Parse structured JSON logs from Agent Monitor pods"
|
||||
enabled = true
|
||||
order = 1
|
||||
|
||||
rule_subgroups {
|
||||
rules {
|
||||
name = "JSON Extract"
|
||||
description = "Extract structured fields from JSON application logs"
|
||||
source_field = "text"
|
||||
enabled = true
|
||||
|
||||
parse_json_field {
|
||||
destination_field = "json"
|
||||
keep_source_field = false
|
||||
keep_destination_field = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
rule_subgroups {
|
||||
rules {
|
||||
name = "Severity Mapping"
|
||||
description = "Map log level field to Coralogix severity"
|
||||
source_field = "json.level"
|
||||
enabled = true
|
||||
|
||||
extract {
|
||||
regexp = "(?P<severity>debug|info|warn|error|fatal)"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ── Recording Rules ──────────────────────────────────────────────────────────
|
||||
# Pre-aggregate SLO metrics for efficient dashboard queries
|
||||
|
||||
resource "coralogix_recording_rule_group_set" "slo_metrics" {
|
||||
name = "${local.alert_prefix} SLO Recording Rules"
|
||||
|
||||
groups {
|
||||
name = "agent_monitor_slo"
|
||||
interval = 60 # seconds
|
||||
|
||||
rules {
|
||||
record = "agent_monitor:http_availability:ratio_rate5m"
|
||||
expr = <<-EOT
|
||||
1 - (
|
||||
sum(rate(http_requests_total{job="agent-monitor", status=~"5.."}[5m]))
|
||||
/
|
||||
sum(rate(http_requests_total{job="agent-monitor"}[5m]))
|
||||
)
|
||||
EOT
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
|
||||
rules {
|
||||
record = "agent_monitor:http_latency_p95:seconds_rate5m"
|
||||
expr = <<-EOT
|
||||
histogram_quantile(0.95,
|
||||
sum(rate(http_request_duration_seconds_bucket{job="agent-monitor"}[5m])) by (le)
|
||||
)
|
||||
EOT
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
|
||||
rules {
|
||||
record = "agent_monitor:websocket_connections:total"
|
||||
expr = <<-EOT
|
||||
sum(websocket_connections_active{job="agent-monitor"})
|
||||
EOT
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ── Alert Rules ──────────────────────────────────────────────────────────────
|
||||
|
||||
resource "coralogix_alert" "instance_down" {
|
||||
name = "${local.alert_prefix} Instance Down"
|
||||
description = "No metrics received from agent-monitor pods for > 2 minutes"
|
||||
severity = "Critical"
|
||||
enabled = true
|
||||
|
||||
metric {
|
||||
promql {
|
||||
text = "up{job=\"agent-monitor\"} == 0"
|
||||
condition = "more_than"
|
||||
threshold = 0
|
||||
}
|
||||
duration = "2m"
|
||||
}
|
||||
|
||||
notifications_group {
|
||||
dynamic "notification" {
|
||||
for_each = var.pagerduty_webhook_id != "" ? [1] : []
|
||||
content {
|
||||
integration_id = var.pagerduty_webhook_id
|
||||
}
|
||||
}
|
||||
dynamic "notification" {
|
||||
for_each = var.slack_webhook_id != "" ? [1] : []
|
||||
content {
|
||||
integration_id = var.slack_webhook_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
team = "platform"
|
||||
}
|
||||
}
|
||||
|
||||
resource "coralogix_alert" "high_error_rate" {
|
||||
name = "${local.alert_prefix} High Error Rate"
|
||||
description = "5xx error rate exceeds 5% of total requests for 5 minutes"
|
||||
severity = "Critical"
|
||||
enabled = true
|
||||
|
||||
metric {
|
||||
promql {
|
||||
text = <<-EOT
|
||||
(
|
||||
sum(rate(http_requests_total{job="agent-monitor", status=~"5.."}[5m]))
|
||||
/
|
||||
sum(rate(http_requests_total{job="agent-monitor"}[5m]))
|
||||
) * 100 > 5
|
||||
EOT
|
||||
condition = "more_than"
|
||||
threshold = 5
|
||||
}
|
||||
duration = "5m"
|
||||
}
|
||||
|
||||
notifications_group {
|
||||
dynamic "notification" {
|
||||
for_each = var.pagerduty_webhook_id != "" ? [1] : []
|
||||
content {
|
||||
integration_id = var.pagerduty_webhook_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
|
||||
resource "coralogix_alert" "high_latency" {
|
||||
name = "${local.alert_prefix} High Latency"
|
||||
description = "P95 request latency exceeds 2 seconds for 5 minutes"
|
||||
severity = "Warning"
|
||||
enabled = true
|
||||
|
||||
metric {
|
||||
promql {
|
||||
text = "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"agent-monitor\"}[5m])) by (le)) > 2"
|
||||
condition = "more_than"
|
||||
threshold = 2
|
||||
}
|
||||
duration = "5m"
|
||||
}
|
||||
|
||||
notifications_group {
|
||||
dynamic "notification" {
|
||||
for_each = var.slack_webhook_id != "" ? [1] : []
|
||||
content {
|
||||
integration_id = var.slack_webhook_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
|
||||
resource "coralogix_alert" "high_memory" {
|
||||
name = "${local.alert_prefix} High Memory Usage"
|
||||
description = "Container memory usage exceeds 85% of limit"
|
||||
severity = "Warning"
|
||||
enabled = true
|
||||
|
||||
metric {
|
||||
promql {
|
||||
text = <<-EOT
|
||||
(
|
||||
container_memory_working_set_bytes{namespace=~"agent-monitor.*", container="agent-monitor"}
|
||||
/
|
||||
container_spec_memory_limit_bytes{namespace=~"agent-monitor.*", container="agent-monitor"}
|
||||
) * 100 > 85
|
||||
EOT
|
||||
condition = "more_than"
|
||||
threshold = 85
|
||||
}
|
||||
duration = "5m"
|
||||
}
|
||||
|
||||
notifications_group {
|
||||
dynamic "notification" {
|
||||
for_each = var.slack_webhook_id != "" ? [1] : []
|
||||
content {
|
||||
integration_id = var.slack_webhook_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
|
||||
resource "coralogix_alert" "pod_restart_loop" {
|
||||
name = "${local.alert_prefix} Pod Restart Loop"
|
||||
description = "Agent Monitor pod has restarted > 5 times in 15 minutes"
|
||||
severity = "Critical"
|
||||
enabled = true
|
||||
|
||||
metric {
|
||||
promql {
|
||||
text = "increase(kube_pod_container_status_restarts_total{namespace=~\"agent-monitor.*\", container=\"agent-monitor\"}[15m]) > 5"
|
||||
condition = "more_than"
|
||||
threshold = 5
|
||||
}
|
||||
duration = "1m"
|
||||
}
|
||||
|
||||
notifications_group {
|
||||
dynamic "notification" {
|
||||
for_each = var.pagerduty_webhook_id != "" ? [1] : []
|
||||
content {
|
||||
integration_id = var.pagerduty_webhook_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
labels = {
|
||||
service = local.app_name
|
||||
environment = var.environment
|
||||
}
|
||||
}
|
||||
|
||||
# ── Outputs ──────────────────────────────────────────────────────────────────
|
||||
|
||||
output "parsing_rule_group_id" {
|
||||
description = "ID of the Coralogix parsing rule group"
|
||||
value = coralogix_rules_group.agent_monitor_parsing.id
|
||||
}
|
||||
|
||||
output "recording_rule_set_id" {
|
||||
description = "ID of the Coralogix recording rule group set"
|
||||
value = coralogix_recording_rule_group_set.slo_metrics.id
|
||||
}
|
||||
|
||||
output "alert_ids" {
|
||||
description = "IDs of all provisioned Coralogix alerts"
|
||||
value = {
|
||||
instance_down = coralogix_alert.instance_down.id
|
||||
high_error_rate = coralogix_alert.high_error_rate.id
|
||||
high_latency = coralogix_alert.high_latency.id
|
||||
high_memory = coralogix_alert.high_memory.id
|
||||
pod_restart = coralogix_alert.pod_restart_loop.id
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Coralogix Custom Dashboard – Agent Monitor
|
||||
#
|
||||
# Import via Coralogix UI: Dashboards → Custom Dashboards → Import
|
||||
# Or via API:
|
||||
# curl -X POST "https://api.coralogix.com/api/v1/external/grafana/api/dashboards/db" \
|
||||
# -H "Authorization: Bearer $CORALOGIX_API_KEY" \
|
||||
# -H "Content-Type: application/json" \
|
||||
# -d @dashboards.yaml
|
||||
#
|
||||
# This dashboard mirrors the Grafana dashboard (../grafana/dashboards/) while
|
||||
# leveraging Coralogix-native features: DataPrime queries, log correlation,
|
||||
# distributed tracing waterfall, and Apdex scoring.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
dashboard:
|
||||
name: "Agent Monitor – Operations"
|
||||
description: "Claude Code Agent Monitor: real-time operations, SLOs, and infrastructure health"
|
||||
folder: "Agent Monitor"
|
||||
tags:
|
||||
- agent-monitor
|
||||
- operations
|
||||
- sre
|
||||
|
||||
# ── Row 1: Overview ──────────────────────────────────────────────────────
|
||||
rows:
|
||||
- name: "Overview"
|
||||
panels:
|
||||
- title: "Active Sessions"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: 'agent_monitor_active_sessions'
|
||||
legend: "{{namespace}}"
|
||||
span: 4
|
||||
|
||||
- title: "Request Rate (req/s)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: 'sum(rate(http_requests_total{job="agent-monitor"}[5m]))'
|
||||
legend: "Requests/sec"
|
||||
span: 4
|
||||
|
||||
- title: "WebSocket Connections"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: 'websocket_connections_active{job="agent-monitor"}'
|
||||
legend: "{{pod}}"
|
||||
span: 4
|
||||
|
||||
# ── Row 2: HTTP Performance ────────────────────────────────────────────
|
||||
- name: "HTTP Performance"
|
||||
panels:
|
||||
- title: "Latency Distribution (P50 / P95 / P99)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job="agent-monitor"}[5m])) by (le))
|
||||
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="agent-monitor"}[5m])) by (le))
|
||||
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="agent-monitor"}[5m])) by (le))
|
||||
span: 6
|
||||
|
||||
- title: "Error Rate (%)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
sum(rate(http_requests_total{job="agent-monitor", status=~"5.."}[5m]))
|
||||
/
|
||||
sum(rate(http_requests_total{job="agent-monitor"}[5m])) * 100
|
||||
legend: "5xx %"
|
||||
thresholds:
|
||||
- value: 1
|
||||
color: yellow
|
||||
- value: 5
|
||||
color: red
|
||||
span: 3
|
||||
|
||||
- title: "Status Code Distribution"
|
||||
type: bar-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: 'sum by (status) (increase(http_requests_total{job="agent-monitor"}[1h]))'
|
||||
span: 3
|
||||
|
||||
# ── Row 3: Logs (DataPrime) ────────────────────────────────────────────
|
||||
- name: "Application Logs"
|
||||
panels:
|
||||
- title: "Error Logs"
|
||||
type: dataprime
|
||||
query: |
|
||||
source logs
|
||||
| filter $d.cx.application.name == 'agent-monitor'
|
||||
| filter $d.severity == 'ERROR' || $d.severity == 'FATAL'
|
||||
| select $m.timestamp, $d.k8s.pod.name, $d.message
|
||||
| order by $m.timestamp desc
|
||||
| limit 100
|
||||
span: 6
|
||||
|
||||
- title: "Log Volume by Severity"
|
||||
type: bar-chart
|
||||
query:
|
||||
type: dataprime
|
||||
expression: |
|
||||
source logs
|
||||
| filter $d.cx.application.name == 'agent-monitor'
|
||||
| count_group_by $d.severity as count
|
||||
span: 3
|
||||
|
||||
- title: "Hook Event Throughput"
|
||||
type: line-chart
|
||||
query:
|
||||
type: dataprime
|
||||
expression: |
|
||||
source logs
|
||||
| filter $d.cx.application.name == 'agent-monitor'
|
||||
| filter $d.message matches 'hook.*event'
|
||||
| count_per_time 1m as throughput
|
||||
span: 3
|
||||
|
||||
# ── Row 4: Infrastructure ──────────────────────────────────────────────
|
||||
- name: "Infrastructure"
|
||||
panels:
|
||||
- title: "CPU Usage (%)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
rate(container_cpu_usage_seconds_total{
|
||||
namespace=~"agent-monitor.*",
|
||||
container="agent-monitor"
|
||||
}[5m]) * 100
|
||||
legend: "{{pod}}"
|
||||
span: 4
|
||||
|
||||
- title: "Memory Usage (MiB)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
container_memory_working_set_bytes{
|
||||
namespace=~"agent-monitor.*",
|
||||
container="agent-monitor"
|
||||
} / 1024 / 1024
|
||||
legend: "{{pod}}"
|
||||
span: 4
|
||||
|
||||
- title: "Pod Status"
|
||||
type: gauge
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
count by (phase) (
|
||||
kube_pod_status_phase{namespace=~"agent-monitor.*"}
|
||||
)
|
||||
span: 4
|
||||
|
||||
# ── Row 5: Database & Storage ──────────────────────────────────────────
|
||||
- name: "Database & Storage"
|
||||
panels:
|
||||
- title: "SQLite Query Duration (ms)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: 'sqlite_query_duration_seconds{job="agent-monitor"} * 1000'
|
||||
span: 4
|
||||
|
||||
- title: "PV Usage (%)"
|
||||
type: gauge
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
(kubelet_volume_stats_used_bytes{namespace=~"agent-monitor.*"}
|
||||
/ kubelet_volume_stats_capacity_bytes{namespace=~"agent-monitor.*"}) * 100
|
||||
thresholds:
|
||||
- value: 70
|
||||
color: yellow
|
||||
- value: 90
|
||||
color: red
|
||||
span: 4
|
||||
|
||||
- title: "Network I/O (bytes/s)"
|
||||
type: line-chart
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
sum by (pod) (rate(container_network_receive_bytes_total{namespace=~"agent-monitor.*"}[5m]))
|
||||
sum by (pod) (rate(container_network_transmit_bytes_total{namespace=~"agent-monitor.*"}[5m]))
|
||||
span: 4
|
||||
|
||||
# ── Row 6: SLO Tracking ────────────────────────────────────────────────
|
||||
- name: "SLO Tracking"
|
||||
panels:
|
||||
- title: "Availability SLO (99.9% target)"
|
||||
type: gauge
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
(1 - sum(rate(http_requests_total{job="agent-monitor", status=~"5.."}[30d]))
|
||||
/ sum(rate(http_requests_total{job="agent-monitor"}[30d]))) * 100
|
||||
thresholds:
|
||||
- value: 99.9
|
||||
color: green
|
||||
- value: 99.5
|
||||
color: yellow
|
||||
- value: 99.0
|
||||
color: red
|
||||
span: 4
|
||||
|
||||
- title: "Latency SLO (P95 < 500ms)"
|
||||
type: gauge
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
histogram_quantile(0.95,
|
||||
sum(rate(http_request_duration_seconds_bucket{job="agent-monitor"}[30d])) by (le)
|
||||
) * 1000
|
||||
thresholds:
|
||||
- value: 300
|
||||
color: green
|
||||
- value: 500
|
||||
color: yellow
|
||||
- value: 1000
|
||||
color: red
|
||||
span: 4
|
||||
|
||||
- title: "Error Budget Remaining"
|
||||
type: gauge
|
||||
query:
|
||||
type: metrics
|
||||
promql: |
|
||||
(0.001 - sum(rate(http_requests_total{job="agent-monitor", status=~"5.."}[30d]))
|
||||
/ sum(rate(http_requests_total{job="agent-monitor"}[30d])))
|
||||
/ 0.001 * 100
|
||||
thresholds:
|
||||
- value: 50
|
||||
color: green
|
||||
- value: 25
|
||||
color: yellow
|
||||
- value: 0
|
||||
color: red
|
||||
span: 4
|
||||
@@ -0,0 +1,180 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Coralogix OpenTelemetry Collector – Helm Values
|
||||
#
|
||||
# Deploys the Coralogix OTel collector as a DaemonSet + Gateway for shipping
|
||||
# logs, metrics, and traces from the Agent Monitor cluster.
|
||||
#
|
||||
# Prerequisites:
|
||||
# 1. Add Coralogix Helm repo:
|
||||
# helm repo add coralogix https://cgx.jfrog.io/artifactory/coralogix-charts-virtual
|
||||
# helm repo update
|
||||
# 2. Create the API key secret:
|
||||
# kubectl create secret generic coralogix-keys \
|
||||
# --namespace agent-monitor \
|
||||
# --from-literal=PRIVATE_KEY=<YOUR_CORALOGIX_PRIVATE_KEY>
|
||||
#
|
||||
# Install:
|
||||
# helm install coralogix-otel coralogix/opentelemetry \
|
||||
# --namespace agent-monitor \
|
||||
# -f deployments/monitoring/coralogix/values.yaml
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
global:
|
||||
# Coralogix domain – set to your region's endpoint
|
||||
# Options: coralogix.com | eu2.coralogix.com | coralogix.in | coralogix.us |
|
||||
# cx498.coralogix.com | coralogix.eu | coralogix.sg
|
||||
domain: "coralogix.com"
|
||||
|
||||
# Reference the API key from the pre-created K8s secret
|
||||
clusterName: "agent-monitor"
|
||||
|
||||
# ── Secret reference ─────────────────────────────────────────────────────────
|
||||
secret:
|
||||
enabled: true
|
||||
name: "coralogix-keys"
|
||||
# Key in the secret containing the Coralogix Send-Your-Data API key
|
||||
privateKeySecretRef:
|
||||
key: "PRIVATE_KEY"
|
||||
|
||||
# ── Collector – DaemonSet mode (node-level collection) ───────────────────────
|
||||
opentelemetry-collector:
|
||||
mode: daemonset
|
||||
|
||||
presets:
|
||||
# Collect Kubernetes pod/container logs
|
||||
logsCollection:
|
||||
enabled: true
|
||||
includeCollectorLogs: false
|
||||
|
||||
# Enrich telemetry with Kubernetes metadata
|
||||
kubernetesAttributes:
|
||||
enabled: true
|
||||
extractAllPodLabels: true
|
||||
extractAllPodAnnotations: false
|
||||
|
||||
# Collect host-level metrics (CPU, memory, disk, network)
|
||||
hostMetrics:
|
||||
enabled: true
|
||||
|
||||
# Collect kubelet/cAdvisor metrics
|
||||
kubeletMetrics:
|
||||
enabled: true
|
||||
|
||||
config:
|
||||
receivers:
|
||||
# Scrape Prometheus metrics from agent-monitor pods
|
||||
prometheus:
|
||||
config:
|
||||
scrape_configs:
|
||||
- job_name: "agent-monitor"
|
||||
scrape_interval: 15s
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
namespaces:
|
||||
names:
|
||||
- agent-monitor
|
||||
- agent-monitor-staging
|
||||
- agent-monitor-production
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
|
||||
action: keep
|
||||
regex: "true"
|
||||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
|
||||
action: replace
|
||||
target_label: __metrics_path__
|
||||
regex: (.+)
|
||||
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
|
||||
action: replace
|
||||
regex: ([^:]+)(?::\d+)?;(\d+)
|
||||
replacement: $$1:$$2
|
||||
target_label: __address__
|
||||
|
||||
# Receive OTLP from in-cluster services (gRPC + HTTP)
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:4317"
|
||||
http:
|
||||
endpoint: "0.0.0.0:4318"
|
||||
|
||||
processors:
|
||||
# Batch telemetry for efficient export
|
||||
batch:
|
||||
send_batch_size: 1024
|
||||
send_batch_max_size: 2048
|
||||
timeout: 5s
|
||||
|
||||
# Enrich with resource attributes
|
||||
resource:
|
||||
attributes:
|
||||
- key: cx.application.name
|
||||
value: "agent-monitor"
|
||||
action: upsert
|
||||
- key: cx.subsystem.name
|
||||
from_attribute: k8s.container.name
|
||||
action: upsert
|
||||
- key: k8s.cluster.name
|
||||
value: "agent-monitor"
|
||||
action: upsert
|
||||
|
||||
# Memory limiter to prevent OOM
|
||||
memory_limiter:
|
||||
check_interval: 5s
|
||||
limit_percentage: 80
|
||||
spike_limit_percentage: 25
|
||||
|
||||
# Filter out noisy internal logs
|
||||
filter/drop-internal:
|
||||
logs:
|
||||
exclude:
|
||||
match_type: regexp
|
||||
bodies:
|
||||
- ".*kube-probe.*"
|
||||
- ".*healthz.*"
|
||||
|
||||
exporters:
|
||||
coralogix:
|
||||
domain: "${CORALOGIX_DOMAIN}"
|
||||
private_key: "${PRIVATE_KEY}"
|
||||
application_name: "agent-monitor"
|
||||
subsystem_name: "kubernetes"
|
||||
timeout: 30s
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource, filter/drop-internal, batch]
|
||||
exporters: [coralogix]
|
||||
metrics:
|
||||
receivers: [otlp, prometheus]
|
||||
processors: [memory_limiter, resource, batch]
|
||||
exporters: [coralogix]
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource, batch]
|
||||
exporters: [coralogix]
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
effect: NoSchedule
|
||||
|
||||
# ── Gateway mode (optional – for centralized export) ─────────────────────────
|
||||
opentelemetry-gateway:
|
||||
enabled: false
|
||||
replicaCount: 2
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1Gi"
|
||||
Reference in New Issue
Block a user