AgentHub/docs/DEPLOYMENT.md

# Deployment Guide

Enterprise deployment strategies for Agent Dashboard across development, staging, and production environments.

---

## Table of Contents

- [Overview](#overview)
- [Deployment Architecture](#deployment-architecture)
- [Local Development](#local-development)
- [Production Deployment](#production-deployment)
- [Cloud Deployment](#cloud-deployment)
- [Process Management](#process-management)
- [Monitoring & Logging](#monitoring--logging)
- [Backup & Recovery](#backup--recovery)
- [Security Hardening](#security-hardening)
- [Performance Tuning](#performance-tuning)
- [Troubleshooting](#troubleshooting)

---

## Overview

Agent Dashboard supports multiple deployment modes:

- **Local Development** - Hot reload for rapid iteration
- **PM2** - Process management for production
- **Systemd** - System service on Linux
- **Cloud** - Deploy to AWS, Azure, GCP, or other cloud providers

```mermaid
graph TB
    subgraph "Deployment Modes"
        Dev[Local Development<br/>npm run dev]
        PM2[PM2 Process Manager<br/>pm2 start]
        Systemd[Systemd Service<br/>systemctl start]
        Cloud[Cloud Platform<br/>Kubernetes, ECS, etc.]
    end
    
    subgraph "Environment"
        DevEnv[Development<br/>Hot reload, verbose logs]
        StagingEnv[Staging<br/>Production build, test data]
        ProdEnv[Production<br/>Optimized, monitoring]
    end
    
    Dev --> DevEnv
    PM2 --> ProdEnv
    Systemd --> ProdEnv
    Cloud --> ProdEnv
    
    style Dev fill:#3B82F6
    style PM2 fill:#10B981
    style Cloud fill:#F59E0B
```

---

## Deployment Architecture

### Single-Server Architecture

```mermaid
graph TB
    subgraph "Server Host"
        subgraph "Node.js Process"
            Express[Express Server<br/>:4820]
            Static[Static File Serving<br/>client/dist/]
            API[REST API]
            WS[WebSocket Server]
            DB[(SQLite DB<br/>data/dashboard.db)]
        end
    end
    
    subgraph "Clients"
        Browser[Web Browsers]
        MCP[MCP Clients]
    end
    
    subgraph "Claude Code"
        Hooks[Hook Events]
    end
    
    Browser -->|HTTP/WS| Express
    MCP -->|HTTP| API
    Hooks -->|HTTP POST| Express
    
    Express --> Static
    Express --> API
    Express --> WS
    API --> DB
    Express --> DB
    
    style Express fill:#000000,color:#fff
    style DB fill:#003B57,color:#fff
```

### High-Availability Architecture

```mermaid
graph TB
    subgraph "Load Balancer"
        LB[Nginx/HAProxy]
    end
    
    subgraph "Application Servers"
        App1[Node.js Server 1<br/>:4820]
        App2[Node.js Server 2<br/>:4820]
        App3[Node.js Server 3<br/>:4820]
    end
    
    subgraph "Data Layer"
        Redis[Redis<br/>WebSocket pub/sub]
        DB[(PostgreSQL<br/>Shared database)]
    end
    
    LB --> App1
    LB --> App2
    LB --> App3
    
    App1 --> Redis
    App2 --> Redis
    App3 --> Redis
    
    App1 --> DB
    App2 --> DB
    App3 --> DB
    
    style LB fill:#10B981
    style Redis fill:#DC2626
    style DB fill:#2563EB
```

---

## Local Development

### Prerequisites

- Node.js >= 20.0.0
- npm >= 9.0.0
- git — every lane operation shells out to it
- tmux — only for the browser run console and `agenthub workspace shell`; both
  execute the binary directly and have no fallback. Per-OS install commands and
  the two-sided check (your PATH vs the server's) are in
  [INSTALL.md → Step 0](../INSTALL.md#step-0--system-packages).

### Setup

```bash
# Clone repository
git clone https://github.com/your-org/agenthub.git
cd agenthub

# Install dependencies
npm run setup

# Start development servers
npm run dev
```

### Development Architecture

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant Server as Dev Server<br/>:4820 (watch mode)
    participant Client as Vite Dev<br/>:5173 (HMR)
    participant Browser
    
    Dev->>Server: Edit server/*.js
    Server->>Server: Auto-reload
    Server-->>Dev: Ready
    
    Dev->>Client: Edit client/src/*
    Client->>Client: HMR rebuild
    Client->>Browser: Hot update
    Browser->>Browser: Re-render
    
    Note over Browser: State preserved!
```

### Running Components Separately

```bash
# Terminal 1: Server only
npm run dev:server

# Terminal 2: Client only
npm run dev:client

# Terminal 3: MCP server (optional; normally Claude Code starts it — docs/MCP.md)
node mcp/server.js
```

---

## Production Deployment

### Build Process

```mermaid
graph TB
    Source[Source Code] --> Install[Install Dependencies<br/>npm ci --production]
    Install --> BuildClient[Build Client<br/>npm run build]
    BuildClient --> Bundle[Bundled Assets<br/>client/dist/]
    Bundle --> Deploy[Deploy to Server]
    
    Deploy --> Server[Start Server<br/>node server/index.js]
    
    style BuildClient fill:#646CFF
    style Server fill:#10B981
```

### Production Checklist

```bash
# 1. Install dependencies (production only)
npm ci --production
cd client && npm ci --production && cd ..

# 2. Build client
npm run build

# 3. Set environment variables
export NODE_ENV=production
export PORT=4820
export DASHBOARD_DB_PATH=/var/lib/agenthub/dashboard.db

# 4. Create data directory
mkdir -p /var/lib/agenthub

# 5. Start server
node server/index.js
```

### Environment Variables

```bash
# Server
DASHBOARD_PORT=4820                    # Server port
NODE_ENV=production                    # Environment mode

# Network exposure (SECURITY — GHSA-gr74-4xfh-6jw9)
# The server binds 127.0.0.1 by default and is NOT network-reachable. It reads
# transcripts, exports all data, and can spawn `claude`, so only widen the bind
# deliberately — and require a token when you do.
DASHBOARD_HOST=127.0.0.1               # set 0.0.0.0 ONLY if you must expose it
DASHBOARD_TOKEN=                        # required on /api/* + WS when set; use with a non-loopback host
DASHBOARD_ALLOWED_HOSTS=                # extra Host names (comma-sep) for a LAN bind

# Database
DASHBOARD_DB_PATH=/var/lib/agenthub/dashboard.db

# Ingestion polls (0 disables any one of them)
DASHBOARD_WORKFLOW_POLL_MS=12000        # Workflow-tool run journals
DASHBOARD_CODEX_POLL_MS=15000           # Codex rollout journals → codex lane running/idle + Codex sessions/tokens
CODEX_HOME=                             # where Codex keeps sessions/ (default ~/.codex)
DASHBOARD_ANTIGRAVITY_POLL_MS=15000     # agy conversation files → Antigravity sessions/tokens (60 s while idle)
AGENTHUB_GEMINI_DIR=                    # where agy keeps its data (default ~/.gemini)
DASHBOARD_CURSOR_POLL_MS=15000          # Cursor agent transcripts → Cursor sessions (no tokens; 60 s while idle)
AGENTHUB_CURSOR_DIR=                    # where Cursor keeps its data (default ~/.cursor)
AGENTHUB_AGY_USAGE=                     # 0 never runs `agy --print /usage` for the Usage panel's Antigravity quota (default on; running agy may update it)

# Pricing and saved settings
AGENTHUB_PRICING_SYNC=                  # 0|false|off disables automatic pricing from LiteLLM (default on)
AGENTHUB_READ_WORKER=                   # 0 runs heavy aggregates (GET /api/workflows, /api/analytics' event scans) inline instead of in the read worker (default worker)
AGENTHUB_READ_TIMEOUT_MS=60000          # a read-worker job past this answers 503 EREADTIMEOUT and gets a fresh worker
DASHBOARD_WAL_CHECKPOINT_MS=5000        # WAL checkpoint from the read worker instead of inside a hook's COMMIT (0 = SQLite's automatic one)
AGENTHUB_SETTINGS_FILE=                 # where dashboard-saved settings persist (default ~/.config/agenthub/settings.env);
                                        # load order: real env > the checkout's .env > this file

# Logging
LOG_LEVEL=info                         # debug | info | warn | error
```

> **Reverse-proxy / container exposure:** the app binds loopback, so publish it to a
> network only through a proxy you control that adds TLS + auth, or set
> `DASHBOARD_HOST=0.0.0.0` **with** `DASHBOARD_TOKEN`. A published `4820` port
> assumes a trusted host network — do not expose it publicly without a
> token and a proxy.

---

## Cloud Deployment

### AWS (Elastic Beanstalk)

```bash
# Install EB CLI
pip install awsebcli

# Initialize
eb init -p node.js agenthub

# Create environment
eb create production

# Deploy
eb deploy

# Open in browser
eb open
```

### Kubernetes

The repo ships no container image or `Dockerfile` — build your own from
`npm ci && npm run build` and push it to your registry, then reference it here.

```yaml
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agenthub
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agenthub
  template:
    metadata:
      labels:
        app: agenthub
    spec:
      containers:
      - name: agenthub
        image: agenthub:latest
        ports:
        - containerPort: 4820
        env:
        - name: NODE_ENV
          value: "production"
        volumeMounts:
        - name: data
          mountPath: /app/data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: agenthub-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: agenthub
spec:
  selector:
    app: agenthub
  ports:
  - protocol: TCP
    port: 80
    targetPort: 4820
  type: LoadBalancer
```

### Kubernetes Architecture

```mermaid
graph TB
    subgraph "Kubernetes Cluster"
        subgraph "LoadBalancer Service"
            LB[Load Balancer<br/>:80]
        end
        
        subgraph "Pods"
            Pod1[agenthub-1<br/>:4820]
            Pod2[agenthub-2<br/>:4820]
            Pod3[agenthub-3<br/>:4820]
        end
        
        subgraph "Storage"
            PVC[PersistentVolumeClaim]
            PV[PersistentVolume]
        end
    end
    
    LB --> Pod1
    LB --> Pod2
    LB --> Pod3
    
    Pod1 --> PVC
    Pod2 --> PVC
    Pod3 --> PVC
    PVC --> PV
    
    style LB fill:#10B981
    style PV fill:#F59E0B
```

---

## Process Management

### PM2 (Production Process Manager)

```bash
# Install PM2
npm install -g pm2

# Start application
pm2 start server/index.js --name agenthub

# Start with environment
pm2 start server/index.js --name agenthub --env production

# View logs
pm2 logs agenthub

# Monitor
pm2 monit

# Restart
pm2 restart agenthub

# Stop
pm2 stop agenthub

# Auto-start on system boot
pm2 startup
pm2 save
```

### PM2 Ecosystem File

```javascript
// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'agenthub',
    script: './server/index.js',
    instances: 2,
    exec_mode: 'cluster',
    env: {
      NODE_ENV: 'development',
      PORT: 4820
    },
    env_production: {
      NODE_ENV: 'production',
      PORT: 4820,
      DASHBOARD_DB_PATH: '/var/lib/agenthub/dashboard.db'
    },
    max_memory_restart: '500M',
    error_file: '/var/log/agenthub/error.log',
    out_file: '/var/log/agenthub/out.log',
    time: true
  }]
};
```

```bash
# Start with ecosystem file
pm2 start ecosystem.config.js --env production
```

### Systemd Service (Linux)

```ini
# /etc/systemd/system/agenthub.service
[Unit]
Description=Agent Dashboard
After=network.target

[Service]
Type=simple
User=agenthub
WorkingDirectory=/opt/agenthub
Environment=NODE_ENV=production
Environment=PORT=4820
Environment=DASHBOARD_DB_PATH=/var/lib/agenthub/dashboard.db
ExecStart=/usr/bin/node server/index.js
# `always`, not `on-failure`: a graceful SIGTERM exits 0, which `on-failure`
# reads as "meant to stop" — one stray kill and the dashboard stays down.
# `systemctl stop` still stops it. Same reasoning as the plugin's own unit.
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

```bash
# Enable and start service
sudo systemctl enable agenthub
sudo systemctl start agenthub

# Check status
sudo systemctl status agenthub

# View logs
sudo journalctl -u agenthub -f
```

---

## Monitoring & Logging

### Health Checks

```bash
# Server health check
curl http://localhost:4820/api/sessions

# Expected: {"sessions": [...]}
```

### Logging Strategy

```mermaid
graph TB
    subgraph "Application Logs"
        App[Express Server] --> Console[Console Output]
        Console --> Stdout[stdout/stderr]
    end
    
    subgraph "Log Aggregation"
        Stdout --> PM2[PM2 Logs]
        Stdout --> Systemd[Systemd Journal]
    end
    
    subgraph "Monitoring Tools"
        PM2 --> File[Log Files]
        Systemd --> Journalctl[journalctl]
    end
    
    subgraph "Analysis"
        File --> Splunk[Splunk/ELK]
        Journalctl --> Splunk
    end
    
    style App fill:#000000,color:#fff
    style Splunk fill:#10B981
```

### Monitoring Metrics

The server already exposes Prometheus metrics at `GET /api/metrics`
([API.md § Prometheus exposition](./API.md#prometheus-exposition)). A scrape
config, recording rules and three Grafana boards ship in `monitoring/`:

```bash
DASHBOARD_ALLOWED_HOSTS=host.docker.internal npm start
GF_SECURITY_ADMIN_PASSWORD='choose-one' docker compose -f docker-compose.monitoring.yml up -d
```

See [MONITORING.md](./MONITORING.md).

---

## Backup & Recovery

### Backup Strategy

```mermaid
graph TB
    subgraph "Backup Process"
        DB[(SQLite DB)] --> Backup[Backup Script]
        Backup --> Local[Local Storage<br/>./backups/]
        Backup --> S3[AWS S3]
        Backup --> Cloud[Cloud Storage]
    end
    
    subgraph "Schedule"
        Cron[Cron Job<br/>Daily at 2 AM]
    end
    
    subgraph "Retention"
        Daily[Daily: 7 days]
        Weekly[Weekly: 4 weeks]
        Monthly[Monthly: 12 months]
    end
    
    Cron --> Backup
    Local --> Daily
    S3 --> Weekly
    Cloud --> Monthly
    
    style DB fill:#003B57,color:#fff
    style S3 fill:#FF9900
```

### Backup Script

```bash
#!/bin/bash
# scripts/backup.sh

BACKUP_DIR="/var/backups/agenthub"
DB_PATH="/var/lib/agenthub/dashboard.db"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/dashboard_$TIMESTAMP.db"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Create backup (online backup with VACUUM INTO)
sqlite3 "$DB_PATH" "VACUUM INTO '$BACKUP_FILE'"

# Compress backup
gzip "$BACKUP_FILE"

# Upload to S3 (optional)
aws s3 cp "$BACKUP_FILE.gz" s3://my-backups/agenthub/

# Delete old backups (keep last 7 days)
find "$BACKUP_DIR" -name "dashboard_*.db.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_FILE.gz"
```

### Restore Process

```bash
#!/bin/bash
# scripts/restore.sh

BACKUP_FILE=$1
DB_PATH="/var/lib/agenthub/dashboard.db"

if [ -z "$BACKUP_FILE" ]; then
  echo "Usage: ./restore.sh <backup_file.db.gz>"
  exit 1
fi

# Stop application
systemctl stop agenthub

# Decompress backup
gunzip -c "$BACKUP_FILE" > /tmp/restore.db

# Restore database
cp /tmp/restore.db "$DB_PATH"
chown agenthub:agenthub "$DB_PATH"

# Start application
systemctl start agenthub

echo "Restore completed from $BACKUP_FILE"
```

---

## Security Hardening

### Security Checklist

```mermaid
graph TB
    subgraph "Network Security"
        Firewall[Firewall Rules<br/>Allow :4820 only from trusted IPs]
        TLS[TLS/SSL<br/>HTTPS + WSS]
        CORS[CORS Configuration<br/>Restrict origins]
    end
    
    subgraph "Application Security"
        Validation[Input Validation]
        Prepared[Prepared Statements<br/>SQL injection prevention]
        Sanitize[Output Sanitization]
    end
    
    subgraph "System Security"
        User[Dedicated User<br/>Non-root]
        Perms[File Permissions<br/>640 for DB]
        SELinux[SELinux/AppArmor]
    end
    
    style Firewall fill:#10B981
    style Prepared fill:#10B981
    style User fill:#10B981
```

### TLS Configuration (Nginx Reverse Proxy)

```nginx
# /etc/nginx/sites-available/agenthub
server {
    listen 443 ssl http2;
    server_name dashboard.example.com;
    
    ssl_certificate /etc/letsencrypt/live/dashboard.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/dashboard.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    
    location / {
        proxy_pass http://localhost:4820;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    
    location /ws {
        proxy_pass http://localhost:4820/ws;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name dashboard.example.com;
    return 301 https://$server_name$request_uri;
}
```

---

## Performance Tuning

### Node.js Optimization

```bash
# Increase memory limit
NODE_OPTIONS="--max-old-space-size=4096" node server/index.js

# Enable V8 optimizations
node --optimize-for-size server/index.js
```

### SQLite Tuning

`server/db.js` already sets WAL, `synchronous = NORMAL`, a 64 MB `cache_size` and a
256 MB `mmap_size`, and the server checkpoints the WAL from its read worker
(`DASHBOARD_WAL_CHECKPOINT_MS`) instead of inside a hook's `COMMIT` — see
[DATABASE.md § SQLite Pragmas](DATABASE.md#sqlite-pragmas). Nothing to add by hand.

### Compression

The client build writes a `.br` and a `.gz` beside every hashed asset, and the
server sends the one the browser accepts (`Content-Encoding`, `Vary: Accept-Encoding`)
— a first Workspace load is ~75 % smaller with no compression CPU at request time.
API responses are not compressed by AgentHub; behind nginx, add:

```nginx
gzip on;
gzip_types application/json;
gzip_min_length 1024;
```

### Nginx Tuning

```nginx
# /etc/nginx/nginx.conf
worker_processes auto;
worker_connections 4096;

http {
    # Enable compression
    gzip on;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript;
    
    # Client body buffer
    client_body_buffer_size 128k;
    
    # Keepalive
    keepalive_timeout 65;
    keepalive_requests 100;
    
    # Proxy buffering
    proxy_buffering on;
    proxy_buffer_size 4k;
    proxy_buffers 8 4k;
}
```

---

## Troubleshooting

### Common Issues

```mermaid
graph TB
    subgraph "Issue Categories"
        Startup[Startup Failures]
        Connection[Connection Errors]
        Performance[Performance Issues]
        Data[Data Inconsistencies]
    end
    
    subgraph "Diagnostics"
        Logs[Check Logs]
        Health[Health Checks]
        Metrics[Monitor Metrics]
        DB[Database Integrity]
    end
    
    Startup --> Logs
    Connection --> Health
    Performance --> Metrics
    Data --> DB
    
    style Logs fill:#F59E0B
```

### Issue Resolution Guide

| Issue | Symptoms | Solution |
|-------|----------|----------|
| Port already in use | `EADDRINUSE: address already in use :::4820` | `lsof -i :4820` then kill process |
| Database locked | `database is locked` | Check for long-running queries, increase timeout |
| WebSocket connection fails | Clients can't connect | Check firewall, verify WebSocket upgrade headers |
| High memory usage | >500MB RAM | Enable memory limits, check for leaks |
| Slow queries | API responses >100ms | Add indexes, use EXPLAIN QUERY PLAN |

### Debug Mode

```bash
# Enable verbose logging
DEBUG=* node server/index.js

# SQLite query logging
NODE_ENV=development node server/index.js
```

---

## Summary

This deployment guide covers:

- ✅ **Multiple deployment modes** - Local, PM2, Systemd, Cloud
- ✅ **Production best practices** - Environment variables, health checks, logging
- ✅ **Process management** - PM2, systemd service files
- ✅ **Monitoring & logging** - Metrics endpoint, log aggregation
- ✅ **Backup & recovery** - Automated backups, restore procedures
- ✅ **Security hardening** - TLS, CORS, firewall rules
- ✅ **Performance tuning** - Node.js, SQLite, Nginx optimizations
- ✅ **Troubleshooting** - Common issues and resolutions

For architecture details, see [ARCHITECTURE.md](../ARCHITECTURE.md).