# Database Schema Reference Comprehensive database schema documentation for Agent Dashboard SQLite database. --- ## Table of Contents - [Overview](#overview) - [Schema Diagram](#schema-diagram) - [Table Definitions](#table-definitions) - [Indexes](#indexes) - [Migrations](#migrations) - [Query Patterns](#query-patterns) - [Performance Optimization](#performance-optimization) - [Data Integrity](#data-integrity) - [Backup Strategies](#backup-strategies) --- ## Overview Agent Dashboard uses **SQLite 3** as its primary data store with the following characteristics: - **File-based** - Single database file, portable across systems - **Embedded** - No separate server process required - **ACID compliant** - Transactions ensure data integrity - **WAL mode** - Write-Ahead Logging for better concurrency - **Prepared statements** - Prevent SQL injection, optimize performance ```mermaid graph TB subgraph "Database File" DB[(dashboard.db)] end subgraph "Tables" Sessions[sessions] Agents[agents] Tools[tool_executions] Notifs[notifications] Pricing[pricing_rules] Remote[remote_sources] end subgraph "Indexes" Idx1[session_id, status, updated_at] Idx2[agent_id, session_id, status] Idx3[agent_id, created_at] end DB --> Sessions DB --> Agents DB --> Tools DB --> Notifs DB --> Pricing DB --> Remote Sessions --> Idx1 Agents --> Idx2 Tools --> Idx3 style DB fill:#003B57,color:#fff ``` **Database Location:** - **Canonical (default):** `~/.claude/agenthub/dashboard.db` — shared by `npm start`, `npm run dev`, and the desktop app when it uses the same data dir - **Override:** set `DASHBOARD_DATA_DIR` (directory) or `DASHBOARD_DB_PATH` (file path) for tests or custom deployments - **Pinned on a CLAUDE_HOME change:** the data dir defaults to `/agenthub`, so saving a new CLAUDE_HOME from Settings → Claude Code → Overview first writes the data dir in use as `DASHBOARD_DATA_DIR` to AgentHub's settings file (`~/.config/agenthub/settings.env`, or `$XDG_CONFIG_HOME/agenthub/settings.env`, or `AGENTHUB_SETTINGS_FILE`). The next start keeps the same database instead of opening an empty one under the new home. An explicit `DASHBOARD_DATA_DIR` is left alone - **Legacy:** repo-local `./data/dashboard.db` is migrated into the canonical location on first launch (see `server/db.js`) - **Pre-rebrand:** an install that predates AgentHub keeps its data in `~/.claude/agent-dashboard/`. Server startup renames that directory to `~/.claude/agenthub/` once, before anything opens the database (`migrateLegacyDataDir()` in `server/lib/claude-home.js`). It is a same-filesystem rename, so it is instant even at multiple gigabytes and never leaves two copies. If the rename fails, or `DASHBOARD_DATA_DIR` is set, nothing moves and the old directory keeps being used. --- ## Schema Diagram ### Entity-Relationship Diagram ```mermaid erDiagram sessions ||--o{ agents : "has many" agents ||--o{ tool_executions : "has many" sessions ||--o{ notifications : "has many" remote_sources ||--o{ sessions : "tags (source)" sessions { integer id PK "Primary key" text session_id UK "Unique session identifier" text model "Raw model slug (e.g., claude-sonnet-4-5-20250514); UI displays via formatModelName()" text status "active | completed" real total_cost "Aggregated cost from all agents" text source "'local' or a remote_sources.id" text created_at "ISO8601 timestamp" text updated_at "ISO8601 timestamp (bumped on every hook)" } agents { integer id PK "Primary key" text agent_id UK "Unique agent identifier" text session_id FK "Foreign key to sessions" text agent_type "explore, task, general-purpose, etc." text status "running | completed | failed" text current_tool "Currently executing tool (or NULL)" integer input_tokens "Cumulative input tokens" integer output_tokens "Cumulative output tokens" real cost "Calculated cost for this agent" text created_at "ISO8601 timestamp" text updated_at "ISO8601 timestamp" } tool_executions { integer id PK "Primary key" text agent_id FK "Foreign key to agents" text tool_name "bash, view, edit, grep, etc." integer duration_ms "Execution time in milliseconds" integer success "1 = success, 0 = failure" text error_message "NULL if success, error details if failed" text created_at "ISO8601 timestamp" } notifications { integer id PK "Primary key" text session_id FK "Foreign key to sessions" text notification_type "backgroundTaskComplete, etc." text message "Notification message" text created_at "ISO8601 timestamp" } pricing_rules { integer id PK "Primary key" text pattern UK "Model pattern (e.g., claude-sonnet-4)" real input_cost_per_1m "Input cost per 1M tokens (USD)" real output_cost_per_1m "Output cost per 1M tokens (USD)" text created_at "ISO8601 timestamp" } remote_sources { text id PK "Remote-source id (also used as sessions.source)" text label "Human-readable name" text host "SSH destination user@host or ~/.ssh/config alias" integer ssh_port "Optional SSH port (NULL = SSH default)" text identity_file "Optional private-key path (NULL = SSH default)" text remote_home "Optional remote Claude home (NULL = remote ~/.claude)" integer enabled "1 = eligible for sync, 0 = disabled" integer sync_codex "1 = also pull Codex history (default 0)" text remote_codex_home "Optional remote Codex home (NULL = ~/.codex)" text status "idle | syncing | ok | error" text last_error "Last failure message, or NULL" text last_sync_at "ISO8601 timestamp of last successful sync, or NULL" text last_sync_counts "JSON blob of last sync counters, or NULL" text created_at "ISO8601 timestamp" text updated_at "ISO8601 timestamp" } ``` ### Relationship Cardinality ```mermaid graph LR Session[Session
1] -->|1:N| Agents[Agents
N] Session -->|1:N| Notifications[Notifications
N] Agents -->|1:N| Tools[Tool Executions
N] style Session fill:#3B82F6 style Agents fill:#10B981 style Tools fill:#F59E0B ``` --- ## Table Definitions ### sessions Tracks Claude Code sessions (one per CLI invocation or background task). Schema mirrors `server/db.js`. > **Cursor (informational):** Rows imported from `~/.claude` JSONL transcripts may also represent **Cursor** agent sessions — Cursor happens to use the same on-disk layout as Claude Code, so such a row still reads `tool = 'claude'`. ```sql CREATE TABLE sessions ( id TEXT PRIMARY KEY, -- UUID from Claude Code name TEXT, status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','completed','error','abandoned')), cwd TEXT, model TEXT, started_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), ended_at TEXT, metadata TEXT, updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), awaiting_input_since TEXT, -- NULL unless Waiting awaiting_reason TEXT, -- notification|stop|session_start|interrupted, or NULL transcript_path TEXT, -- absolute path to JSONL transcript source TEXT NOT NULL DEFAULT 'local', -- data source: 'local' or a remote_sources.id tool TEXT NOT NULL DEFAULT 'claude' -- 'claude', 'codex' or 'antigravity' ); ``` **Columns:** | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `id` | TEXT | NO | Session UUID (assigned by Claude Code) | | `name` | TEXT | YES | Human-readable label. Synced from the transcript title by `routes/hooks.js` (and the 15 s watchdog) on every event: the `custom-title` line (`/rename`, `claude -n`, picker `Ctrl+R`) always wins, otherwise the auto-generated `ai-title` fills a placeholder/auto name, otherwise the session's first user prompt (60-char label) fills it. Falls back to `Session ` | | `status` | TEXT | NO | `active`, `completed`, `error`, or `abandoned` (CHECK-constrained). Besides the `SessionEnd` hook, the 15 s watchdog's **liveness reap** also lands `active` → `completed` when no running `claude` process has the session's `cwd` (a `SessionEnd` lost while the dashboard was down); gated by `DASHBOARD_LIVENESS_IDLE_SECONDS`, disabled via `DASHBOARD_LIVENESS_PROBE=0`. Sessions with a non-`local` `source` (Remote Data Sources) are exempt from the reap and both stale sweeps — their status is reconciled from the SSH mirror by `remote-sync.js` instead | | `cwd` | TEXT | YES | Working directory the CLI was launched from | | `model` | TEXT | YES | Model ID as the CLI reports it — a Claude id (e.g. `claude-opus-4-7`), an OpenAI one for a Codex session (`gpt-…`), a Gemini one for an Antigravity session (`gemini-…`) | | `started_at` | TEXT | NO | ISO 8601 timestamp | | `ended_at` | TEXT | YES | ISO 8601 timestamp on terminal transition | | `metadata` | TEXT | YES | JSON blob for extras (turn duration totals, thinking blocks, …) | | `updated_at` | TEXT | NO | Bumped on every event for staleness detection | | `awaiting_input_since` | TEXT | YES | ISO 8601 stamp set when the session is **Waiting** (Stop, SessionStart with source `startup`/`resume`/`clear`, permission Notification, or watchdog user-interrupt/Esc recovery). NULL otherwise. A SessionStart with source `compact` (auto-compaction fires mid-turn while Claude is working) leaves this column untouched, so a genuinely-active session is not mislabeled Waiting | | `awaiting_reason` | TEXT | YES | Why the row is waiting: `notification`, `stop`, `session_start`, or `interrupted`. Set/cleared in lock-step with `awaiting_input_since` (SessionStart→`session_start`, Stop→`stop`, permission/input Notification→`notification`, watchdog/Esc recovery→`interrupted`). NULL otherwise. Exception: a `compact`-source SessionStart preserves the existing value (neither stamps `session_start` nor clears it) | | `transcript_path` | TEXT | YES | Absolute path to the session's JSONL transcript. Written by `routes/hooks.js` on the first event that carries it (subsequent events no-op via a SQL guard) and read by the periodic compaction sweep — so the sweep touches only active session rows instead of scanning the entire `events` table for `json_extract(data,'$.transcript_path')`. Backfilled once from `events` by the `db.js` migration | | `source` | TEXT | NO | Data source this session was captured from. `'local'` for this machine's own Claude Code history (the default); otherwise the `remote_sources.id` of the remote SSH machine it was pulled from, or `push:` for a session another machine pushed to `POST /api/hooks/ingest-batch` (a pushed hook may only extend a session its own source created). Powers the `sources` query filter on `/api/sessions`, `/api/events`, `/api/agents`, `/api/stats`, and `/api/analytics`, and the `sources` facet on `/api/sessions/facets`. Indexed by `idx_sessions_source` | | `tool` | TEXT | NO | The AI coding CLI that produced the session. `'claude'` for every hook-created and imported row (the default); `'codex'` for a session ingested from Codex's rollout journals by `server/lib/codex-usage.js` (see `codex_rollouts`) or created by a Codex hook (`tool: "codex"`, `server/lib/codex-hooks.js`) — a Codex session keeps the status its hooks give it while they are current (its newest event at or after its newest journal write minus two minutes); once they stop (removed, untrusted) the journal decides again. A codex row has no `transcript_path`, and, going by the journal, its `status` is `active` only while one of its journals was written in the last two minutes. `'antigravity'` for an agy CLI conversation, ingested by `server/lib/antigravity-usage.js` (see `antigravity_conversations`) or created by an Antigravity hook (`server/lib/antigravity-hooks.js`); no `transcript_path`, and `active` until it has been quiet — no write to its conversation file (or a non-empty `-wal`) or transcript, no hook event — for ten minutes when it has hook events, two minutes when it has none; it then settles to `completed`, its agents too. `'cursor'` for a Cursor agent transcript, ingested by `server/lib/cursor-usage.js` (see `cursor_transcripts`): no `transcript_path`, no `model` and no `token_usage` (the transcripts record none), `active` while its transcript was written in the last two minutes, else `completed`. The startup cleanup of stale `active` sessions and a workspace purge touch only the matching tool's rows (a purge takes the sessions of the workspace's own `tool`). The Workspace's attach/resume list offers `'claude'` rows only | | `lane_id` | INTEGER | YES | The workspace that OWNS the session — the only sessions a workspace lists: the lane whose dashboard pane held it, stamped by `bindRunSession` (`server/routes/hooks.js`) from the hook's `run_id` — first writer wins, a nested `claude -p` excluded — or, for a Codex/Antigravity session, by `claimToolSessions` (`server/lib/dashboard-runs.js`) from the same-tool run that was live in its folder when it started. NULL for a session no workspace started (a terminal, an import, a tool with no hooks). No FK: removing a workspace leaves its sessions. Backfilled once from `dashboard_runs`. Indexed by `idx_sessions_lane` | | `archived_at` | TEXT | YES | When the user archived the session out of its workspace's lists (`PATCH /api/sessions/:id {archived}`). Hides only — nothing is deleted | | `pinned_at` | TEXT | YES | When the user pinned the session to the top of its workspace's sidebar list (`PATCH /api/sessions/:id {pinned}`; `GET /api/sessions?pinned_first=1` orders by it). Order only; NULL = not pinned | **Constraints:** - `status` must be one of the four enum values - `awaiting_input_since` is ignored on non-`active` sessions for UI bucketing **Lifecycle:** ```mermaid stateDiagram-v2 [*] --> waiting: SessionStart startup/resume/clear (status=active + awaiting_input_since) active --> active: SessionStart compact (mid-turn — state preserved) waiting --> active: UserPromptSubmit / PreToolUse / PostToolUse active --> waiting: Stop (non-error) / Permission Notification active --> waiting: Esc cancel (watchdog marker or idle timeout) active --> error: Stop (stop_reason=error) waiting --> completed: SessionEnd active --> completed: SessionEnd waiting --> abandoned: Stale > DASHBOARD_STALE_MINUTES active --> abandoned: Stale > DASHBOARD_STALE_MINUTES completed --> active: Resumed error --> active: Resumed abandoned --> active: Resumed completed --> [*] error --> [*] abandoned --> [*] ``` --- ### agents Tracks main agents and subagents within a session. Main agents have id `${session_id}-main`; subagents get a fresh UUID. ```sql CREATE TABLE agents ( id TEXT PRIMARY KEY, session_id TEXT NOT NULL, name TEXT NOT NULL, type TEXT NOT NULL DEFAULT 'main' CHECK (type IN ('main','subagent')), subagent_type TEXT, status TEXT NOT NULL DEFAULT 'idle' CHECK (status IN ('idle','connected','working','completed','error')), task TEXT, current_tool TEXT, started_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), ended_at TEXT, parent_agent_id TEXT, metadata TEXT, updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), awaiting_input_since TEXT, -- main-agent waiting flag awaiting_reason TEXT, -- notification|stop|session_start|interrupted, or NULL FOREIGN KEY (session_id) REFERENCES sessions(id) ON DELETE CASCADE, FOREIGN KEY (parent_agent_id) REFERENCES agents(id) ON DELETE SET NULL ); ``` **Columns:** | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `id` | TEXT | NO | UUID (subagents) or `${session_id}-main` (main agent) | | `session_id` | TEXT | NO | FK to `sessions.id`, cascades on delete | | `name` | TEXT | NO | Display label (e.g. `Main Agent - {session name}` or subagent description) | | `type` | TEXT | NO | `main` or `subagent` | | `subagent_type` | TEXT | YES | `Explore`, `general-purpose`, `code-review`, `compaction`, … | | `status` | TEXT | NO | `idle`, `connected`, `working`, `completed`, `error` (CHECK-constrained). The dashboard's **Waiting** badge is the UI overlay produced by `awaiting_input_since`; it is not a persisted status | | `task` | TEXT | YES | Subagent prompt / brief | | `current_tool` | TEXT | YES | Tool currently running (cleared on `PostToolUse`) | | `parent_agent_id` | TEXT | YES | FK to the spawning agent for nested subagent trees (`ON DELETE SET NULL`). Set to the main agent at insert, then repointed to the true spawner by `reconcileSubagentParents` from each subagent transcript's Task tool result (`toolUseResult.agentId`), so subagents-of-subagents nest correctly instead of flattening under main | | `metadata` | TEXT | YES | JSON blob for extras. For subagents it carries `model` (the subagent's own model, issue #185) and `tokens` — an array of per-agent token buckets parsed from the subagent's transcript. The agent-list endpoints price `tokens` at the current rates to attach a per-agent `cost` (so a subagent card shows its OWN cost, not the session total). Empty `[]` means the subagent did no billable work; absent means its transcript wasn't available to parse | | `awaiting_input_since` | TEXT | YES | Mirrors the parent session's flag for the main agent. NULL on subagents | | `awaiting_reason` | TEXT | YES | Why the row is waiting: `notification`, `stop`, `session_start`, or `interrupted`. Set/cleared in lock-step with `awaiting_input_since`; explains why the main agent is waiting. NULL on subagents | **Lifecycle:** ```mermaid stateDiagram-v2 [*] --> Running: Agent created (SessionStart/PreToolUse) Running --> Running: PreToolUse (set current_tool) Running --> Running: PostToolUse (increment tokens, cost) Running --> Completed: Stop/SubagentStop hook Running --> Failed: Error during processing Completed --> [*] Failed --> [*] ``` **current_tool Behavior:** - Set to tool name on `PreToolUse` hook (e.g., `"bash"`, `"view"`) - Cleared to `NULL` on `PostToolUse` hook - Used to show real-time tool execution in UI --- ### tool_executions Records each tool call made by agents. ```sql CREATE TABLE tool_executions ( id INTEGER PRIMARY KEY AUTOINCREMENT, agent_id TEXT NOT NULL, tool_name TEXT NOT NULL, duration_ms INTEGER, success INTEGER DEFAULT 1, error_message TEXT, created_at TEXT DEFAULT (datetime('now')), FOREIGN KEY (agent_id) REFERENCES agents(agent_id) ); ``` **Columns:** | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `id` | INTEGER | NO | Auto-increment primary key | | `agent_id` | TEXT | NO | Foreign key to `agents.agent_id` | | `tool_name` | TEXT | NO | Tool name (`bash`, `view`, `edit`, `grep`, etc.) | | `duration_ms` | INTEGER | YES | Execution time in milliseconds | | `success` | INTEGER | NO | 1 = success, 0 = failure | | `error_message` | TEXT | YES | NULL if success, error details if failed | | `created_at` | TEXT | NO | ISO8601 timestamp of execution | **Common Tool Names:** - `bash` - Shell command execution - `view` - File/directory viewing - `edit` - File editing - `grep` - Code search - `glob` - File pattern matching - `task` - Sub-agent invocation - `sql` - SQLite query execution **Duration Distribution:** ```mermaid graph TB Tools[Tool Executions] --> Fast[Fast
< 100ms
view, grep] Tools --> Medium[Medium
100ms - 1s
edit, bash] Tools --> Slow[Slow
> 1s
task, build commands] style Fast fill:#10B981 style Medium fill:#F59E0B style Slow fill:#EF4444 ``` --- ### notifications Stores system notifications from Claude Code. ```sql CREATE TABLE notifications ( id INTEGER PRIMARY KEY AUTOINCREMENT, session_id TEXT NOT NULL, notification_type TEXT NOT NULL, message TEXT, created_at TEXT DEFAULT (datetime('now')), FOREIGN KEY (session_id) REFERENCES sessions(session_id) ); ``` **Columns:** | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `id` | INTEGER | NO | Auto-increment primary key | | `session_id` | TEXT | NO | Foreign key to `sessions.session_id` | | `notification_type` | TEXT | NO | Type of notification | | `message` | TEXT | YES | Notification message content | | `created_at` | TEXT | NO | ISO8601 timestamp | **Common Notification Types:** - `backgroundTaskComplete` - Background agent finished - `errorOccurred` - Error during execution - `systemMessage` - General system message --- ### model_pricing Per-model pricing rules for cost calculation, keyed by `model_pattern` (a SQL-style glob; `%` matches any characters). Rates are per **million** tokens (USD). The longest matching pattern wins. **Automatic rates.** `server/lib/pricing-sync.js` runs at server start and every 15 minutes, and fetches LiteLLM's public price list (`model_prices_and_context_window.json` on GitHub) at most once a day. Each pass (1) INSERTS a row, `source = 'sync'`, for every Claude, OpenAI or Gemini model seen in `token_usage` whose family (the id minus a `-YYYYMMDD` / `-YYYY-MM-DD` suffix; OpenAI means `gpt-…`, `o…` or `codex-…`; Gemini means `gemini-…`, taken whole) has no `%` row of its own, AND for every **current** model of the three vendors before anyone uses it — a list key that is a family itself, `mode` `chat`/`responses`, not past its `deprecation_date`, not an audio/search/TTS/realtime/image/embedding variant, and of the current generation (any `claude--…`; `gpt-` 5 and up; `gemini-` 3 and up — `MIN_GEN`); (2) UPDATES the rates of its own `sync` rows when the list changes; (3) adopts a row from before the `source` column (`NULL`: a seed, an old sync insert or an old edit) as `sync` only while its rates still equal the list exactly. A row edited through `PUT /api/pricing` becomes `source = 'user'` and is never touched again; a deleted row's pattern goes to `model_pricing_dismissed` and is never added back. An entry is taken only from its own vendor (`litellm_provider` `anthropic` / `openai`, and `vertex_ai-language-models` or `gemini` for Gemini); an OpenAI or Gemini row carries no cache-write charge unless the list names one, and no fast rate. A Claude entry's `*_above_100k_tokens` input/output rates fill `long_input_per_mtok` / `long_output_per_mtok` (Haiku 5.5); Gemini's >200k-token rate is not modelled (the base rate is used). A proxy's model id (`clinepass/…`) is never priced automatically. `POST /api/settings/reset-pricing` empties `model_pricing_dismissed` for its scope and runs a sync right after reseeding (waiting up to 3 s), so the OpenAI and Gemini rows a reset wiped come straight back; with `{scope: "claude"}` (the Claude Code tab's **Reset Defaults**) it deletes and reseeds only `claude-…` rows, so other vendors' rows and a custom proxy row survive. A rate outside 0–1000 $/MTok is rejected, and a failed fetch changes nothing. Set `AGENTHUB_PRICING_SYNC=0` (or `false`/`off`) to turn it off. ```sql CREATE TABLE model_pricing ( model_pattern TEXT PRIMARY KEY, display_name TEXT NOT NULL, input_per_mtok REAL NOT NULL DEFAULT 0, output_per_mtok REAL NOT NULL DEFAULT 0, cache_read_per_mtok REAL NOT NULL DEFAULT 0, cache_write_per_mtok REAL NOT NULL DEFAULT 0, cache_write_1h_per_mtok REAL NOT NULL DEFAULT 0, -- 1h-ephemeral cache-write tier fast_input_per_mtok REAL NOT NULL DEFAULT 0, -- fast-mode premium rates fast_output_per_mtok REAL NOT NULL DEFAULT 0, long_input_per_mtok REAL NOT NULL DEFAULT 0, -- prompt above 100k tokens long_output_per_mtok REAL NOT NULL DEFAULT 0, -- Time-limited introductory (promo) rates. When intro_until is set, usage on -- or before that date (YYYY-MM-DD) is priced at the intro_* rates and usage -- after it at the standard rates. All 0 / NULL = no promo. intro_input_per_mtok REAL NOT NULL DEFAULT 0, intro_output_per_mtok REAL NOT NULL DEFAULT 0, intro_cache_read_per_mtok REAL NOT NULL DEFAULT 0, intro_cache_write_per_mtok REAL NOT NULL DEFAULT 0, intro_cache_write_1h_per_mtok REAL NOT NULL DEFAULT 0, intro_until TEXT, -- promo cutoff YYYY-MM-DD, or NULL source TEXT, -- 'sync' | 'user' | NULL (from before the column) updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); ``` **Columns (highlights):** | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `model_pattern` | TEXT | NO | Primary key. SQL-style glob (e.g. `claude-opus-4-7%`, `claude-%-haiku`). Rules are matched longest-pattern-first | | `display_name` | TEXT | NO | Human-readable model name shown in Settings | | `input_per_mtok` / `output_per_mtok` | REAL | NO | Standard input / output rate per 1M tokens | | `cache_read_per_mtok` / `cache_write_per_mtok` / `cache_write_1h_per_mtok` | REAL | NO | Cache read + 5m/1h cache-write rates | | `fast_input_per_mtok` / `fast_output_per_mtok` | REAL | NO | Fast-mode premium rates (0 = no premium) | | `long_input_per_mtok` / `long_output_per_mtok` | REAL | NO | Rates for a request whose whole prompt (input + cache read + cache write) is above 100k tokens (`LONG_CONTEXT_TOKENS` in `server/lib/pricing-constants.js`); the cache rates scale by `long_input / input`. 0 = none. They price `token_usage`'s `long_*_tokens` subsets — the part of each count spent by such requests, kept at its high-water mark (`stmts.raiseLongTokens`; the exact reconcile path uses `setLongTokens`) | | `intro_*_per_mtok` | REAL | NO | Introductory (promo) rates, mirroring the standard fields | | `intro_until` | TEXT | YES | Promo cutoff `YYYY-MM-DD`. Usage on/before it uses the intro rates; NULL = no promo. Editable per-rule in Settings | | `source` | TEXT | YES | `sync` = kept at LiteLLM's rate by the automatic sync (shown **Auto** in Settings), `user` = edited in Settings and never touched by the sync, NULL = from before this column (adopted as `sync` only while it equals the list) | | `updated_at` | TEXT | NO | ISO8601 timestamp of the last edit | Standard rates and intro rates are edited independently: the pricing update path writes intro columns only when the caller sends intro fields, so a standard-rate edit never disturbs a promo (and vice versa). Clearing `intro_until` also zeroes the intro rates. `model_pricing_dismissed (model_pattern TEXT PRIMARY KEY)` holds the pattern of every rule deleted through `DELETE /api/pricing/:pattern`, so the sync never adds it back; a reset empties it for the scope it reseeds. **Example default rules:** | Pattern | Input | Output | Cache read | Fast in/out | |---------|-------|--------|------------|-------------| | `claude-opus-5%` | $5.00 | $25.00 | $0.50 | $10 / $50 | | `claude-sonnet-5%` | $2.00 | $10.00 | $0.20 | — | | `claude-fable-5-1%` | $10.00 | $50.00 | $0.25 | — | No rule ships an intro rate today: Sonnet 5's launch promo ran through `2026-08-31` and its discounted rate became the standard one, so `DEFAULT_INTRO_PRICING` is empty. `claude-fable-5-1%` exists alongside `claude-fable-5%` only because 5.1 caches cheaper — that overlap is what the longest-pattern-first match rule above resolves. **Retired models.** Claude 3.x, Opus 4/4.1 and Sonnet 4 were dropped from the defaults. Because the startup top-up only *adds* missing rows, those rows are also deleted once from existing databases, guarded by `PRAGMA user_version` (0 → 1) so a model you deliberately re-add in Settings survives later restarts. The same one-shot migration corrects the pre-existing Sonnet 5 row to its post-promo $2 / $10 standard rate. Deleting a rule never touches `token_usage` — only the rates used to price it. --- ### remote_sources Config for remote SSH machines the dashboard pulls Claude Code history from, so a single dashboard can consolidate sessions from several machines. **No secrets are stored** — SSH authentication defers entirely to the host's SSH stack (ssh-agent, `~/.ssh/config`, key files). Each row's `id` is used as the `source` value on every session imported from that machine (see `sessions.source`). ```sql CREATE TABLE remote_sources ( id TEXT PRIMARY KEY, label TEXT NOT NULL, host TEXT NOT NULL, ssh_port INTEGER, identity_file TEXT, remote_home TEXT, enabled INTEGER NOT NULL DEFAULT 1, status TEXT NOT NULL DEFAULT 'idle' CHECK (status IN ('idle','syncing','ok','error')), last_error TEXT, last_sync_at TEXT, last_sync_counts TEXT, created_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), updated_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), -- added by migration (additive): sync_codex INTEGER NOT NULL DEFAULT 0, remote_codex_home TEXT ); ``` **Columns:** | Column | Type | Nullable | Description | |--------|------|----------|-------------| | `id` | TEXT | NO | Primary key. Also used as `sessions.source` for sessions pulled from this machine | | `label` | TEXT | NO | Human-readable name shown in the UI | | `host` | TEXT | NO | SSH destination (`user@host`) or a `~/.ssh/config` alias | | `ssh_port` | INTEGER | YES | Optional SSH port; NULL defers to the SSH default / `~/.ssh/config` | | `identity_file` | TEXT | YES | Optional private-key path passed to ssh (`-i`); NULL to omit | | `remote_home` | TEXT | YES | Optional remote Claude home to read transcripts from; NULL defaults to remote `~/.claude` | | `enabled` | INTEGER | NO | `1` = eligible for scheduled/manual syncs, `0` = disabled (default `1`) | | `sync_codex` | INTEGER | NO | `1` = a sync also mirrors the machine's Codex `sessions/` and reads it through `server/lib/codex-usage.js` into `tool = 'codex'` sessions tagged with this source (default `0`) | | `remote_codex_home` | TEXT | YES | Optional remote CODEX_HOME; NULL = `~/.codex` (`wsl:~/.codex` when `remote_home` is a `wsl:` path) | | `status` | TEXT | NO | Last sync status: `idle`, `syncing`, `ok`, or `error` (CHECK-constrained) | | `last_error` | TEXT | YES | Error message from the last failed sync/test, or NULL | | `last_sync_at` | TEXT | YES | ISO 8601 timestamp of the last successful sync, or NULL | | `last_sync_counts` | TEXT | YES | JSON blob of the last sync's counters (imported/skipped/backfilled/errors/sessions_seen/sessions_tagged, plus codex_files/codex_sessions with Codex on), or NULL | | `created_at` | TEXT | YES | ISO 8601 creation timestamp | | `updated_at` | TEXT | YES | ISO 8601 timestamp of the last edit | Managed through the `/api/remote-sources/*` routes; sync/status changes are broadcast over the WebSocket as `remote_source.status` and, on success, `remote_data.updated` plus per-session `session_created` / `session_updated`. See [docs/API.md → Remote Data Sources](./API.md#remote-data-sources). --- ### token_usage_hourly The same buckets as `token_usage` (session, model, speed, inference_geo, service_tier, the eight count columns and the five `long_*_tokens` subsets, no baselines), split by `hour` — the UTC hour `YYYY-MM-DDTHH` a Claude transcript message was written in. The hook writer fills it from `TranscriptCache`'s `tokensByHour` on every transcript-bearing event (`writeHourly` in `server/lib/hourly-usage.js`, `stmts.upsertHourlyUsage`, MAX per column so a rewritten transcript never takes an hour back), and the server back-fills sessions that have tokens but no hourly rows a few seconds after each start (`backfillHourly`, one transcript per tick). Only the daily split of `GET /api/pricing/cost` reads it; `token_usage` stays the total, and what it holds beyond these rows is booked to the session's start date. Cascades with its session. --- ### codex_rollouts How far `server/lib/codex-usage.js` has read each Codex rollout journal (`$CODEX_HOME/sessions/YYYY/MM/DD/rollout--.jsonl`). A file whose `size` and `mtime_ms` are unchanged is not read again; a grown one is read from `offset` to its last newline, at most 16 MB per pass (`state.more` marks a file with more to read); a shrunken one from 0. There is one row per `thread_id`: the same journal met under a new path (CODEX_HOME moved, or reached through a symlink) replaces its old row instead of being counted twice. Because it lives in the DB, a Codex session the user deleted stays gone until Codex writes to that journal again, while a fresh database re-imports every journal. `POST /api/settings/clear-data` empties this table (and `antigravity_conversations`, `cursor_transcripts`) with the sessions, so a cleared session is imported again whole instead of never, or half on its next write. ```sql CREATE TABLE codex_rollouts ( path TEXT PRIMARY KEY, -- absolute journal path session_id TEXT NOT NULL, -- ROOT session (a sub-agent thread's parent) thread_id TEXT NOT NULL, -- the journal's own thread size INTEGER NOT NULL DEFAULT 0, mtime_ms INTEGER NOT NULL DEFAULT 0, offset INTEGER NOT NULL DEFAULT 0, state TEXT NOT NULL DEFAULT '{}', -- JSON: meta, current model, last total, per-model usage sums updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); CREATE INDEX idx_codex_rollouts_session ON codex_rollouts(session_id); CREATE INDEX idx_codex_rollouts_thread ON codex_rollouts(thread_id); ``` A session's `token_usage` rows are the sum of `state.usage` over every row with its `session_id`, rewritten whole on every change (delete, then `stmts.setTokenUsage`), bucketed `standard`/`global`/`standard`. ### antigravity_conversations Which Antigravity CLI conversation files `server/lib/antigravity-usage.js` has read (`~/.gemini/antigravity-cli/conversations/.db`, agy's own SQLite), at what mtime — the newer of the file's and its non-empty `-wal`'s (an empty one is what AgentHub's own read-only open leaves). A file whose mtime is unchanged is not opened again. One that is not agy's (`SQLITE_NOTADB`, `SQLITE_CORRUPT`, `no such table`) is recorded with `session_id` `''`, so it is tried again only once it changes or at the next server start (`startAntigravityUsagePoll` deletes the `''` rows); any other read failure (a busy file, AgentHub's own error) is recorded nowhere and retried on the next pass; a changed one is re-read whole (they are small) and its session's `token_usage` rewritten whole, per model (`gen_metadata`'s `ModelUsageStats`: input without cache reads, output with thinking, cache read, cache write). As with `codex_rollouts`, a session the user deleted stays gone until agy writes to that conversation again. ```sql CREATE TABLE antigravity_conversations ( path TEXT PRIMARY KEY, -- absolute conversation file path session_id TEXT NOT NULL, -- the file name's id = the hooks' conversationId; '' = not agy's file mtime_ms INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); ``` ### cursor_transcripts Which Cursor agent transcripts `server/lib/cursor-usage.js` has read (`~/.cursor/projects//agent-transcripts//.jsonl`), at what mtime. An unchanged file is not read again (only its `active` session settled once quiet); a changed one rewrites its session row. As with `codex_rollouts`, a session the user deleted stays gone until Cursor writes to that transcript again. Additive: `CREATE TABLE IF NOT EXISTS`. ```sql CREATE TABLE cursor_transcripts ( path TEXT PRIMARY KEY, -- absolute transcript path session_id TEXT NOT NULL, -- the file name's id mtime_ms INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ); ``` ### session_snapshots Frozen copies of a session (`server/routes/snapshots.js`, `POST /api/snapshots`). `payload` is one JSON document — `{version, captured_at, redacted: {file_paths}, session, agents, events, stats, cost}` — read back only by `token`. A trigger refuses every `UPDATE`; `session_id` has no foreign key, so a snapshot outlives its session and survives Clear Data (deleted only by `DELETE /api/snapshots/:id`). Additive: `CREATE TABLE IF NOT EXISTS`. ```sql CREATE TABLE session_snapshots ( id INTEGER PRIMARY KEY AUTOINCREMENT, token TEXT NOT NULL UNIQUE, -- crypto.randomBytes(24) base64url session_id TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')), payload TEXT NOT NULL ); CREATE TRIGGER session_snapshots_immutable BEFORE UPDATE ON session_snapshots BEGIN SELECT RAISE(ABORT, 'session snapshots are immutable'); END; ``` ### report_definitions / report_runs Scheduled reports (`server/lib/reports.js`, `/api/reports`). A definition names the window (`window_days` full local days before the run day), the schedule (`frequency` daily | weekly, `weekday` 0 = Sunday, local `hour`, at `tz_offset` = the browser's `getTimezoneOffset()`), the `formats` (JSON array of `html` / `json`) and `enabled`. `last_slot` is the scheduled instant last claimed — the scheduler claims a slot with one conditional `UPDATE … WHERE last_slot IS NULL OR last_slot < ?`, so of two dashboards on one database only one runs it. Each run is a `report_runs` row; its files are `/reports//.` (`files` lists the formats written). Runs cascade with their definition. Both tables survive Clear Data. Additive: `CREATE TABLE IF NOT EXISTS`. ```sql CREATE TABLE report_definitions ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, window_days INTEGER NOT NULL DEFAULT 7, frequency TEXT NOT NULL DEFAULT 'weekly' CHECK(frequency IN ('daily','weekly')), hour INTEGER NOT NULL DEFAULT 8, weekday INTEGER NOT NULL DEFAULT 1, tz_offset INTEGER NOT NULL DEFAULT 0, formats TEXT NOT NULL DEFAULT '["html"]', enabled INTEGER NOT NULL DEFAULT 1, last_slot TEXT, -- ISO instant of the last claimed slot created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) ); CREATE TABLE report_runs ( id INTEGER PRIMARY KEY AUTOINCREMENT, report_id INTEGER NOT NULL REFERENCES report_definitions(id) ON DELETE CASCADE, created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')), trigger TEXT NOT NULL, -- schedule | manual status TEXT NOT NULL, -- ok | error window_from TEXT, -- local YYYY-MM-DD, inclusive window_to TEXT, stamp TEXT, -- file name stem, made by the server files TEXT NOT NULL DEFAULT '[]', error TEXT ); CREATE INDEX idx_report_runs_report ON report_runs(report_id, id); ``` ### users / auth_sessions Server mode's accounts ([SERVER-MODE.md](SERVER-MODE.md), `server/lib/auth.js`). They are created on every install and stay empty in local mode. | Table | Columns | | --- | --- | | `users` | `id`, `username` (unique, case-insensitive), `role` (`admin`\|`analyst`\|`member`), `password_hash` (`scrypt$$`), `disabled`, `created_at` | | `auth_sessions` | `token_hash` (SHA-256 of the cookie token, PK), `user_id` → `users.id` (cascade), `created_at`, `expires_at` (30 days) | `dashboard_runs.account` names the Claude pool account a run's pane was started on (`server/lib/claude-pool.js`; `NULL` = the machine's own login). The balancer sums each account's spend from the `token_usage_hourly` rows of those runs' sessions. The pool's tokens live in files under `/agenthub-accounts/`, never in the database. `lanes.owner_id` (server mode) is the user who created a workspace; `NULL` for one made in local mode or by the host account. A member sees only sessions of their own workspaces, and everything hanging off them (`server/lib/scope.js`). `audit_log` (`user_id`, `owner_id`, `session_id`, `method`, `path`, `at`) records each admin/analyst read of a session a member owns. `dashboard_runs.pane_token_hash` is the SHA-256 of the pane credential (`AGENTHUB_PANE_TOKEN`) a server-mode pane was started with; it is never returned by any route. `push_subscriptions.user_id` is the user whose browser a subscription is (`NULL` in local mode). Users are never deleted, only disabled. Disabling a user or changing their password deletes their `auth_sessions` rows. ### lane_sources (and `lanes.instructions`) A workspace's own context, as on its home page (`server/lib/lane-sources.js`). `lanes.instructions` (TEXT, nullable, ≤ 16 000 characters) is appended to every new pane's system prompt; `lane_sources` holds its extra **folders** (the user's directories, never deleted by AgentHub) and uploaded context **files** (AgentHub's own copies under `/workspace-context//`, deleted with the row and with the workspace). Both are read at spawn from these rows only (`launchExtras`), never from a request. Both are additive migrations; the rows cascade when the lane row is deleted. ```sql CREATE TABLE lane_sources ( id INTEGER PRIMARY KEY, lane_id INTEGER NOT NULL REFERENCES lanes(id) ON DELETE CASCADE, kind TEXT NOT NULL CHECK (kind IN ('folder', 'file')), path TEXT NOT NULL, -- real path of the folder, or AgentHub's copy of the file name TEXT NOT NULL, -- folder basename / the stored file name size INTEGER, -- bytes, files only added_at TEXT NOT NULL ); CREATE INDEX idx_lane_sources_lane ON lane_sources(lane_id); ``` --- ## Indexes ### sessions Indexes ```sql CREATE INDEX idx_sessions_session_id ON sessions(session_id); CREATE INDEX idx_sessions_status ON sessions(status); CREATE INDEX idx_sessions_updated_at ON sessions(updated_at DESC); CREATE INDEX idx_sessions_source ON sessions(source); -- powers the `sources` query filter -- Partial index covering only the rows the periodic compaction sweep reads: -- active sessions with a known transcript_path. Writes to other sessions skip -- the index entirely, so the maintenance cost stays bounded by the small set -- of live sessions. CREATE INDEX idx_sessions_active_tp ON sessions(status, transcript_path) WHERE status='active' AND transcript_path IS NOT NULL; ``` **Query Patterns:** - `SELECT * FROM sessions WHERE session_id = ?` - Primary key lookup - `SELECT * FROM sessions WHERE status = 'active'` - Filter by status - `SELECT * FROM sessions WHERE source IN ('local', ?)` - Filter by data source (covered by `idx_sessions_source`) - `SELECT * FROM sessions ORDER BY updated_at DESC LIMIT 50` - Recent sessions - `SELECT id, transcript_path FROM sessions WHERE status='active' AND transcript_path IS NOT NULL ORDER BY updated_at DESC` — periodic compaction sweep (covered by the partial index above) ### agents Indexes ```sql CREATE INDEX idx_agents_agent_id ON agents(agent_id); CREATE INDEX idx_agents_session_id ON agents(session_id); CREATE INDEX idx_agents_status ON agents(status); ``` **Query Patterns:** - `SELECT * FROM agents WHERE agent_id = ?` - Primary key lookup - `SELECT * FROM agents WHERE session_id = ?` - All agents for session - `SELECT * FROM agents WHERE status = 'running'` - Active agents ### events Indexes ```sql -- Keeps the per-tool-event dedup used by subagent import an index seek instead -- of a full events scan. importSubagentFromJsonl checks -- `... WHERE agent_id = ? AND event_type = ? AND data LIKE '%"tool_use_id":"X"%'` -- before inserting; on a subagent-heavy re-import this drops a large sweep from -- tens of seconds to sub-second. CREATE INDEX idx_events_agent_type ON events(agent_id, event_type); -- A session's events in insert order. NOT a prefix of idx_events_session_type_tool -- (not id-ordered): every hook's duplicate guard reads a session's latest event -- through it. Dropping it (ad6e801) made that read sort the whole session per -- hook and made a correlated "next tool event" query quadratic — minutes on a -- 29k-event session, the event loop frozen for every page. CREATE INDEX idx_events_session ON events(session_id, id); -- Per-tool counts (analytics, workflows) and the events filter's tool list -- read this alone instead of every event row (~300 ms -> ~20 ms at 233k rows). CREATE INDEX idx_events_tool ON events(tool_name); -- A session's events by type and tool, covering: session stats (per-tool and -- per-type counts), the hook's TurnDuration/APIError dedupe (one read per hook) -- and the import's newest-per-type resume point read no event row -- (1.2 s -> 20 ms on a 29K-event session). Replaces idx_events_session_type -- (session_id, event_type), dropped at startup, so a hook writes no more index -- entries than before. CREATE INDEX idx_events_session_type_tool ON events(session_id, event_type, tool_name, created_at); ``` ### tool_executions Indexes ```sql CREATE INDEX idx_tools_agent_id ON tool_executions(agent_id); CREATE INDEX idx_tools_created_at ON tool_executions(created_at DESC); ``` **Query Patterns:** - `SELECT * FROM tool_executions WHERE agent_id = ?` - All tools for agent - `SELECT * FROM tool_executions ORDER BY created_at DESC LIMIT 100` - Recent tools ### notifications Indexes ```sql CREATE INDEX idx_notifications_session_id ON notifications(session_id); ``` **Query Patterns:** - `SELECT * FROM notifications WHERE session_id = ?` - All notifications for session --- ## Migrations Every migration in `server/db.js` runs at module load as a check-then-act (probe a column, `ALTER` when missing; rebuild a table whose old constraint is still there). Two dashboards may start on one database at the same instant, so the whole section — the legacy-DB copy, `CREATE TABLE IF NOT EXISTS`, every probe and rebuild, the pricing top-up — runs under a cross-process lock: a `BEGIN IMMEDIATE` on the separate file `-migrate.lock` (busy timeout 120 s), committed just before the prepared statements are built. The lock is a separate file because several rebuilds open their own `BEGIN` and switch `foreign_keys` off, which a surrounding transaction on the main database would break. A process that dies holding it releases it with its file handle. ### Schema Versioning ```mermaid graph TB V1[Version 1
Initial schema] --> V2[Version 2
Add updated_at] V2 --> V3[Version 3
Add pricing_rules] V3 --> VN[Version N
Future migrations] style V1 fill:#3B82F6 style V2 fill:#10B981 style V3 fill:#F59E0B ``` ### Migration Strategy ```javascript // db.js - Schema versioning const SCHEMA_VERSION = 3; function runMigrations() { const currentVersion = db.pragma('user_version', { simple: true }); if (currentVersion < 1) { // Initial schema db.exec(` CREATE TABLE sessions (...); CREATE TABLE agents (...); -- etc. `); db.pragma('user_version = 1'); } if (currentVersion < 2) { // Add updated_at column db.exec(`ALTER TABLE sessions ADD COLUMN updated_at TEXT DEFAULT (datetime('now'))`); db.pragma('user_version = 2'); } if (currentVersion < 3) { // Add pricing_rules table db.exec(`CREATE TABLE pricing_rules (...)`); db.pragma('user_version = 3'); } } ``` ### Migration Workflow ```mermaid sequenceDiagram participant App participant DB participant Migrations App->>DB: Open connection DB->>Migrations: Check PRAGMA user_version Migrations->>Migrations: Compare with SCHEMA_VERSION alt Version mismatch Migrations->>DB: Run migration scripts DB->>Migrations: Success Migrations->>DB: Update user_version else Version match Migrations->>App: Ready end App->>DB: Application queries ``` --- ## Query Patterns ### Common Queries #### List Recent Sessions ```sql SELECT s.*, COUNT(DISTINCT a.id) as agent_count, COUNT(DISTINCT t.id) as tool_count FROM sessions s LEFT JOIN agents a ON s.session_id = a.session_id LEFT JOIN tool_executions t ON a.agent_id = t.agent_id GROUP BY s.id ORDER BY s.updated_at DESC LIMIT 50; ``` **Performance:** ~5-10ms (with indexes) #### Get Session with Agents ```sql SELECT * FROM sessions WHERE session_id = 'sess_abc123'; SELECT * FROM agents WHERE session_id = 'sess_abc123'; ``` **Performance:** ~1-2ms per query #### Get Agent Tools ```sql SELECT * FROM tool_executions WHERE agent_id = 'agent_xyz789' ORDER BY created_at DESC; ``` **Performance:** ~2-5ms #### Calculate Total Cost ```sql SELECT SUM(cost) as total_cost FROM agents WHERE session_id = 'sess_abc123'; ``` **Performance:** ~1-2ms ### Query Optimization ```mermaid graph TB Query[SQL Query] --> Explain[EXPLAIN QUERY PLAN] Explain --> Scan{Full Table
Scan?} Scan -->|Yes| AddIndex[Add Index] Scan -->|No| Check{Query Time
>10ms?} AddIndex --> Retest[Re-test Query] Retest --> Check Check -->|Yes| Optimize[Optimize Query
Rewrite, Denormalize] Check -->|No| Done[Acceptable Performance] style AddIndex fill:#F59E0B style Optimize fill:#EF4444 style Done fill:#10B981 ``` --- ## Performance Optimization ### SQLite Pragmas What `server/db.js` actually sets on every open: ```javascript db.pragma("journal_mode = WAL"); // readers never block the writer db.pragma("foreign_keys = ON"); db.pragma("busy_timeout = 5000"); // two dashboards may share the file db.pragma("synchronous = NORMAL"); // safe with WAL: a power cut loses only the last commits, never the file db.pragma("journal_size_limit = 67108864"); // a checkpoint truncates the WAL back to 64 MB db.pragma("cache_size = -65536"); // 64 MB page cache (default 2 MB) db.pragma("mmap_size = 268435456"); // memory-mapped reads, 256 MB ``` The read worker (`server/lib/read-pool.js`) sets the same `cache_size`/`mmap_size`. **Checkpoints.** The running server sets `wal_autocheckpoint = 0` on its main connection and checkpoints every `DASHBOARD_WAL_CHECKPOINT_MS` (default 5000, `0` = SQLite's automatic checkpoint as before) with `wal_checkpoint(PASSIVE)` from the read worker's own writable connection. SQLite's automatic checkpoint runs inside whichever `COMMIT` crosses 1000 WAL pages — a hook's, on the event loop — and measured up to 2.7 s per commit under load. A failed checkpoint switches the automatic one back on. A second process on the same file (another dashboard, the CLI) keeps its own setting. **Upkeep.** `maintainDatabase()` (exported by `server/db.js`) runs hourly from `server/index.js` (`DASHBOARD_DB_MAINTAIN_MS`, `0` = off): a `wal_checkpoint(PASSIVE)` — it never waits on a reader, such as the other dashboard — after which the next write shrinks the WAL to `journal_size_limit`, and `PRAGMA optimize`. Then `runScheduledMaintenance` (`server/lib/db-retention.js`) applies the OPT-IN retention (`AGENTHUB_RETENTION_DAYS`, off by default: finished sessions older than N days, purged 25 per transaction like the manual cleanup) and, when the database is in `auto_vacuum = INCREMENTAL` and more than 1024 pages are free, a bounded `incremental_vacuum` (1024-page slices, ≤ 65536 pages per run) that returns them to the disk. A new database is created INCREMENTAL; an existing one up to 32 MB is converted by one `VACUUM` at startup (`convertAutoVacuum`), a larger one only by Settings → Data → **Compact** (`POST /api/settings/compact`: a full `VACUUM` in a worker thread, write lock held while it runs). No automatic path runs a full `VACUUM` on a large file. See docs/API.md § Retention and Compact. **Growth.** `events.data` is the hook payload TRIMMED by `trim_event_data(?)` (`server/lib/event-payload.js`, a SQL function registered on the connection in `server/db.js`, used by every event `INSERT` and by the duplicate guard): Edit/MultiEdit/NotebookEdit/Write drop `tool_response.originalFile`/`oldString`/`newString` (Write also `content`), Read drops `tool_response.file.content`/`base64`, and any string inside `tool_input`/`tool_response` over 8 KB keeps its first 6 KB and last 2 KB; `structuredPatch` and every field outside those two are kept, and `_trimmed` lists what was dropped. The full text stays in the session transcript and AgentHub's snapshot of it under `/transcripts/`, which the Conversation tab reads. Rows stored before this are trimmed once in the background (`sweepEventPayloads`, 200 rows per transaction, `user_version` 4 after marker 3); measured on a 1.2 GB install it took `events.data` from 913 MB to ~400 MB. Freed pages are reused, not returned to the disk (no `VACUUM`). No ROW is pruned automatically. The only deletions are the user's: Settings → clear data / cleanup, a workspace's purge, `scripts/clear-data.js`. Clear data runs in ONE transaction (foreign keys off around it, so workflows are deleted explicitly), cleanup deletes in 500-id chunks inside one transaction. `GET /api/settings/export` streams `events` in keyset pages of 500 instead of loading the table. ### Prepared Statements ```javascript // db.js - Prepared statements prevent SQL injection + optimize performance const stmts = { findSession: db.prepare('SELECT * FROM sessions WHERE session_id = ?'), createSession: db.prepare('INSERT INTO sessions (session_id, model) VALUES (?, ?)'), updateSession: db.prepare('UPDATE sessions SET status = ?, total_cost = ? WHERE session_id = ?'), touchSession: db.prepare("UPDATE sessions SET updated_at = datetime('now') WHERE session_id = ?") }; // Usage const session = stmts.findSession.get('sess_abc123'); stmts.touchSession.run('sess_abc123'); ``` ### Transaction Batching ```javascript // Batch multiple writes in a transaction const insertMany = db.transaction((tools) => { for (const tool of tools) { stmts.createToolExecution.run(tool.agent_id, tool.tool_name, tool.duration_ms); } }); insertMany([ { agent_id: 'agent_1', tool_name: 'bash', duration_ms: 100 }, { agent_id: 'agent_1', tool_name: 'view', duration_ms: 50 }, // ... more tools ]); ``` ### Performance Benchmarks | Operation | Without Optimization | With Optimization | Improvement | |-----------|---------------------|-------------------|-------------| | Session list (50) | 25ms | 5ms | 5x faster | | Hook processing | 15ms | 2ms | 7.5x faster | | Batch insert (100 tools) | 500ms | 50ms | 10x faster | --- ## Data Integrity ### Foreign Key Constraints ```sql -- Enabled by default in db.js PRAGMA foreign_keys = ON; ``` **Constraint Enforcement:** ```mermaid graph TB Insert[INSERT agent] --> Check{session_id exists?} Check -->|Yes| Allow[Insert Allowed] Check -->|No| Reject[FOREIGN KEY constraint failed] Delete[DELETE session] --> Cascade{Cascade enabled?} Cascade -->|Yes| DeleteChildren[Delete agents and tools] Cascade -->|No| BlockDelete[Cannot delete FK exists] style Allow fill:#10B981 style Reject fill:#EF4444 style DeleteChildren fill:#F59E0B ``` ### Data Validation ```javascript // Validate before insert function validateSession(session) { if (!session.session_id) throw new Error('session_id required'); if (session.total_cost < 0) throw new Error('total_cost must be >= 0'); if (!['active', 'completed'].includes(session.status)) { throw new Error('Invalid status'); } } ``` --- ## Backup Strategies ### Built-in daily backup (Settings → Data → Daily backup) Set a folder — best on another disk — and the server writes `agenthub-YYYYMMDD-HHMMSS.db` there once a day at a chosen local hour (default 02:00), keeping the newest 7 (`server/lib/db-backup.js`). It is the `VACUUM INTO` below, run on a read-only connection in a worker thread, so the dashboard keeps serving during the copy. Rotation deletes only those exact file names. To restore: `agenthub stop`, delete `dashboard.db-wal` and `dashboard.db-shm`, copy a backup over `dashboard.db`, `agenthub start`. Routes and settings: [API.md → Daily database backup](./API.md#daily-database-backup). ### Online Backup (Recommended) ```sql -- Using VACUUM INTO (SQLite 3.27+) VACUUM INTO '/backups/dashboard_20240318.db'; ``` ### Offline Backup ```bash #!/bin/bash # Stop application systemctl stop agenthub # Copy database file cp /var/lib/agenthub/dashboard.db /backups/dashboard_$(date +%Y%m%d).db # Start application systemctl start agenthub ``` --- ## Summary The database schema provides: - ✅ **Normalized design** - Minimal redundancy, clear relationships - ✅ **Performance optimized** - Indexes, prepared statements, WAL mode - ✅ **Data integrity** - Foreign keys, constraints, transactions - ✅ **Migration support** - Schema versioning with PRAGMA user_version - ✅ **Comprehensive indexing** - Fast queries for common access patterns - ✅ **Backup strategies** - Online + offline backup options For API usage, see [docs/API.md](./API.md).