perf: hourly update check only with a browser open; opt-in retention, incremental auto-vacuum and Compact

- update-scheduler: default interval 1 h (DASHBOARD_UPDATE_CHECK_INTERVAL_MS and its 60 s floor kept);
  a scheduled tick with no /ws client is skipped, and the next browser to connect gets one catch-up
  check when the last one is stale (websocket.js onConnection). Manual check unchanged.
- db-retention.js: shared session purge (used by POST /api/settings/cleanup too), opt-in
  AGENTHUB_RETENTION_DAYS applied by the hourly maintenance in 25-session transactions with yields
  (max 5000/run), bounded incremental_vacuum when the freelist is large, and a manual Compact
  (VACUUM in a worker, converting to auto_vacuum=INCREMENTAL).
- db.js: new databases are INCREMENTAL; an existing one up to 32 MB is converted at startup.
- GET/PUT /api/settings/retention, POST /api/settings/compact (confirm: true); Settings → Data →
  Session cleanup gets the Automatic cleanup control and Compact now (en + vi).
This commit is contained in:
2026-10-16 10:20:38 +07:00
parent ecbbb4dd8f
commit e07993c536
21 changed files with 1323 additions and 58 deletions
+10
View File
@@ -3057,6 +3057,16 @@ POST /api/settings/backup/run 202 — started (or already running); poll GE
Settings → Data → **Daily backup** (`server/lib/db-backup.js`). Once a day, at `hour` (0–23, the server's local time, default 2), the database is copied to `dir` as `agenthub-YYYYMMDD-HHMMSS.db`: `VACUUM INTO` on a read-only connection in a worker thread, so the copy is a consistent snapshot (writes from another dashboard on the same database never restart it), compacted, and the event loop keeps serving while it runs (≈30 s for a 1.25 GB database). It is written as `<name>.partial` and renamed when complete. Beyond `keep` (1–365, default 7) the oldest are deleted — only regular files matching that exact name; nothing else in the folder, and no symlink, is ever touched. A day missed while the server was down runs at the next start; a failed run is retried the next day or by **Back up now**, never every minute. The three values persist to the settings file as `AGENTHUB_BACKUP_DIR` / `AGENTHUB_BACKUP_HOUR` / `AGENTHUB_BACKUP_KEEP`; one the environment set is `env_owned` (`409 EENVOWNED`). Writes are same-origin, and admin-only in server mode. `400 INVALID_PATH` for a relative, missing, non-directory or unwritable folder, `400 INVALID_INPUT` for an hour or count out of range, `400 ENOBACKUPDIR` for a run with no folder. The scheduler beat is `DASHBOARD_BACKUP_TICK_MS` (default 60000, `0` = off).
Restoring is manual, on purpose: `agenthub stop`, delete `dashboard.db-wal` and `dashboard.db-shm`, copy a backup over `dashboard.db` (its path is on Settings → System), `agenthub start`.
### Retention and Compact
```
GET /api/settings/retention { days, max_days, env_owned: { days }, last_run, space: { auto_vacuum, page_size, page_count, freelist_pages, free_bytes }, db_size, compacting, last_compact }
PUT /api/settings/retention { days } → the same state; 0 turns it off (the default)
POST /api/settings/compact { confirm: true } → 202 — started (or already running); poll GET while `compacting`
```
Settings → Data → **Session cleanup** (`server/lib/db-retention.js`). Automatic retention is OFF unless `days` (1–3650, `AGENTHUB_RETENTION_DAYS` in the settings file) is set; nothing is deleted by default. When set, the hourly maintenance (`DASHBOARD_DB_MAINTAIN_MS`) purges finished (completed/error/abandoned) sessions that started more than `days` ago — the same rule and the same deletion (`purgeSessionIds`: events, agents, token usage, the session) as `POST /api/settings/cleanup` with `purge_days` — 25 sessions per transaction with a yield between, at most 5000 per run. A new database is created with `auto_vacuum = INCREMENTAL`, an existing one up to 32 MB is converted by a VACUUM at startup; the same hourly run then returns free pages to the OS (`incremental_vacuum`, in 1024-page slices, at most 65536 pages per run) once more than 1024 are free. A larger, older database is converted only by **Compact**: a full VACUUM in a worker thread followed by a TRUNCATE checkpoint — it holds the write lock while it runs (tens of seconds on a GB-size file), so hooks wait meanwhile; it runs only on a click. In server mode both writes are an admin's (`/settings` is in `ADMIN_WRITE_PREFIXES`).
---
## WebSocket API
+1 -1
View File
@@ -915,7 +915,7 @@ worker's own writable connection. SQLite's automatic checkpoint runs inside whic
per commit under load. A failed checkpoint switches the automatic one back on. A second
process on the same file (another dashboard, the CLI) keeps its own setting.
**Upkeep.** `maintainDatabase()` (exported by `server/db.js`) runs hourly from `server/index.js` (`DASHBOARD_DB_MAINTAIN_MS`, `0` = off): a `wal_checkpoint(PASSIVE)` — it never waits on a reader, such as the other dashboard — after which the next write shrinks the WAL to `journal_size_limit`, and `PRAGMA optimize`. Nothing runs `VACUUM`: it rewrites the whole file under an exclusive lock, so space freed by a cleanup is reused for new rows instead of returned to the disk.
**Upkeep.** `maintainDatabase()` (exported by `server/db.js`) runs hourly from `server/index.js` (`DASHBOARD_DB_MAINTAIN_MS`, `0` = off): a `wal_checkpoint(PASSIVE)` — it never waits on a reader, such as the other dashboard — after which the next write shrinks the WAL to `journal_size_limit`, and `PRAGMA optimize`. Then `runScheduledMaintenance` (`server/lib/db-retention.js`) applies the OPT-IN retention (`AGENTHUB_RETENTION_DAYS`, off by default: finished sessions older than N days, purged 25 per transaction like the manual cleanup) and, when the database is in `auto_vacuum = INCREMENTAL` and more than 1024 pages are free, a bounded `incremental_vacuum` (1024-page slices, ≤ 65536 pages per run) that returns them to the disk. A new database is created INCREMENTAL; an existing one up to 32 MB is converted by one `VACUUM` at startup (`convertAutoVacuum`), a larger one only by Settings → Data → **Compact** (`POST /api/settings/compact`: a full `VACUUM` in a worker thread, write lock held while it runs). No automatic path runs a full `VACUUM` on a large file. See docs/API.md § Retention and Compact.
**Growth.** `events.data` is the hook payload TRIMMED by `trim_event_data(?)` (`server/lib/event-payload.js`, a SQL function registered on the connection in `server/db.js`, used by every event `INSERT` and by the duplicate guard): Edit/MultiEdit/NotebookEdit/Write drop `tool_response.originalFile`/`oldString`/`newString` (Write also `content`), Read drops `tool_response.file.content`/`base64`, and any string inside `tool_input`/`tool_response` over 8 KB keeps its first 6 KB and last 2 KB; `structuredPatch` and every field outside those two are kept, and `_trimmed` lists what was dropped. The full text stays in the session transcript and AgentHub's snapshot of it under `<data dir>/transcripts/`, which the Conversation tab reads. Rows stored before this are trimmed once in the background (`sweepEventPayloads`, 200 rows per transaction, `user_version` 4 after marker 3); measured on a 1.2 GB install it took `events.data` from 913 MB to ~400 MB. Freed pages are reused, not returned to the disk (no `VACUUM`). No ROW is pruned automatically. The only deletions are the user's: Settings → clear data / cleanup, a workspace's purge, `scripts/clear-data.js`. Clear data runs in ONE transaction (foreign keys off around it, so workflows are deleted explicitly), cleanup deletes in 500-id chunks inside one transaction. `GET /api/settings/export` streams `events` in keyset pages of 500 instead of loading the table.