feat(health): System Health in three groups, a score that sees lag; admin user detail

Dashboard → System Health is grouped into AgentHub, Machine and Agent
activity. The machine (GET /api/host, every signed-in user) is drawn by
components/health/HostHealth.tsx: host card beside the CPU/memory history,
then CPU (threads as compact cells + temperatures/GPUs), disks & network,
busiest processes, and per-user tiles. Concurrency bars each show their own
agent type on hover.

The Health Score (client/src/lib/healthScore.ts) is now 100 minus what each
check costs, between an amber and a red threshold: event-loop p99
(server/lib/loop-delay.js, shared with /api/metrics), Linux PSI pressure for
CPU, memory and disk I/O (/api/host `pressure`; load per core without PSI),
RAM, the database's disk, V8 heap, hooks silent while agents work, and agent
errors in the last 24 h. All-time success rate, API errors and the transcript
cache hit rate no longer count. /api/settings/info adds server.event_loop and
activity {last_event_at, working_agents, agents_24h}.

Administration: a user row opens its detail in place (GET
/api/users/:id/details — spend, account pool share, live panes, workspaces,
newest sessions); the server panel is gone, member tmux folded into the table;
the Sessions list shows each row's owner in server mode.

docs/API.md

@@ -89,7 +89,7 @@ A dashboard shared by several people runs in **server mode** ([SERVER-MODE.md](S
 | `POST /api/auth/login` `{username, password}` | `{user}` + the cookie; `401 EBADLOGIN`, `429 ETOOMANY`, `404 ENOTSERVER` in local mode |
 | `POST /api/auth/logout` | `{ok: true}`, cookie cleared |
 
-In server mode a **developer** sees only rows of the workspaces they own (`lanes.owner_id`): every list, count, chart and cost answer is limited to them, and a request naming another user's session, agent, workspace or run (in the path, query or body) answers `404 ENOTFOUND`. Admin and analyst see everything; `GET /api/users` lists every account (`displayName`, `email`, `createdAt`, `logins`, `lastLoginAt` — the last login ever, kept after it ends —, `mustChangePassword` included) with its workspaces, sessions and `today` (since the viewer's local midnight — `?tz_offset=` is the browser's `getTimezoneOffset()`, UTC without it) / 7 / 30-day `{cost, tokens}`, `GET /api/users/storage[?refresh=1]` each user's home size (`{storage: {<id>: {bytes, files, partial, measuredAt} | null}, measuring}` — measured in the background, see [SERVER-MODE.md](SERVER-MODE.md)), `GET /api/users/tmux` each member's own tmux server under one Linux user per account (`{available, installed, matched, users: [{id, username, running, stuck?, version, pid, startedAt, sessions, busy, outdated}]}`; an admin's `POST /api/users/:id/tmux-restart` and `POST /api/users/tmux-restart-idle` restart them with `currentPassword`, audited, `409 ETMUXSTUCK` for a server that will not stop), `GET /api/host` the machine and each user's live share of it (`{at, hostname, os, uptime, cpu: {model, cores, load}, memory: {total, available, used, swapTotal, swapUsed}, disks: [{device, mount, type, kind: "HDD"|"SSD"|null, total, used, free}], homesMount, gpus: [{name, utilization, memoryUsed, memoryTotal}], usage: {cpu, sampleMs, groups: [{key, kind: "user"|"host"|"system", userId?, username?, cpu, memory, memoryPercent, processes, top: [{pid, name, cpu, memory}]}]} | null}` — read-only, cached 3 s, `usage` null off Linux), and `GET /api/users/audit[?limit=]` the staff reads of developers' sessions and admins' account changes (kept `AGENTHUB_AUDIT_DAYS` days, default 180, `0` = forever) (`reader`/`owner` are usernames, `reader_display`/`owner_display` their display names or null) (all `403 EADMIN` for a developer). See [SERVER-MODE.md](SERVER-MODE.md#what-each-user-sees).
+In server mode a **developer** sees only rows of the workspaces they own (`lanes.owner_id`): every list, count, chart and cost answer is limited to them, and a request naming another user's session, agent, workspace or run (in the path, query or body) answers `404 ENOTFOUND`. Admin and analyst see everything; `GET /api/users` lists every account (`displayName`, `email`, `createdAt`, `logins`, `lastLoginAt` — the last login ever, kept after it ends —, `mustChangePassword` included) with its workspaces, sessions and `today` (since the viewer's local midnight — `?tz_offset=` is the browser's `getTimezoneOffset()`, UTC without it) / 7 / 30-day `{cost, tokens}`, `GET /api/users/storage[?refresh=1]` each user's home size (`{storage: {<id>: {bytes, files, partial, measuredAt} | null}, measuring}` — measured in the background, see [SERVER-MODE.md](SERVER-MODE.md)), `GET /api/users/tmux` each member's own tmux server under one Linux user per account (`{available, installed, matched, users: [{id, username, running, stuck?, version, pid, startedAt, sessions, busy, outdated}]}`; an admin's `POST /api/users/:id/tmux-restart` and `POST /api/users/tmux-restart-idle` restart them with `currentPassword`, audited, `409 ETMUXSTUCK` for a server that will not stop), `GET /api/host` the machine and each user's live share of it (`{at, hostname, os, uptime, cpu: {model, cores, load}, memory: {total, available, used, swapTotal, swapUsed}, disks: [{device, mount, type, kind: "HDD"|"SSD"|null, total, used, free}], homesMount, gpus: [{name, utilization, memoryUsed, memoryTotal}], usage: {cpu, sampleMs, groups: [{key, kind: "user"|"host"|"system", userId?, username?, cpu, memory, memoryPercent, processes, top: [{pid, name, cpu, memory}]}]} | null, bootTime, network: [{name, rx, tx}], diskIo: [{name, read, write}], temperatures: [{chip, label, celsius}], pressure: {cpu, memory, io: {some10, some60, full10} | null} | null (Linux PSI, `/proc/pressure/*`: % of the last 10 / 60 s tasks stalled on it), topProcesses: [{pid, name, cpu, memory, group}], process: {pid, node, uptime, rss, heapUsed}}`, `cpu` also carrying `physicalCores` and `perCore` (busy % per thread), rates in bytes/s over the sample — read-only, cached 3 s, `usage` null and the lists empty off Linux), `GET /api/host/history?range=1h|6h|24h` the CPU/memory history every server on Linux keeps in memory since it started (both read by the Dashboard's System Health tab and open to every signed-in user — `/host` is not staff-only) (`{range, intervalMs, running, startedAt, points: [{at, cpu, memory, memoryUsed, swapUsed, load1, users: {<id>: {cpu, memory}}}]}`, one point per 30 s, averaged to ≤ 360), `GET /api/users/:id/details[?tz_offset=]` one account in full for the Administration page's user detail (`{user (with logins, lastLoginAt), sessions: {total, active, firstAt, lastAt}, usage: {today, last7d, last30d, daily: [{date, cost, tokens}] (30 local days, oldest first), byTool, byModel: [{key, cost, tokens, sessions}]}, workspaces: [{id, title, cwd, tool, kind, branch, status, createdAt, sessions, lastActiveAt, livePanes}], panes: [{id, laneId, laneTitle, laneCwd, tool, model, account, proxy, sessionId, startedAt, status}], recentSessions (15, with cost/tokens), pool: {claude|codex|cursor: {pin, group, accounts: [{name, kind, email, expired, allowed, pinned, quota, quotaPercent, full, livePanes, spend5h, spend7d, userSpend5h, userSpend7d, userTokens5h, userTokens7d, userRuns7d, userLivePanes, userLastAt}]}}, audit}` — `user*` fields are only the runs on that user's workspaces; never a token; `404 ENOUSER`), and `GET /api/users/audit[?limit=]` the staff reads of developers' sessions and admins' account changes (kept `AGENTHUB_AUDIT_DAYS` days, default 180, `0` = forever) (`reader`/`owner` are usernames, `reader_display`/`owner_display` their display names or null) (all `403 EADMIN` for a developer). See [SERVER-MODE.md](SERVER-MODE.md#what-each-user-sees).
 
 An admin changes accounts with `POST /api/users` (`{username, password, role}` → `201 {user}`; the password may be any non-empty starting one and the account gets `mustChangePassword: true`), `GET /api/users/:id` (`{user, workspaces: [{id, title, cwd}]}`, the account with `logins` and `lastLoginAt`, for the Edit form; admin or analyst), `GET /api/users/:id/avatar` (that account's picture for the sidebar's owner groups — admin or analyst, `nosniff` + `sandbox` CSP, `404 ENOAVATAR`; a workspace in `GET /api/lanes` carries `owner_avatar_at` to version its URL), `PATCH /api/users/:id` (any of `{displayName?, email?, role?, disabled?, password?, addWorkspaces?: [laneId], removeWorkspaces?: [laneId], signOut?: true}` — every field validated before any is written; `removeWorkspaces` leaves the workspace with no owner and answers `409 ENOTOWNED` for one the user does not own; an admin's own `password`/`signOut` keeps the login it came from; a password for SOMEONE ELSE may be anything non-empty and sets their `mustChangePassword`, the admin's OWN must pass every rule below, `400 EWEAKPASSWORD` with `rules`), `POST /api/users/:id/password` (`{password}`; that user's logins end; the same two bars) `POST /api/users/:id/workspaces` (`{laneId}`; the workspace's owner becomes that user) and `DELETE /api/users/:id` (delete for good: the user's logins, panes, workspaces' records, prefs and — under S3 — their Linux account and whole home, removed FIRST so a refusal (`502 EPROVISION`) leaves every row; `409 ESELF` for yourself, `409 ELASTADMIN` for the last admin; session history and the audit log stay). No path — this API or the `agenthub users` CLI — may demote, disable or delete the last enabled admin (`409 ELASTADMIN`). Every one carries `currentPassword`, the admin's own password: `403 EREAUTH` when it is wrong or missing, `429 ETOOMANY` after 10 wrong tries in 15 minutes. `409 ESELF` for an admin's own role or disable, `409 EEXISTS` for a taken name, `400 EBADUSER`/`EBADPASSWORD`/`EBADROLE`/`EBADEMAIL`/`EBADFIELD`, `404 ENOUSER`/`ENOLANE`; a developer or analyst gets `403 EADMIN`, and local mode `403 ELOCALMODE`. No answer carries a password hash; each change is written to the audit log.
 
@@ -117,7 +117,7 @@ An admin (server mode) adds an account with `POST /api/accounts` `{tool: "claude
 
 The keys are the browser's own localStorage names: `theme`, `agenthub.theme.dark`, `agenthub.theme.light`, `i18nextLng`, `agenthub.terminal.font`, `agenthub.terminal.fontSize`, `agenthub.ui.fontSize`, `agenthub.pet`, `agenthub.pet.size`, `agenthub.pet.names`, `agent-dashboard-sagi-enabled`, `agent-dashboard-sagi-muted`, `agent-monitor-notifications`, `agenthub.defaultTool`, `agenthub.permissionMode` (`bypassPermissions` allowed — set only from Settings → AI tools → Session defaults, behind a confirm), `agenthub.codex.sandbox`, `agenthub.codex.approval`, `agenthub.codex.bypass` (`"true"`/`"false"`: a new Codex session's defaults), `agenthub.tour`, and `agenthub.chatUi` (`"1"` = Claude/Codex sessions open as a chat; read and written on the server only, by `client/src/lib/chatUi.ts`). The request never names an account, and writes are same-origin. An account's save writes only their own row, and under S3 a `theme` of `dark` or `light` also becomes the `theme` in their own `~/.claude/settings.json`, written as them.
 
-`GET /api/settings/info` includes `server.version` (the running dashboard release). Pair with `agenthub version` or Settings → System to confirm client and server builds match after deploy.
+`GET /api/settings/info` includes `server.version` (the running dashboard release). It also carries what the Dashboard's System Health score reads beyond `/api/host`: `server.event_loop` `{p50, p99, max}` (event-loop delay over the last minute, ms — the same figures as `agenthub_event_loop_delay_seconds` in `/api/metrics`) and `activity` `{last_event_at, working_agents, agents_24h: {completed, error}}` (newest event, agents working now, agents that ended in the last 24 h — scoped to the caller's rows). Pair with `agenthub version` or Settings → System to confirm client and server builds match after deploy.
 
 ```mermaid
 sequenceDiagram
@@ -1866,6 +1866,10 @@ classDiagram
         +string pinned_at "null unless pinned"
         +number cost
         +number agent_count
+        +number owner_id "server mode: workspace owner; null if none"
+        +string owner_name "server mode: owner's username"
+        +string owner_display_name "server mode"
+        +string owner_avatar_at "server mode"
     }
     
     SessionListResponse --> Session

docs/SERVER-MODE.md

@@ -166,20 +166,48 @@ Workspaces made before server mode, or by the host account, have none.
   sessions, and the tokens and $ it used today (the browser's local day) and in the last 7 and 30 days
   (booked like the cost routes' daily split, so Codex/omp/Antigravity and sub-agent usage count) — the
   **Administration** page in the sidebar (`/admin`; the old `/users` link
-  redirects there) shows it on its **Users** tab, with the audit log under it.
-- `GET /api/host` (admin/analyst, `/host` is staff-only) is Administration's
-  **Server** tab: the machine's hostname, OS, uptime, CPU model, cores and
+  redirects there) shows it — one page, no tabs: the users table, the account
+  pool, then the audit log. The machine itself is on the Dashboard's
+  **System Health** tab (below).
+  The table's spend column is today's (the browser's local day), with 7 and 30
+  days under it. Clicking a user expands a short detail right under their row
+  (`/admin?user=<id>`, `GET /api/users/:id/details`): last sign-in, last
+  activity and what runs now, then spend over 30 days by AI tool and top
+  models, the account pool per tool (pin or group, each account's quota left
+  beside their own 7-day share of its spend), their live panes and
+  workspaces, and their five newest sessions.
+- `GET /api/host` (every signed-in user — `/host` is deliberately NOT
+  staff-only, so a developer also sees other users' usernames, CPU / RAM share
+  and busiest process names) is the machine on the Dashboard's **System
+  Health** tab — its **Machine** group, between **AgentHub** (score, storage,
+  concurrency) and **Agent activity** (tokens by model, tools, subagents): the machine's hostname, OS, uptime, CPU model, cores and
   load, memory and swap, every real disk (HDD or SSD, how full; the one
-  holding the users' homes is marked), any NVIDIA GPU, and each user's live
-  share of it — CPU % of the whole machine (100 % = every core busy) and
+  holding the users' homes is marked), any NVIDIA GPU. Each user's live
+  share of it is a **Users · CPU / RAM now** card on the same tab (a tile per
+  user, their busiest processes on click; their home share stays in
+  Administration's users table), and the processes of no user are listed apart — CPU % of the whole machine (100 % = every core busy) and
   resident memory over a 1 s `/proc` sample, a process count, the busiest
   processes by their kernel name (never argv or env), and their home's size
   as a share of the homes disk. A process counts for a user when it runs as
   their Linux account (S3) or, for the host account, when it inherited a
   workspace's `AGENTHUB_LANE_ID` — so without S3 a user's share is only what
   runs in their workspaces' panes; the rest is **Host account** or
-  **System**. Read-only (`server/lib/host-stats.js`); Linux only for the
-  per-user part. The page re-reads every 5 s while shown.
+  **System**. The same 1 s sample also gives every CPU thread's busy %,
+  network download/upload per interface (no `lo`), read/write per whole
+  disk, and the ten busiest processes machine-wide with who they count for;
+  plus `/sys/class/hwmon` temperatures, physical cores, boot time and the
+  AgentHub process's own pid, Node version, uptime and RSS — laid out in the
+  tab's own card style (`client/src/components/health/HostHealth.tsx`): the
+  machine card (with the V8 heap from `/api/settings/info`), then CPU threads,
+  temperatures / GPUs, disks with their I/O and network, then history and the
+  busiest processes. Read-only (`server/lib/host-stats.js`); Linux only for
+  the per-user part. The tab re-reads it every 5 s while shown.
+- `GET /api/host/history?range=1h|6h|24h` (every signed-in user) is the tab's
+  **History** charts: machine CPU % and memory % (plus swap, load and each
+  user's CPU/memory in the payload), one point every 30 s for 24 h. Every
+  server on Linux (local or server mode) keeps it IN MEMORY from the moment it starts
+  (`startHistory`, server/index.js) — a restart starts it empty, and nothing
+  is written to disk.
 - `GET /api/users/storage` (admin/analyst) is the page's **Storage** column:
   how much disk each user's home takes (their Linux home under S3, their
   folder under `AGENTHUB_USERS_ROOT` otherwise; an admin with none reads
@@ -916,12 +944,12 @@ a pane really is. Each case below is refused or bounded, with a test in
   an older server. Until `agenthub server install-system` has installed
   it, calls use `TMUX_BIN` directly, as before. A member's server keeps its
   version until their last pane closes; the next start runs `TMUX_BIN`.
-- **Administration → Server → Members' tmux** (admin/analyst read, admin acts): each member's
+- **Administration → users table, tmux column** (admin/analyst read, admin acts): each member's
   running server with its version beside the installed one (an *older than
   installed* badge), its sessions and whether an agent of theirs is working.
-  **Restart** ends one member's server — every pane of theirs closes, their
+  The row's **Restart** button ends one member's server — every pane of theirs closes, their
   Claude sessions resume from the workspace — and **Restart N idle members**
-  does it for every outdated member with no agent working; both take the
+  (above the table) does it for every outdated member with no agent working; both take the
   admin's password again and are audited per member. A server that will not
   stop answers `409 ETMUXSTUCK` naming its pid: root has to `kill -9` it.
 - **A member's huge or sparse files.** Transcripts and journals are read line
@@ -999,7 +1027,7 @@ agenthub server install-system
 
 A member server that a newer tmux already reached before `tmux-member` was
 installed can stay wedged: every call on that socket — `tmux-member`'s too —
-prints `server exited unexpectedly`. If Administration → Server → Members' tmux shows it with
+prints `server exited unexpectedly`. If the users table's tmux column on Administration shows it with
 no panes, stop that one server (its pid is the `ah-<name>` process running
 `tmux -S <USERS_ROOT>/<name>/.agenthub/tmux/sock`) with `sudo kill -9 <pid>`;
 the next start runs a fresh server on `TMUX_BIN`. Never `tmux kill-server`.