feat(server): S3 box is for coding — OOM kills the command not the session, no tests from Claude, 4 AI sessions per member
- tool-run (CLAUDE_CODE_SHELL_PREFIX in member panes) runs each command Claude runs at oom_score_adj 1000 (session 300, Terminal tab 800, dashboard/tmux -100), so an OOM inside the capped unit kills the test run and tells Claude why, instead of killing the conversation. - member-guard.js, a synchronous managed hook for ah-* accounts: denies test runners and container commands from Claude's Bash, and refuses the prompts of a session started by hand past MAX_AI_SESSIONS. - ai-procs.js counts a member's top-level AI processes from /proc; spawnRun refuses a new pane past the limit (429 ESESSIONLIMIT). Needs `agenthub server install-system` once, then a restart.
This commit is contained in:
+1
-1
@@ -89,7 +89,7 @@ A dashboard shared by several people runs in **server mode** ([SERVER-MODE.md](S
|
||||
| `POST /api/auth/login` `{username, password}` | `{user}` + the cookie; `401 EBADLOGIN`, `429 ETOOMANY`, `404 ENOTSERVER` in local mode |
|
||||
| `POST /api/auth/logout` | `{ok: true}`, cookie cleared |
|
||||
|
||||
In server mode a **developer** sees only rows of the workspaces they own (`lanes.owner_id`): every list, count, chart and cost answer is limited to them, and a request naming another user's session, agent, workspace or run (in the path, query or body) answers `404 ENOTFOUND`. Admin and analyst see everything; `GET /api/users` lists every account (`displayName`, `email`, `createdAt`, `logins`, `lastLoginAt` — the last login ever, kept after it ends —, `mustChangePassword` included) with its workspaces, sessions and `today` (since the viewer's local midnight — `?tz_offset=` is the browser's `getTimezoneOffset()`, UTC without it) / 7 / 30-day `{cost, tokens}`, `GET /api/users/storage[?refresh=1]` each user's home size (`{storage: {<id>: {bytes, files, partial, measuredAt} | null}, measuring}` — measured in the background, see [SERVER-MODE.md](SERVER-MODE.md)), `GET /api/users/tmux` each member's own tmux server under one Linux user per account (`{available, installed, matched, users: [{id, username, running, stuck?, version, pid, startedAt, sessions, busy, outdated}]}`; an admin's `POST /api/users/:id/tmux-restart` and `POST /api/users/tmux-restart-idle` restart them with `currentPassword`, audited, `409 ETMUXSTUCK` for a server that will not stop), `GET /api/host` the machine and each user's live share of it (`{at, hostname, os, uptime, cpu: {model, cores, load}, memory: {total, available, used, swapTotal, swapUsed}, disks: [{device, mount, type, kind: "HDD"|"SSD"|null, total, used, free}], homesMount, gpus: [{name, utilization, memoryUsed, memoryTotal}], usage: {cpu, sampleMs, groups: [{key, kind: "user"|"host"|"system", userId?, username?, displayName?, avatarAt?, cpu, memory, memoryPercent, processes, top: [{pid, name, cpu, memory}]}]} | null, bootTime, network: [{name, rx, tx}], diskIo: [{name, read, write}], temperatures: [{chip, label, celsius}], pressure: {cpu, memory, io: {some10, some60, full10} | null} | null (Linux PSI, `/proc/pressure/*`: % of the last 10 / 60 s tasks stalled on it), topProcesses: [{pid, name, cpu, memory, group}], process: {pid, node, uptime, rss, heapUsed}}`, `cpu` also carrying `physicalCores` and `perCore` (busy % per thread), rates in bytes/s over the sample — read-only, cached 3 s, `usage` null and the lists empty off Linux), `GET /api/host/history?range=1h|6h|24h` the CPU/memory history every server on Linux keeps in memory since it started (both read by the Dashboard's System Health tab and open to every signed-in user — `/host` is not staff-only) (`{range, intervalMs, running, startedAt, points: [{at, cpu, memory, memoryUsed, swapUsed, load1, users: {<id>: {cpu, memory}}}]}`, one point per 30 s, averaged to ≤ 360), `GET /api/users/:id/details[?tz_offset=]` one account in full for the Administration page's user detail (`{user (with logins, lastLoginAt), sessions: {total, active, firstAt, lastAt}, usage: {today, last7d, last30d, daily: [{date, cost, tokens}] (30 local days, oldest first), byTool, byModel: [{key, cost, tokens, sessions}]}, workspaces: [{id, title, cwd, tool, kind, branch, status, createdAt, sessions, lastActiveAt, livePanes}], panes: [{id, laneId, laneTitle, laneCwd, tool, model, account, proxy, sessionId, startedAt, status}], recentSessions (15, with cost/tokens), pool: {claude|codex|cursor: {pin, group, accounts: [{name, kind, email, expired, allowed, pinned, quota, quotaPercent, full, livePanes, spend5h, spend7d, userSpend5h, userSpend7d, userTokens5h, userTokens7d, userRuns7d, userLivePanes, userLastAt}]}}, audit}` — `user*` fields are only the runs on that user's workspaces; never a token; `404 ENOUSER`), and `GET /api/users/audit[?limit=]` the staff reads of developers' sessions and admins' account changes (kept `AGENTHUB_AUDIT_DAYS` days, default 180, `0` = forever) (`reader`/`owner` are usernames, `reader_display`/`owner_display` their display names or null) (all `403 EADMIN` for a developer). `GET /api/users` also carries each account's [spend quota](SERVER-MODE.md#spend-quotas) — `quotaSetting` (`{mode: "default"|"unlimited"|"custom", limit_usd?, period?, tz_offset?}`) and `quota` (the one binding them now with `spend`, `pct`, `exceeded`, `resetsAt`, or null) — and the top-level `quotaDefault`; an admin sets the default with `PUT /api/users/quota-default` `{quota: {limit_usd, period, tz_offset} | null, currentPassword}` and a user's own with `PATCH /api/users/:id` `{quota}` (`400 EBADQUOTA`). A workspace owner over their quota gets `429 EQUOTA` from `POST /api/run` and `POST /api/lanes/:id/start`. See [SERVER-MODE.md](SERVER-MODE.md#what-each-user-sees).
|
||||
In server mode a **developer** sees only rows of the workspaces they own (`lanes.owner_id`): every list, count, chart and cost answer is limited to them, and a request naming another user's session, agent, workspace or run (in the path, query or body) answers `404 ENOTFOUND`. Admin and analyst see everything; `GET /api/users` lists every account (`displayName`, `email`, `createdAt`, `logins`, `lastLoginAt` — the last login ever, kept after it ends —, `mustChangePassword` included) with its workspaces, sessions and `today` (since the viewer's local midnight — `?tz_offset=` is the browser's `getTimezoneOffset()`, UTC without it) / 7 / 30-day `{cost, tokens}`, `GET /api/users/storage[?refresh=1]` each user's home size (`{storage: {<id>: {bytes, files, partial, measuredAt} | null}, measuring}` — measured in the background, see [SERVER-MODE.md](SERVER-MODE.md)), `GET /api/users/tmux` each member's own tmux server under one Linux user per account (`{available, installed, matched, users: [{id, username, running, stuck?, version, pid, startedAt, sessions, busy, outdated}]}`; an admin's `POST /api/users/:id/tmux-restart` and `POST /api/users/tmux-restart-idle` restart them with `currentPassword`, audited, `409 ETMUXSTUCK` for a server that will not stop), `GET /api/host` the machine and each user's live share of it (`{at, hostname, os, uptime, cpu: {model, cores, load}, memory: {total, available, used, swapTotal, swapUsed}, disks: [{device, mount, type, kind: "HDD"|"SSD"|null, total, used, free}], homesMount, gpus: [{name, utilization, memoryUsed, memoryTotal}], usage: {cpu, sampleMs, groups: [{key, kind: "user"|"host"|"system", userId?, username?, displayName?, avatarAt?, cpu, memory, memoryPercent, processes, top: [{pid, name, cpu, memory}]}]} | null, bootTime, network: [{name, rx, tx}], diskIo: [{name, read, write}], temperatures: [{chip, label, celsius}], pressure: {cpu, memory, io: {some10, some60, full10} | null} | null (Linux PSI, `/proc/pressure/*`: % of the last 10 / 60 s tasks stalled on it), topProcesses: [{pid, name, cpu, memory, group}], process: {pid, node, uptime, rss, heapUsed}}`, `cpu` also carrying `physicalCores` and `perCore` (busy % per thread), rates in bytes/s over the sample — read-only, cached 3 s, `usage` null and the lists empty off Linux), `GET /api/host/history?range=1h|6h|24h` the CPU/memory history every server on Linux keeps in memory since it started (both read by the Dashboard's System Health tab and open to every signed-in user — `/host` is not staff-only) (`{range, intervalMs, running, startedAt, points: [{at, cpu, memory, memoryUsed, swapUsed, load1, users: {<id>: {cpu, memory}}}]}`, one point per 30 s, averaged to ≤ 360), `GET /api/users/:id/details[?tz_offset=]` one account in full for the Administration page's user detail (`{user (with logins, lastLoginAt), sessions: {total, active, firstAt, lastAt}, usage: {today, last7d, last30d, daily: [{date, cost, tokens}] (30 local days, oldest first), byTool, byModel: [{key, cost, tokens, sessions}]}, workspaces: [{id, title, cwd, tool, kind, branch, status, createdAt, sessions, lastActiveAt, livePanes}], panes: [{id, laneId, laneTitle, laneCwd, tool, model, account, proxy, sessionId, startedAt, status}], recentSessions (15, with cost/tokens), pool: {claude|codex|cursor: {pin, group, accounts: [{name, kind, email, expired, allowed, pinned, quota, quotaPercent, full, livePanes, spend5h, spend7d, userSpend5h, userSpend7d, userTokens5h, userTokens7d, userRuns7d, userLivePanes, userLastAt}]}}, audit}` — `user*` fields are only the runs on that user's workspaces; never a token; `404 ENOUSER`), and `GET /api/users/audit[?limit=]` the staff reads of developers' sessions and admins' account changes (kept `AGENTHUB_AUDIT_DAYS` days, default 180, `0` = forever) (`reader`/`owner` are usernames, `reader_display`/`owner_display` their display names or null) (all `403 EADMIN` for a developer). `GET /api/users` also carries each account's [spend quota](SERVER-MODE.md#spend-quotas) — `quotaSetting` (`{mode: "default"|"unlimited"|"custom", limit_usd?, period?, tz_offset?}`) and `quota` (the one binding them now with `spend`, `pct`, `exceeded`, `resetsAt`, or null) — and the top-level `quotaDefault`; an admin sets the default with `PUT /api/users/quota-default` `{quota: {limit_usd, period, tz_offset} | null, currentPassword}` and a user's own with `PATCH /api/users/:id` `{quota}` (`400 EBADQUOTA`). A workspace owner over their quota gets `429 EQUOTA` from `POST /api/run` and `POST /api/lanes/:id/start`. Under one Linux user per account, a member already running `MAX_AI_SESSIONS` AI sessions (default 4) gets `429 ESESSIONLIMIT` from the same two routes. See [SERVER-MODE.md](SERVER-MODE.md#what-each-user-sees).
|
||||
|
||||
An admin changes accounts with `POST /api/users` (`{username, password, role}` → `201 {user}`; the password may be any non-empty starting one and the account gets `mustChangePassword: true`), `GET /api/users/:id` (`{user, workspaces: [{id, title, cwd}]}`, the account with `logins` and `lastLoginAt`, for the Edit form; admin or analyst), `GET /api/users/:id/avatar` (that account's picture for the sidebar's owner groups and System Health — the one `/api/users` read EVERY signed-in account may make, a developer included, `nosniff` + `sandbox` CSP, `404 ENOAVATAR`; a workspace in `GET /api/lanes` carries `owner_avatar_at` to version its URL), `PATCH /api/users/:id` (any of `{displayName?, email?, role?, disabled?, password?, addWorkspaces?: [laneId], removeWorkspaces?: [laneId], signOut?: true}` — every field validated before any is written; `removeWorkspaces` leaves the workspace with no owner and answers `409 ENOTOWNED` for one the user does not own; an admin's own `password`/`signOut` keeps the login it came from; a password for SOMEONE ELSE may be anything non-empty and sets their `mustChangePassword`, the admin's OWN must pass every rule below, `400 EWEAKPASSWORD` with `rules`), `POST /api/users/:id/password` (`{password}`; that user's logins end; the same two bars) `POST /api/users/:id/workspaces` (`{laneId}`; the workspace's owner becomes that user) and `DELETE /api/users/:id` (delete for good: the user's logins, panes, workspaces' records, prefs and — under S3 — their Linux account and whole home, removed FIRST so a refusal (`502 EPROVISION`) leaves every row; `409 ESELF` for yourself, `409 ELASTADMIN` for the last admin; session history and the audit log stay). No path — this API or the `agenthub users` CLI — may demote, disable or delete the last enabled admin (`409 ELASTADMIN`). Every one carries `currentPassword`, the admin's own password: `403 EREAUTH` when it is wrong or missing, `429 ETOOMANY` after 10 wrong tries in 15 minutes. `409 ESELF` for an admin's own role or disable, `409 EEXISTS` for a taken name, `409 ENAMETAKEN` for a `displayName` that reads as another account's (the same rule as `PATCH /api/me/account`), `400 EBADUSER`/`EBADPASSWORD`/`EBADROLE`/`EBADEMAIL`/`EBADFIELD`, `404 ENOUSER`/`ENOLANE`; a developer or analyst gets `403 EADMIN`, and local mode `403 ELOCALMODE`. No answer carries a password hash; each change is written to the audit log.
|
||||
|
||||
|
||||
+38
-17
@@ -1156,26 +1156,47 @@ The same re-run installs `agenthub-provision delete`, behind the Users page's
|
||||
**Delete user**: until then a delete answers `502 EPROVISION` ("unknown
|
||||
action: delete") and changes nothing.
|
||||
|
||||
The same re-run (v2.8.68) keeps a member's test run from freezing the box —
|
||||
on 11-02 some 36 open sessions plus test runs filled RAM and the 4 GB swap on
|
||||
a slow disk, and only a hard reset brought it back:
|
||||
The same re-run (v2.8.68) keeps members from freezing the box — on 11-02
|
||||
some 36 open sessions plus test runs filled RAM and the 4 GB swap on a slow
|
||||
disk, and only a hard reset brought it back. The box is for writing code;
|
||||
tests run on each member's own machine:
|
||||
|
||||
- `pane-launch` starts every member pane at `nice 10`, best-effort IO
|
||||
priority 7 and `oom_score_adj 500`, and defaults `VITEST_MAX_THREADS` /
|
||||
`VITEST_MAX_FORKS` to 2 (Vitest ≤ 3; a member may set their own). Only
|
||||
panes started after the re-run get these; the nice cannot be taken back
|
||||
without root.
|
||||
- `agenthub.service` gets `MemoryMax=85%`, `MemorySwapMax=1G`,
|
||||
- **Memory.** `agenthub.service` gets `MemoryMax=85%`, `MemorySwapMax=1G`,
|
||||
`OOMPolicy=continue` and `OOMScoreAdjust=-100`. The cgroup holds the server
|
||||
AND every member pane, so memory runs out inside the unit first, and the
|
||||
kernel kills the biggest pane process there — not the dashboard, tmux, or
|
||||
the rest of the machine. `daemon-reload` (which the re-run does) applies
|
||||
the memory caps to the running unit; `OOMScoreAdjust` takes effect at the
|
||||
next restart (`sudo -n /usr/bin/systemctl restart --no-block
|
||||
agenthub.service` — panes survive, `KillMode=process`).
|
||||
AND every member pane, so memory runs out inside the unit first and the
|
||||
kernel kills one process there by `oom_score_adj`: every command a member's
|
||||
Claude runs is 1000 (`CLAUDE_CODE_SHELL_PREFIX=/opt/agenthub/libexec/tool-run`,
|
||||
which also tells Claude why its command died), a Terminal tab 800, an AI
|
||||
session 300, the dashboard and tmux -100. A test run dies; the session
|
||||
holding the conversation does not. Codex runs its commands without that
|
||||
wrapper, so its commands sit at 300 beside it.
|
||||
- **CPU and disk.** `pane-launch` starts every member pane at `nice 10` and
|
||||
best-effort IO priority 7 (the nice cannot be taken back without root), and
|
||||
defaults `VITEST_MAX_THREADS` / `VITEST_MAX_FORKS` to 2.
|
||||
- **No tests or containers from Claude.** A synchronous `PreToolUse` hook
|
||||
(`/opt/agenthub/app/scripts/member-guard.js`, Bash only, `ah-*` accounts
|
||||
only) denies `vitest`, `jest`, `phpunit`, `pest`, `pytest`, `npm test`,
|
||||
`npm run test:*`, `php artisan test`, `go test`, `cargo test`, `docker`,
|
||||
`podman`, … and tells Claude to leave testing to the member's machine.
|
||||
A person typing a command in a Terminal tab, and Codex, are not covered.
|
||||
- **At most 4 AI sessions per member at once** (`MAX_AI_SESSIONS=<n>` in
|
||||
`/etc/agenthub/server.conf`, `0` = no limit). Counted from the account's
|
||||
running processes (`server/lib/ai-procs.js`: `claude`, `codex`, `agy`,
|
||||
`cursor-agent`, `omp`, a nested one counted once), so a `claude` typed into
|
||||
a Terminal tab counts and the tab itself does not. A new pane past the
|
||||
limit answers `429 ESESSIONLIMIT`; a session started by hand past it has
|
||||
every prompt refused by the guard's `UserPromptSubmit` until another one
|
||||
ends. The host account (admins) is not limited.
|
||||
|
||||
Check with `systemctl show agenthub.service -p MemoryMax,MemorySwapMax,OOMPolicy`
|
||||
and, in a new member pane, `ps -o ni= -p $$; cat /proc/self/oom_score_adj`.
|
||||
The memory caps apply to the running unit at once (the re-run does
|
||||
`daemon-reload`); `OOMScoreAdjust`, the session limit on new panes and the
|
||||
guard script in `/opt/agenthub/app` arrive with the next restart
|
||||
(`sudo -n /usr/bin/systemctl restart --no-block agenthub.service` — panes
|
||||
survive, `KillMode=process`). Panes started before the re-run keep their old
|
||||
priority. Check with
|
||||
`systemctl show agenthub.service -p MemoryMax,MemorySwapMax,OOMPolicy`
|
||||
and, in a new member pane, `ps -o ni= -p $$; cat /proc/self/oom_score_adj;
|
||||
echo $CLAUDE_CODE_SHELL_PREFIX`.
|
||||
|
||||
### Deleting a user
|
||||
|
||||
|
||||
Reference in New Issue
Block a user