AgentHub/docs/SERVER-MODE.md

# Server mode

AgentHub has one codebase and two ways to run it:

| | Local mode (default) | Server mode |
| --- | --- | --- |
| Who uses it | You, on your own machine | Several people over the web |
| Login | None (the loopback bind is the trust boundary) | Every page, API call and WebSocket needs one |
| Turn on | Nothing to do: every install starts here | `agenthub server setup` |
| Extra code loaded | None: `/api/auth/me` answers `{mode: "local"}` and nothing else changes | `server/lib/auth.js`, the login page |

Local mode does not change at all. The login code stays dormant until
`AGENTHUB_MODE=server` is set.

> **What server mode separates, and what it does not.** In the dashboard and
> its API, each developer sees only their own workspaces and the sessions,
> prompts, events, tokens and cost under them (see [What each user sees](#what-each-user-sees)).
> Until [one Linux user per account](#one-linux-user-per-account) is installed
> (`agenthub server install-system`), every terminal pane runs as the one Linux
> account that runs the dashboard. An agent in a pane can then read every
> user's folders, transcripts and the database straight from disk. Without it,
> use server mode only with people you trust.

## Set it up

On the machine that will host the dashboard, logged in as the account that
runs it:

```bash
# 1. Turn server mode on, allow the public name, and create the first admin
agenthub server setup --hostname agenthub.example.com --admin boss
#    (asks for the admin's password twice; from a script: pipe it on stdin)

# 2. Restart the dashboard so it picks the mode up
systemctl --user restart agenthub     # or: agenthub start

# 3. Add the other people
agenthub users add alice                      # developer
agenthub users add carol --role analyst       # reads everyone, works in their own
```

`setup` writes `AGENTHUB_MODE=server` and `DASHBOARD_ALLOWED_HOSTS` into
AgentHub's settings file (`~/.config/agenthub/settings.env`). The real
environment and a checkout's `.env` still take priority over that file. Run
setup again whenever you need to: it creates an admin only when none exists.

## Put HTTPS in front of it

Keep the dashboard on its default loopback bind (do **not** set
`DASHBOARD_HOST`) and serve it through a reverse proxy on the same machine.
The proxy has to:

- pass the original `Host` (the dashboard checks it against
  `DASHBOARD_ALLOWED_HOSTS` and compares it with the browser's `Origin`);
- forward WebSocket upgrades (`/ws` for live updates, `/ws-pty/*` for terminals);
- set `X-Forwarded-For` and `X-Forwarded-Proto`.

nginx:

```nginx
server {
  listen 443 ssl;
  server_name agenthub.example.com;
  ssl_certificate     /etc/ssl/agenthub.crt;
  ssl_certificate_key /etc/ssl/agenthub.key;

  location / {
    proxy_pass http://127.0.0.1:4820;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 1d;
  }
}
```

The login cookie gets `Secure` when the request came over HTTPS
(`X-Forwarded-Proto: https`).

## Roles

| Role | Can |
| --- | --- |
| `admin` | Everything, including the host's own configuration and every user's data |
| `analyst` | Reads everything (usage, prompts, every user's sessions), and works in their OWN workspaces like a developer. Never changes another user's rows or the host's configuration |
| `developer` | Their own workspaces: start sessions, terminals, git, files. Sees nothing of other users, and cannot open or change the host's configuration |

`developer` was called `member` before; a database with members is migrated on
start (`server/db.js`), and `member` is no longer accepted.

"The host's configuration" means a write (anything but GET/HEAD) under
`/api/settings`, `/updates`, `/cc-config`, `/codex-config`,
`/git-credentials`, `/ai-tools`, `/import`, `/pricing`, `/budgets`,
`/webhooks`, `/remote-sources`, `/reports`, `/pets`, `/users` and
`/alerts/rules` (`ADMIN_WRITE_PREFIXES` in `server/lib/auth.js`), plus a write
to a shared AI Proxy (see [AI Proxy profiles](#ai-proxy-profiles)). A
developer or analyst who tries one gets `403 EADMIN`, shown in the UI as "Only
an admin can do this". An analyst's other writes are scoped like a
developer's: one naming another user's row gets `404 ENOTFOUND` (`memberId` in
`server/lib/scope.js` limits every non-GET request of an analyst to their own
rows), and under S3 an analyst's workspace runs as their own Linux account
exactly like a developer's.

The UI hides what a role cannot use (`usePermissions` in
`client/src/components/AuthGate.tsx`): for anyone but an admin, Settings shows
no Data, Pricing or per-tool pages, no scheduled reports or git credentials,
no start-at-login/icon/Quit/Uninstall on System, no alert rules or channels,
no install/update/enable on AI tools, no pet upload or delete, and no update
check; the AI proxy page shows them only their personal profiles' Edit. An
analyst's Workspace page and sidebar list only
their own workspaces (`useWorkLanes`); other users' work is read on Sessions,
Analytics and Users, where Resume, Fork, Snapshot and "Open in workspace" are
hidden for a session that is not theirs. The sidebar's workspace list shows an
admin or analyst EVERY workspace, grouped under its owner (each lane carries
`owner_name`, `owner_display_name` and `owner_avatar_at`; the group shows the owner's picture (`GET /api/users/:id/avatar`) and the display name when
the owner set one, else the username; yours first, no owner last, each group folds): for an analyst
another user's workspace there is read only — no menu, rename or new session,
no live tabs, and a session opens its analysis page (`/sessions/<id>`). An analyst's alert feed
(`GET /api/alerts`) carries `can_ack` on each alert — true only for their own
sessions' — and the feed offers Acknowledge only there; Ack all acknowledges
only theirs.

## What each user sees

Every workspace has an owner: the user who created it (`lanes.owner_id`).
Workspaces made before server mode, or by the host account, have none.

- A session belongs to the workspace that owns it: `sessions.lane_id`, or the
  workspace whose pane holds it. Its agents, events, tokens, cost, alerts,
  snapshots and workflow runs belong to that session.
- A **developer** sees only rows of their own workspaces, everywhere: lists,
  counts, charts, cost, the Sessions/Events/Agents tables, exports, live
  WebSocket updates and terminals. A request naming someone else's row (in the
  path, the query or the body) gets `404 ENOTFOUND`, the same answer as a row
  that does not exist (`scopeGuard`, `server/lib/scope.js`). Sessions with no
  workspace, such as ones started in a terminal outside AgentHub, are not shown
  to developers.
- An **admin** or **analyst** sees everything. Each time one of them opens a
  session a developer owns (its page, transcript, events or cost), the read is
  written to the audit log: `agenthub users audit`, or `GET /api/users/audit`.
- `GET /api/users` (admin/analyst) lists every account with its workspaces,
  sessions, and the tokens and $ it used in the last 7 and 30 days — the
  Users page in the sidebar shows it, with the audit log under it.
- `GET /api/users/storage` (admin/analyst) is the page's **Storage** column:
  how much disk each user's home takes (their Linux home under S3, their
  folder under `AGENTHUB_USERS_ROOT` otherwise; an admin with none reads
  `null`). Homes are walked in a worker thread, one at a time, `du -x`-like
  (allocated blocks, `lstat` only, no symlink followed, a hard link once),
  and cached for 15 minutes; the answer comes back at once with
  `{storage: {<userId>: {bytes, files, partial, measuredAt} | null}, measuring}`
  and the page asks again while `measuring`. `?refresh=1` (the page's Refresh)
  re-measures. A walk over `AGENTHUB_STORAGE_TIMEOUT_MS` (default 120 s) or
  one meeting an unreadable folder is a lower bound, `partial: true` (shown
  as `≥`).

A developer also cannot open the host's own configuration, or any view across all
users: Claude Code / Codex config (except the command and skill lists the
prompt box uses), the AI tools' config files (`/api/ai-tools/<id>/config…`),
git credentials, remote sources, webhooks, reports, metrics, imports, data
export, alert rules and `/api/users`. Those return `403 EADMIN`, and Settings
hides them for developers. The command and skill lists show a developer only their
own workspaces' project items: `?cwd=` outside one of them answers
`403 EFOLDER`, and with no `cwd` they list no project items at all.

### AI Proxy profiles

Each user may keep **personal** AI Proxy profiles beside the **shared** ones:

- A developer's or analyst's new profile is always personal. An admin's is shared unless
  they choose **Only me (personal)** in the form (`scope: "personal"`).
- A personal profile is named `u<user id>_<slug>`. Only its owner lists,
  edits, deletes or tests it, and only the owner's workspaces can start a
  session on it. Anyone else, an admin included, gets `404`, the same answer
  as a profile that does not exist. A resume of a session that ran on someone
  else's personal profile goes back to native.
- Only an admin writes a shared profile. Everyone can choose one.
- **No pane is given a proxy's key, and no route returns one.**
  - A session's CLI talks to AgentHub's own loopback gateway,
    `http://127.0.0.1:<port>/api/proxy-gw/<profile>/v1/…`. Its key there is
    its pane's token (`ANTHROPIC_AUTH_TOKEN` for Claude, `AGENTHUB_PROXY_KEY`
    for Codex).
  - AgentHub swaps in the real key and streams the request to the profile's
    endpoint.
  - The gateway answers only on this machine, only while the pane lives, and
    only for the profile the pane was started on.
  - An agent can still spend through the profile while its pane runs, but it
    cannot carry the key away.
  - Claude account-pool tokens are not covered: they still reach the pane (see
    [Claude account pool](#claude-account-pool)).

### Keep developers' folders apart

```bash
# in ~/.config/agenthub/settings.env
AGENTHUB_USERS_ROOT=/srv/agenthub/users
```

With this set, a developer or analyst can create a workspace only inside
`/srv/agenthub/users/<username>`; any other folder gets `403 EFOLDER`. The
same root limits a developer's folder browser (`GET /api/lanes/browse`, which
opens on their own folder) and `GET /api/lanes/branches`, both judged by real
path, so a link cannot point them elsewhere. Under S3 the home itself is no
workspace (its dot folders hold the member's logins); use a folder in it. A folder
that already belongs to another user's workspace gets `409 EOWNED`. Admins are
not limited. Without [one Linux user per account](#one-linux-user-per-account)
you create each user's folder yourself, and this keeps workspaces apart, not
agents. `install-system` sets this value to `/srv/agenthub/users`, and adding a
user makes the folder (it is the Linux account's home). With S3 installed, a
settings file that lacks the variable falls back to `USERS_ROOT` in
`/etc/agenthub/server.conf`, so a developer is never shown the host's home.

## What the gate does

- `/api/health` and `/api/auth/*` answer without a login. Everything else
  under `/api` answers `401 EAUTH` until someone logs in, and the UI shows the
  login page. Paths are checked the way Express routes them: decoded, with
  repeated slashes collapsed, and case-insensitive.
- The login page follows the OS's light/dark. After a login the dashboard
  wears that account's own appearance (Settings → Appearance, saved in
  `/api/me/prefs`); an account that never chose one follows the OS.
- Hooks (`/api/hooks/*`) are accepted only from this machine: a loopback socket
  with no forwarding header. A request through the proxy gets
  `403 ELOCALONLY`. `POST /api/hooks/ingest-batch` keeps its own bearer token.
- Every write (any method but GET/HEAD) that carries an `Origin` or `Referer`
  must name the dashboard's own allowed host, or loopback. Anything else gets
  `403 EBADORIGIN`. This is on top of the `SameSite=Strict` cookie, which a
  sibling subdomain would still satisfy.
- WebSockets need a login, and an `Origin` that is the request's own allowed
  Host. A terminal (`/ws-pty/*`) is refused to anyone but an admin and the
  owner of the pane's workspace (an analyst included). Open sockets are re-checked: `/ws` every 30 s,
  terminals every 10 s (`AGENTHUB_PTY_RECHECK_MS`). A logout, a disable, a
  password change, or losing the pane's workspace closes them.
- A login lasts 30 days in the httpOnly, `SameSite=Strict` cookie
  `agenthub_sid`. Only the token's SHA-256 is stored. Logging out, a password
  change and disabling the user each end it.
- Passwords are stored as scrypt hashes; any non-empty password is accepted (no minimum length).
  Every login attempt counts before its password is checked, so attempts sent
  in parallel count too. After 10 failed logins from one address, or 30 for one
  username, in 15 minutes, further logins answer `429 ETOOMANY` until the
  window passes. The counters are kept in memory, so a restart resets them.
- **Inside a pane**, the `agenthub` CLI, the MCP server and the statusline act
  as the owner of the pane's workspace. Each pane the dashboard starts gets its
  own `AGENTHUB_PANE_TOKEN` through tmux `-e`, and the database stores only its
  hash (`dashboard_runs.pane_token_hash`). The token is accepted only from this
  machine and only while that pane is alive, as header `x-agenthub-pane-token`
  (with one Linux user per account on: only from the Linux account that pane
  runs as, read from the kernel like a hook's sender).
  A workspace with no owner has no such user. Give old workspaces an owner with
  `agenthub users assign <workspace id> <user>`. With one Linux user per
  account on, a workspace inside a member's home can be owned by that member
  only, and one outside the users root (its panes run as the host account) by
  an admin only: creating, assigning or starting one against that answers
  `409 EOWNERFOLDER` — by the user's role at that moment, and demoting or
  disabling a user ends their panes that run as the host. A pane's folder must
  be its workspace's account's too (`409 ECWDACCOUNT`).
- Named locks (`agenthub lock …`) are per user: a developer's lock names live in
  their own namespace, so one user's agents never see or take another's.
- Browser push subscriptions belong to the user who made them. A
  notification goes only to that user's own browsers, never to the host's
  desktop.
- A developer or analyst cannot create a session by hand (`POST /api/sessions` → `403`), and
  cannot change a workspace's pane or session columns: `PATCH` accepts only
  `title`, `pipeline`, `notes`, `links` and `branch` (`403 EFIELD`).
  Pet uploads are admin-only.
- The native file picker stays local-only (`EREMOTE`) and every page falls back
  to its folder browser.

## Accounts

An admin manages accounts on the **Users** page (sidebar; admins and analysts
see it, an analyst only reads): **Add user**, and one **Edit** form per account
that changes, in a single save, its display name and email, role,
active/disabled, password (typed or generated), the workspaces it owns (give
one, or take one back — it is then left with no owner) and "sign out
everywhere". Every field is checked before any is written, so a bad email or
a workspace the user does not own changes nothing. Every change asks for the admin's OWN
password again — the server refuses without it (`EREAUTH`, and `ETOOMANY`
after 10 wrong tries in 15 minutes) — an admin can never demote or disable
themselves (`ESELF`), and each change is written to the audit log. In local
mode these routes refuse (`ELOCALMODE`): there is no login to check.

Wherever the dashboard names a person — the sidebar's sign-out tooltip, the
workspace list's owner groups, the Users page and its audit log, the Edit user
form — it shows their display name when one is set, else the username (the
Users page keeps `@username` beside a display name). Logging in, the CLI and
the Linux account always use the username.

Under S3 an admin's folder browser opens on their own home too; ticking
**Server folder** opens the host account's home (`/home/<host>`), where a
workspace runs as the host account that every admin shares. The tick is never
remembered, so the default is always the admin's own account.

Every signed-in user has **Settings → Account**: their username, role, member
since, workspace and session counts and active logins; their own picture
(PNG, JPEG, GIF or WebP up to 1 MB, kept in `<data dir>/avatars/`; shown with
their name and role at the foot of the sidebar, which opens this page); their
own display name and email; **Change password** (the current password is required, the login
it is changed from stays and every other one ends); and **Sign out other
devices**. These are taken from the login cookie only — an agent in a pane,
which acts as its workspace owner through its pane token, cannot change its
owner's password or logins.

The same from the host's shell:

```bash
agenthub users list
agenthub users add <name> [--role admin|analyst|developer]
agenthub users passwd <name>        # ends that user's logins
agenthub users role <name> <role>
agenthub users disable <name>       # ends that user's logins; users are never deleted
agenthub users enable <name>
agenthub users assign <workspace id> <name>   # give a pre-server-mode workspace an owner
agenthub users audit [--limit N]    # who read which developer's session
agenthub server status
```

## Claude account pool

An admin can sign every Claude account the company has into AgentHub. Each new
Claude session then runs on the account with the most room left, and nobody has
to log in or switch accounts by hand. The pool works in local mode too, and it
stays empty (changing nothing) until you add an account.

```bash
# Once per account: sign in as that account and make a one-year token
claude setup-token
# Add it to the pool (the token is typed hidden, or piped on stdin)
agenthub accounts add team-1
agenthub accounts list        # live panes, $ and tokens in the last 5 h / 7 d, quota, state
agenthub accounts remove team-1
```

In server mode an admin can do the same on the **Users** page, in the
**Account pool** card: **Add account** and the bin icon to remove one. A Claude
account comes from one of three sources:

- **A saved login**: a profile from Settings → Claude config → Switch account
  (log in with `claude auth login`, then save it). It is MOVED into the pool, so
  the profile leaves the switcher. The account this server's own Claude Code is
  logged into can be moved in too. It moves with the login the CLI holds right
  now, keeps working on the server, and the server's own `.credentials.json`
  becomes one more home the pool refreshes and writes into, keeping its other
  keys. Only a login kept in the macOS Keychain is refused (`EACTIVE`). Removing
  that account later leaves the server logged in.
- **A pasted `.credentials.json`**: from `~/.claude/.credentials.json`, or on a
  Mac `security find-generic-password -s "Claude Code-credentials" -w`. Stop
  using that login where you copied it from.
- **A `claude setup-token` token** (`sk-ant-…`), as with `agenthub accounts add`.

A Codex account is an OpenAI API key, or the whole `~/.codex/auth.json` after
`codex login`.

A **login** (the first two sources) works like a Codex ChatGPT login. Claude
Code reads it from no env var, so before a member's pane starts, AgentHub writes
it as that member's `~/.claude/.credentials.json`, as the member (0600). It is
never written into a host pane's home, never over a login the member made
themselves, and never a second account into a home where a pane of theirs runs
on one. Host panes therefore get only setup-token accounts — and the login the
host itself was moved in from (the profile marked active), as long as the
host's `~/.claude/.credentials.json` still holds that pair: the pane runs on it
as it is and its usage is booked to that account. The refresh token of
a login is single use, so the server is the one that refreshes it, before it
expires (`claude-pool.refreshLogins`, checked every 60 s; `DASHBOARD_POOL_REFRESH_MS=0` turns it off). It then writes the new pair into every
home it placed the login in, and Claude Code picks up tokens another process
refreshed. A pair a member's Claude refreshed first is copied back only after
Anthropic confirms it is the same account. Removing the account deletes it from
the homes that still hold AgentHub's copy. Each
change asks for the admin's own password again and is written to the audit log
by account name. The token only goes in; the page, like `agenthub accounts list`,
never shows it again. An analyst sees the card read-only; a developer does not
see it. Every user shares the pool: a user created later needs no setup of their own.

How a new session picks its account (`server/lib/claude-pool.js`):

1. An account whose last quota reading is at or above 90% in the 5-hour or
   weekly window goes last. It is still used if every account is that full.
2. Next, the account with the fewest live panes, so concurrent sessions spread out.
3. Then the account with the least $ spent in the last 5 hours, then the last
   7 days. AgentHub sums this from the usage of the sessions its runs held on
   each account (`dashboard_runs.account`), over the hours each run held them —
   a session resumed on another account counts there from then on — with
   sub-agent usage included and the same prices as the cost pages
   (`server/lib/pool-usage.js`). The table also shows the tokens behind it.
   Sessions started outside AgentHub on the same account are not counted; the
   quota column is Anthropic's own reading and does include them.

- The pane gets the token through tmux `-e CLAUDE_CODE_OAUTH_TOKEN=…`. That
  token outranks the machine's own `/login` for that process only.
- A session on an AI Proxy keeps the proxy's key and does not use the pool.
- With one Linux user per account (S3), a member's home starts empty, and a
  fresh Claude Code shows its "Select login method" screen even with the token
  set. So before a member's pane starts on a pool account or an AI Proxy,
  AgentHub adds `hasCompletedOnboarding: true` to that member's
  `~/.claude.json`, written as the member, with every other key kept
  (`run-as.markClaudeOnboarded`). With an empty pool and no proxy, a member's
  pane still needs that member's own `/login`.
- A token more than a year old is skipped.
- A resumed session picks an account again, like a new one.
- Quota is read with each account's own token, in the background, at most every
  5 minutes. If the usage endpoint refuses a setup-token, the state column shows
  `no quota (…)` and ranking uses spend alone.
- `GET /api/accounts` returns the same table, without tokens.

> **Ceiling:** anyone with a terminal in AgentHub can read the token of the
> account their pane runs on (`echo $CLAUDE_CODE_OAUTH_TOKEN`). Without one
> Linux user per account they can also read every file of the pool. With it,
> they still see their own pane's token (a member's pane gets it through its
> launch file instead of tmux `-e`), but no other account's process or file.
> The AI Proxy gateway does not cover pool tokens.

## Codex account pool

The same for Codex (`server/lib/codex-pool.js`), with two kinds of account:

```bash
# An OpenAI API key (typed hidden, or piped on stdin)
agenthub accounts add team-key --tool codex --api-key
# A ChatGPT account: sign in in a scratch home, then hand that home over
CODEX_HOME=$(mktemp -d) codex login
agenthub accounts add team-chatgpt --tool codex --from /tmp/tmp.XXXX
agenthub accounts list --tool codex
agenthub accounts remove team-key --tool codex
```

- **API key**: any number of panes at once. The pane gets the key through
  tmux `-e AGENTHUB_POOL_KEY=…`, and Codex is pointed at it with
  `-c model_providers.agenthub_pool={…env_key="AGENTHUB_POOL_KEY"…}` — the
  argv names the variable, never the key.
- **ChatGPT**: Codex reads this login only from `$CODEX_HOME/auth.json`, so it
  is used only for a pane that runs as its developer (one Linux user per
  account): AgentHub writes the account's `auth.json` into that developer's
  `~/.codex` as them (0600) just before the pane starts.
  - A host pane never gets one: its `CODEX_HOME` holds the host's own login.
  - A developer who signed Codex in themselves keeps their own login.
  - While a pane of theirs runs on a pool account, their next pane uses the
    same account (one `auth.json` per home).
  - Codex rotates refresh tokens. Every 10 s AgentHub copies a developer's
    `auth.json` back into the pool when its `last_refresh` is newer, so a new
    pane starts from the newest login — only from a home AgentHub placed that
    account in, never with a `last_refresh` in the future, and only when its
    `id_token` names the same account and email. The `id_token` signature is
    not checked: a developer the account was placed with can still spoil it
    for others until a real refresh from another home wins it back. Several panes may share one account
    (checked on codex 0.161: an older refresh token still refreshes).
- Ranking is the Claude pool's: quota ≥ 90% last, then fewest live Codex
  panes, then $ in 5 h / 7 d. The quota is the newest `rate_limits` in the
  rollouts of the sessions that account's runs started (no network), each
  read only from the `CODEX_HOME` of the account its workspace runs as, and
  never one dated in the future.
- A session on an AI Proxy keeps the proxy and does not use the pool.

> **Ceiling:** a pane can read its own account's key (`$AGENTHUB_POOL_KEY`) or
> `auth.json`, like the Claude pool's token.

## One Linux user per account

Every pane of a user, an agent's shell included, runs as that user's own Linux
account (`ah-<username>`). An agent in Alice's terminal then cannot read Bob's
folders, transcripts or the dashboard's database: the OS stops it, not just the
UI. This is phase S3 of `docs/plans/server-mode/S3-PLAN.md`. It is ON when
server mode is on AND `/etc/agenthub/server.conf` exists
(`server/lib/linux-accounts.js`, `s3Enabled`); without that file everything
below is a no-op and server mode behaves as described above. To install it,
follow the [operator runbook](#operator-runbook).

> **Status (2026-10-09):** built and checked on a real machine: the host
> setup, the accounts, panes, git and file saves as the member, hooks checked
> by account, reads across homes, shared tools in `/opt`, and the fixes from
> the 2026-10-08 security review (P8 F1–F11). Built and tested with fake
> accounts only: per-member quota (P4), the opt-in plugin baseline (P6) and
> the Codex account pool (P7). The second review (2026-10-09, P2–P7) is fixed
> (see [What the second review closed](#what-the-second-review-closed)).
>
> Not done yet (`docs/plans/server-mode/S3-PLAN.md`):
> - the admin's plugin baseline through managed settings (P6 spike, not run):
>   the opt-in fallback ([Plugin baseline](#plugin-baseline-opt-in)) is
>   built; otherwise a member installs plugins in their own terminal
>   (`/plugin`);
> - Codex and Antigravity hooks for members: provisioning installs none, so a
>   member's Codex and Antigravity sessions are tracked from their journals
>   (the pollers read every member's home), not from hooks;
> - per-member quota, the plugin baseline and the Codex account pool (P4, P6,
>   P7) are not yet tried with real member accounts or a real Codex login.

### The model

- **Identity.** AgentHub user `alice` is Linux user `ah-alice`, home
  `/srv/agenthub/users/alice` (`0750`, owner `ah-alice`, group `agenthub`).
  The name must be lower case: a letter, then up to 28 letters, digits, `_` or
  `-` (`LINUX_NAME_RE`). While S3 is on, a user with any other name cannot be
  created (`400 EBADUSER`). Admins and analysts get a Linux account too. A
  workspace's folder decides who runs it (`server/lib/run-as.js`,
  `accountForPath`): a folder under `/srv/agenthub/users/<name>/` runs as
  `ah-<name>`; any other folder runs as the host account, as before (a
  workspace made before S3 outside the users root, for example). A folder under the users root whose
  user is unknown, disabled or not provisioned is an error (`409
  EMEMBERGONE`), never "run it as the host".
- **The dashboard.** A SYSTEM unit, `agenthub.service`, runs it as the host
  account with `CAP_DAC_READ_SEARCH` (it reads every member's tool files —
  Claude writes transcripts `0600` — but writes none of them). Every program
  it starts goes through `setpriv --inh-caps=-all --ambient-caps=-all` first
  (`server/lib/drop-caps.js`), so host panes, git hooks and npm do not inherit
  the capability.
- **Run-as through sudo.** Everything done AS a member is
  `sudo -n -u ah-<name> <command>`, and the sudoers file allows exactly
  `tmux`, `git`, `gh` (by the absolute paths in `server.conf`),
  `/opt/agenthub/libexec/write-as` and `/opt/agenthub/tools/bin/claude` (the
  plugin baseline) that way — never root. As root, the host may
  run only `/opt/agenthub/sbin/agenthub-provision` (create, disable or enable
  a member) and `systemctl start|stop|restart|enable|disable
  agenthub.service`. Members are never in sudoers.
- **sudo logs every command line**, so nothing secret or private crosses it:
  a pane's environment (its pane token, a pool token) and its command (the
  prompt) go into a launch file that `write-as` writes as the member (`0600`,
  from stdin) and `/opt/agenthub/libexec/pane-launch` reads, deletes and
  execs. File contents go to `write-as` on stdin.
- **hidepid is required.** `/proc` is mounted `hidepid=invisible`, with group
  `agenthub-proc` (the host account, `polkitd`) exempt. Without it any account
  could read another's command lines. `install-system` stops if the remount
  fails.
- **Hooks are authenticated by the sender's uid**, read from the kernel's
  table of loopback connections (`/proc/net/tcp*`, `server/lib/hook-auth.js`),
  not from anything the hook says.
- **The AI Proxy gateway.** A pane on an AI Proxy profile gets
  `http://127.0.0.1:<port>/api/proxy-gw/<profile>` and its own pane token as
  the key; AgentHub swaps in the real key upstream
  (`server/lib/proxy-gateway.js`, see [AI Proxy profiles](#ai-proxy-profiles)).
  A Claude pane gets the token in a per-run `--settings` file, so a user or
  project `env.ANTHROPIC_*` cannot override it.
- **Shared tools.** Members run every AI tool, node and AgentHub's own scripts
  from `/opt/agenthub` (`server/lib/shared-tools.js`), first on their pane's
  `PATH`, with self-update off (`DISABLE_AUTOUPDATER=1`).

### What install-system sets up

The pieces, and what each is for (the install steps are in the
[runbook](#install)):

| Piece | What it is |
| --- | --- |
| `ah-<name>` | Made by `agenthub-provision`, not by the installer. One Linux user per AgentHub user, home `/srv/agenthub/users/<name>` (`0750`, group `agenthub`): the member owns it, the dashboard reads it, nobody else gets in. `.agenthub/{tmux,handoff}` (0700) is made by the member: their tmux socket, and the files the dashboard writes for their panes (as them) |
| groups | `agenthub` (reads member homes: the host account), `agenthub-members` (every `ah-*` account), `agenthub-proc` (sees every process: the host account, `polkitd`) |
| `agenthub.service` | A SYSTEM unit, `User=<host>`, with `SupplementaryGroups=agenthub agenthub-proc` and `AmbientCapabilities=CAP_DAC_READ_SEARCH`: the dashboard can read every member's tool files (Claude writes transcripts `0600`), but cannot write them. Every program it starts has the capability removed first (`setpriv`), so host panes, git and npm do not inherit it. It reads node and the plugin root from `<data dir>/runtime/agenthub.system.env`, a file the host owns, so a plugin update repoints it without root |
| `/etc/sudoers.d/agenthub` | The host may run only `tmux`, `git`, `gh` (by the absolute paths recorded in `/etc/agenthub/server.conf`), `/opt/agenthub/libexec/write-as` and `/opt/agenthub/tools/bin/claude` (the [plugin baseline](#plugin-baseline-opt-in)) AS a member, plus, as root, only `/opt/agenthub/sbin/agenthub-provision` and `systemctl start/stop/restart/enable/disable agenthub.service`. Members are never in sudoers |
| `/etc/agenthub/server.conf` | Root's, `0644`: the host account and uid, the users root, the port, and the `tmux`/`git`/`gh` paths sudoers lists. Its presence turns S3 on |
| `write-as`, `pane-launch` | `/opt/agenthub/libexec/`, root's. `write-as <path> [append\|replace\|private\|mkdir\|remove]` runs AS a member and does the one change, content on stdin. `pane-launch <file>` starts a member's pane from its launch file |
| `agenthub-provision` | The one root command besides `systemctl` on this unit: `create`, `disable` or `enable` `ah-<name>`. It refuses an account that exists outside `agenthub-members` and never deletes anything |
| `/etc/claude-code/managed-settings.json` | Members' Claude Code hooks, through `/opt/agenthub/libexec/member-hook`, which does nothing for the host account (its plugin reports). A foreign file there is left alone |
| `hidepid=invisible` | `/proc` (now and in `/etc/fstab`): an account sees only its own processes, so no member can read another's command line. A token passed with `tmux -e` is in one |
| `/opt/agenthub/{tools,app}` | Owned by the host account: the dashboard updates them, every account runs them. `tools/bin` holds node and every installed AI tool; this install's hook handler, CLI, MCP server and statusline go in `app`, with the `agenthub` launcher in `tools/bin`. The AI tools are published there after each install or update from the AI Tools page, at server start, and by `agenthub server tools`. A publish copies a native binary, an npm package's folder, or a script's own version folder. A tool that is not published yet answers a member's start with `409 ENOTSHARED` |

What runs as the member, and how:

- **Panes.**
  - Each member has their own tmux server (`~/.agenthub/tmux/sock`). Every tmux command on it, attaching a browser terminal included, runs as `sudo -u ah-<name> tmux …`. The dashboard never connects to it as itself: the member controls that socket.
  - sudo logs every command line it runs, so a pane's environment (its tokens) and its command (the prompt) are not on it. `write-as` writes them to a launch file (`0600`, from stdin), and `/opt/agenthub/libexec/pane-launch` reads it, deletes it and starts the command. A prompt pasted later goes in on stdin.
  - A pane gets the member's `HOME`, `CODEX_HOME` and `PATH` (`/opt/agenthub/tools/bin` first), with `DISABLE_AUTOUPDATER=1` and none of the host's desktop variables.
- **Git, `gh` and file saves.**
  - In a member's folder, git and `gh` run as `sudo -n -u ah-<name> git …`, so git's own ownership check passes and every file stays the member's.
  - The editor and `.gitignore` saves go through `/opt/agenthub/libexec/write-as`.
  - A file uploaded in the Files tab (`POST /api/lanes/:id/git/upload`) is written the same way, AS the member through `write-as` (content on stdin), and only as a NEW file in an existing folder of the workspace — never over an existing name, never into `.git`, never a home dot folder, at most 25 MB. With **Download** on a file (`GET /api/lanes/:id/git/download`, read like any member file below, up to 512 MB) this is how a member with no SSH or `scp` to the host moves files in and out of their own folder; both work in a folder that is not a git repo.
  - AgentHub's stored git logins are never handed to a member's git, and a password typed in the git bar goes into the member's OWN store (below) for that one call — through `write-as`'s stdin, never sudo's argv or env — and out again unless "remember" was ticked; a store that cannot be read is never rewritten (`409 ESTOREUNREADABLE`). The member keeps their OWN identity in Settings → Git → **Your git identity** (`/api/me/git`): a commit author (`user.name`/`user.email` in their `~/.gitconfig` — commits fail without one; a new member has none), an SSH key AgentHub makes for them and HTTPS logins, both under `~/.config/agenthub/` in their home, written as them by `write-as` (0600). Add workspace → **Clone** (`POST /api/lanes/clone`) clones as them with it and writes it into the clone's own config (`core.sshCommand` / `credential.helper=store --file=…`), so the git bar and their agents' pushes in that repo use it too. A login typed in the Clone dialog goes to their store through `write-as`'s stdin, never sudo's argv or env, and is removed again after the clone unless "remember" was ticked.
- **Files a pane reads.** These are written into `~/.agenthub/handoff/` AS the member (`write-as`). The dashboard never writes, changes or deletes anything in a member's home itself: the member could have swapped the name for a link. The files are:
  - the workspace prompt and its context files;
  - the AI Proxy settings: no key, only that pane's own token for the gateway;
  - files dropped into the prompt box;
  - a pane's launch file.

  Codex gets its proxy as `-c` overrides, never a file in the member's `CODEX_HOME`.
- **Reading a member's files.** A transcript, a Codex, Antigravity or Cursor journal, or a file in the workspace is read only when the opened file is a regular file the member owns. A link to another account's file is refused, and so is a FIFO. A journal speaks only for its own account: its session's folder must be inside that home. A member's Antigravity conversation (SQLite, which opens its `-journal`/`-wal` by name) is read from a private copy; the walk that turns a Cursor project slug back into a folder follows no link and gives up after 500 folders.
- **Hooks.** A hook is matched to its sender's Linux account, read from the kernel's table of loopback connections.
  - A member's Claude Code reports through `/etc/claude-code/managed-settings.json` → `/opt/agenthub/libexec/member-hook` → `/opt/agenthub/app/scripts/hook-handler.js`.
  - The host's hooks are accepted as before.
  - A member's hook may report only work inside their own home: another account's folder, session or pane is `403 EFOREIGN`, and a transcript path outside the home is dropped.
  - Any other account is `403 EPEER`.
- **Liveness.** A member's session counts as alive while the pane holding it lives, because the dashboard cannot read another account's process directory. A session the member started in their own shell ends once it has been idle for the idle timeout.

Not offered for a workspace in a member's folder (`403 EMEMBERLANE`): the
Add-workspace **Worktree** mode and the runtime layer (`lanes up/down/hook`,
profiles, proof, lane MCP). Both run as the dashboard's own account. Use
**Repo** mode and the prompt box's **New worktree** instead.

**First sign-in.** Every account gets AgentHub's own setup once, in the
browser: the default agent and permission mode, the look (System, Dark or
Light, a theme for each, language, terminal font), the pet and notifications,
then a guided tour to the first workspace and session. The choices are stored
per account (`/api/me/prefs`), so they follow the user to any browser; an
account that already has a workspace skips it, and Settings → Appearance →
**Run setup again** brings it back.

A member's own Claude Code skips its first-run questions: `agenthub-provision`
marks its onboarding done and trusts the member's home (which covers every
folder under it), and the dark/light mode picked in AgentHub becomes Claude's
theme. Accepting Bypass Permissions mode stays the member's own choice, made
in their terminal.

### Plugin baseline (opt-in)

Write `/etc/agenthub/plugin-baseline.json` (as
root) to give every NEW member your plugins:

```json
{ "marketplaces": ["owner/repo"], "plugins": ["name@marketplace"] }
```

After a user is created, AgentHub runs, as that member and in the
background, `claude plugin marketplace add <source>` for each marketplace and
then `claude plugin install --scope user <plugin>` for each plugin; each
result is logged as `[plugin-baseline]`. `agenthub users provision <name>`
(or `--all`) runs it again in the foreground. No file, nothing runs. `-y` is
never passed, so a plugin that installs by running a marketplace-declared
command fails here; the member installs it from their own terminal. Members
add their own plugins with `/plugin` as usual.

### What the second review closed

The 2026-10-09 review looked for ways a member can point the dashboard's own
rights (`CAP_DAC_READ_SEARCH`, the host account, sudo as any member) at files
that are not theirs. Each case below is refused, with a test in
`server/__tests__/server-s3-review2.test.js`:

- **A workspace folder swapped for a link.** The file view, image view, blame
  and the file editor resolve the workspace's real path and refuse it
  (`403 EFOREIGN`) when it lands in another account's files — another
  member's folder, the host's home, `/etc`. The account a read or save runs as
  is never taken from where a link points.
- **A member's git dir naming a path outside their home.** The `info/exclude`
  line AgentHub adds for a new worktree or `.mcp.json` is written only when the
  repo's real git dir is the same account's; a FIFO there cannot hang the
  dashboard.
- **Browsing and branch lists.** A developer's `GET /api/lanes/browse`, the
  browser's New folder (`POST /api/lanes/browse/mkdir`, made as the member) and
  `GET /api/lanes/branches` stay inside their own folder (`403 EFOLDER`), and
  so does a workspace's extra folder (`POST /api/lanes/:id/sources/folder`,
  refused before the path is looked at, so the answer says nothing about
  another member's files).
- **The top of a member's home.** Its dot folders (`.claude`, `.codex`,
  `.config`, `.ssh`, `.agenthub`…) hold logins and pane tokens: no workspace
  file view serves them to anyone (`403 ESECRET`), and the home itself is no
  workspace.
- **Transcripts by session id.** A scan by id looks only in the session
  owner's home, so a file another member plants under that id is never shown.
- **Codex journals.** A rollout journal sets the status only of a lane in its
  own account's home, and the newest-40 window is per home.
- **A disabled member's sessions.** Opening one reads from their home instead
  of crashing the server, and the liveness reaper skips them instead of
  stopping.
- **Host terminals.** A browser terminal's `tmux attach` (node-pty, which
  forks outside `child_process`) is started through `setpriv` too, so it does
  not keep `CAP_DAC_READ_SEARCH`.
- **AI Proxy edits.** Editing a profile rewrites a running gateway pane's own
  settings file (with its own token), and deletes the file of a pane that has
  ended.

### What review round 5 closed

The fifth review (2026-10-09) looked at the dashboard's ONE thread and at who
a pane really is. Each case below is refused or bounded, with a test in
`server/__tests__/server-s3-review5.test.js`:

- **AgentHub's data dir is never a config file.** `GET /api/cc-config/file`
  refuses anything in it — the database, the push keys, and each gateway
  pane's settings file, which holds that pane's live token.
- **A member's stopped or flooding tmux server.** Every call on it has a 3 s
  limit and a 1 MiB answer cap; one that hit either is not asked again for a
  minute. Meanwhile only that member's panes count as unknown (kept, not
  ended); everyone else's are still checked. Listing a member's panes (every
  poll: the pane list, the workspace list, the 10 s heal beat, the liveness
  reaper) never waits for them: it runs in the background and the poll reads
  its last answer, so a stopped server stalls nobody. Until a first answer,
  or once the last is 15 s old, their panes count as unknown too. A call a
  click makes on a member's pane (start, type, stop) still waits, up to the
  3 s. Every other synchronous call
  that runs as a member (git in their folder, `write-as`) stops after 5 s.
- **A member's huge or sparse files.** Transcripts and journals are read line
  by line with a 64 MiB cap per line (a 512 MB line used to crash the server);
  a whole member file is read only up to a cap (1 MiB for a login or a
  settings file, 16 MiB for an index or a workflow journal, 256 MiB for a
  SQLite copy, 64 MiB otherwise); a member transcript over 256 MiB is neither
  parsed nor snapshotted, and the snapshot keeps a sparse file's holes. The
  snapshot copy is asynchronous (1 MiB at a time), so a big one never stalls
  the dashboard for other users, and it lands by rename — never half-written.
- **A request that makes a route throw** answers 500 instead of stopping the
  server (a member folder of a gone user crashed `POST /api/lanes/worktree`).
- **Who a pane runs as.** A non-admin owns no workspace outside their home
  (above), a pane's folder is its workspace's account's, and a pane token
  works only from that account's uid.
- **A path an analyst names.** Browsing, branch lists and the `@` file search
  stay in their own folder or workspaces, like a developer's.
- **Drops** are saved where the pane that reads them can read them: as the
  workspace's account (the client names the workspace), not the dropper's.
- **The Codex pool and ChatGPT Teams.** A login is the pool's only when
  AgentHub placed it in that home AND it is the same person (email): every
  seat of a Team shares one `account_id`, so a member's own seat is theirs.

### Accepted ceilings

- The dashboard process can read any file, `/etc/shadow` included. `sudo`
  needs that file, so the unit cannot hide it (`InaccessiblePaths` hides only
  `/root`). Treat the host account as privileged, and keep it out of the
  `sudo` group on a real server: there the sudoers file above is all it may
  do.
- Anything that runs as the host account can run anything as any member: the
  sudoers file lets it start `tmux` as them. A host compromise is every
  member's compromise.
- `install-system` installs and runs code from the host account's install, as
  root. Re-run it only from a tree you trust.
- sudo's log (readable by root and `adm`) shows members' commit messages,
  branch names, pull-request titles and file paths, never a token or a prompt.
- A member may point a personal AI Proxy at any URL, so the dashboard will
  POST to it. They have a shell on the machine anyway.
- The gateway forwards any path under `/v1/` to the profile's upstream, and an
  agent can spend through a profile while its pane lives.
- A Claude pool token is in its pane's environment: the member of that pane
  can read it (see [Claude account pool](#claude-account-pool)). The same goes
  for a Codex pool API key, and for a Codex ChatGPT login written into the
  member's `~/.codex/auth.json` ([Codex account pool](#codex-account-pool)).
- The Codex pool's quota reading comes from rollout journals, which their
  member writes: a member can forge the reading of their OWN session (say a
  pool account is full, or empty) and so steer the balancer for everyone
  until a newer reading wins. They cannot speak for another account's run:
  a run's rollout is read only from its own account's `CODEX_HOME`, and never
  one dated in the future.
- A member session started in the member's own shell, outside a dashboard
  pane, counts as alive only until its idle timeout: the dashboard cannot read
  another account's `/proc/<pid>`.

## Operator runbook

Every command below runs as the **host account** (the one that runs the
dashboard) unless it says `sudo`. `agenthub server install-system` calls sudo
itself, once.

### After an update that changes the member helpers

`write-as`, `pane-launch` and the sudoers file are copied to
`/opt/agenthub/libexec` by `install-system` only — an update of AgentHub does
not touch root-owned files. When a release changes them (v2.7.7: `write-as`
learned `move`/`copy` for the File Explorer), re-run it once:

```bash
sudo agenthub server install-system
```

Until then the File Explorer's move, copy and delete in a member's folder
answer `409 EUPGRADE` (listing, upload, new folder/file and zip work).
The same re-run installs `agenthub-provision delete`, behind the Users page's
**Delete user**: until then a delete answers `502 EPROVISION` ("unknown
action: delete") and changes nothing.

### Deleting a user

Users page → Edit → **Delete user** (the admin's own password, then a red
confirm). Order: the user's logins end and their workspaces' panes stop;
under S3 `agenthub-provision delete <name>` locks `ah-<name>`, kills its
processes, `userdel`s it and removes `<USERS_ROOT>/<name>` (a real
directory exactly there — never a link — `rm -rf --one-file-system`); only
then are the rows removed (the user, their workspaces' records, prefs, push
subscriptions, personal AI Proxy profiles). Session history and the audit log
stay. Never yourself (`ESELF`) or the last admin (`ELASTADMIN`). A user made
again under the same name gets a NEW, empty Linux account and home.

### What a new machine needs before `install-system`

`install-system` does not do any of this:

1. A Linux machine with systemd, kernel ≥ 5.8 (for `hidepid=invisible`),
   `sudo` with `visudo`, the shadow tools (`useradd`, `usermod`, `groupadd`),
   util-linux (`runuser`, `setpriv` at `/usr/bin/setpriv`, `mount`), `pkill`,
   `sha256sum`.
2. A host account that is not root, with sudo rights for this one run (take
   them away again afterwards, see step 9 below).
3. Node (the version AgentHub needs), `tmux`, `git` and, if you want it, `gh`
   on that account's `PATH`. Their absolute paths are recorded once, in
   `/etc/agenthub/server.conf` and the sudoers file.
4. AgentHub installed for the host account, as the plugin or a checkout
   (`npm run setup && npm run build`), and running once
   (`agenthub start`).
5. Server mode on, with the first admin:
   `agenthub server setup --hostname agenthub.example.com --admin boss`
   (see [Set it up](#set-it-up)).
6. The AI tools members will use, installed for the host account (the AI
   Tools page or by hand).
7. The HTTPS reverse proxy (see [Put HTTPS in front of it](#put-https-in-front-of-it)).

### Install

```bash
agenthub server install-system --dry-run   # print the one sudo command it will run
agenthub server install-system             # install, then switch to the system unit
agenthub users provision --all             # Linux accounts for users made earlier (the admin from setup)
agenthub server tools                      # publish the AI tools to /opt now (the server also does it at start)
```

What `agenthub server install-system` does, in order (`installSystem` in
`bin/agenthub.js`, root half `scripts/server/install-system.sh`; running it
again changes nothing that is already in place):

1. Refuses to run as root. Resolves, as the host account: its name, node, this
   install's root, the data dir, the port (`DASHBOARD_PORT`, else 4820) and
   the paths of `tmux`, `git` and `gh`.
2. As root (`sudo bash install-system.sh …`), after checking each value:
   1. creates groups `agenthub`, `agenthub-members`, `agenthub-proc`; adds the
      host account to `agenthub` and `agenthub-proc`, and `polkitd` (if it
      exists) to `agenthub-proc`;
   2. creates `/srv/agenthub/users` (`0751` root), `/opt/agenthub/{sbin,libexec}`
      and `/etc/agenthub` (root), `/opt/agenthub/{tools,tools/bin,app}` (the
      host account's);
   3. writes `/etc/agenthub/server.conf` (`HOST_USER`, `HOST_UID`,
      `USERS_ROOT`, `PORT`, `TMUX_BIN`, `GIT_BIN`, `GH_BIN`);
   4. installs `agenthub-provision`, `write-as` and `pane-launch` from
      `scripts/server/`;
   5. copies node to `/opt/agenthub/tools/node/<version>` (linked from
      `tools/bin/node`) and this install's `scripts/` and `server/lib/` to
      `/opt/agenthub/app`;
   6. writes `/opt/agenthub/libexec/member-hook` (the port is baked in);
   7. writes `/etc/sudoers.d/agenthub`, only after `visudo -c` accepts it;
   8. writes `/etc/claude-code/managed-settings.json` (the member hooks) unless
      a file that is not AgentHub's is already there;
   9. `chmod o-rwx` the host's `~/.claude`, and warns when the host's home is
      open to other accounts;
   10. adds the `hidepid=invisible,gid=agenthub-proc` line to `/etc/fstab`
       (backup: `/etc/fstab.agenthub-bak`) and remounts `/proc` with it;
   11. writes `/etc/systemd/system/agenthub.service` and runs
       `systemctl daemon-reload`. It does not start it.
3. As the host account again: writes `<data dir>/runtime/agenthub.system.env`
   (node, plugin root, port, client dist), sets `AGENTHUB_USERS_ROOT` in
   AgentHub's settings file, and disables the `systemd --user` unit (its tmux
   panes stay).
4. Stops a dashboard still running without the system unit, then
   `sudo -n systemctl enable` and `start --no-block agenthub.service`, and
   fails unless the unit is active 3 seconds later.

What a new machine still needs by hand after it:

1. `agenthub users provision --all`: users created before S3 was on (the
   admin from `server setup` is one) have no Linux account yet.
2. Log out and back in once, so your own shell joins `agenthub` and
   `agenthub-proc` (the unit gets them on its own).
3. `chmod o-rwx ~` for the host account if the installer warned: the database
   lives under it.
4. If the installer left a foreign `/etc/claude-code/managed-settings.json`
   alone, add the member hooks to it yourself: for each of `PreToolUse`,
   `PostToolUse` (both with `"matcher": "*"`), `Stop`, `SubagentStop`,
   `Notification`, `SessionStart`, `SessionEnd` and `UserPromptSubmit`, a
   `command` hook `/opt/agenthub/libexec/member-hook <Event>` with
   `"timeout": 5` — and `"async": true` on `PreToolUse` and `PostToolUse`, so
   Claude Code does not wait a Node start before and after every tool call.
   Re-running `install-system` rewrites a managed file it wrote itself.
5. Any other daemon that must see every process (monitoring agents, for
   example) needs group `agenthub-proc`: `sudo usermod -aG agenthub-proc <it>`.
6. A tool `agenthub server tools` reports as `ESHAREDLAYOUT` (a script in a
   shared folder) must be installed under `/opt/agenthub/tools` by hand.
7. Remove the host account from the `sudo` group (`sudo deluser <host> sudo`
   on Debian/Ubuntu). Re-running `install-system` later needs it back for
   that run.
8. Pool accounts, if you use a pool: `agenthub accounts add <name>` (Claude)
   or `agenthub accounts add <name> --tool codex --api-key|--from <dir>`
   ([Codex account pool](#codex-account-pool)).
9. Optional: `/etc/agenthub/plugin-baseline.json` for new members' plugins
   ([Plugin baseline](#plugin-baseline-opt-in)).

### Add, disable and enable a user

```bash
agenthub users add alice [--role admin|analyst|developer]   # or the Users page
agenthub users disable alice
agenthub users enable alice
agenthub users provision alice      # the Linux account only (a user made before S3, or after a failed create)
```

- **Add** creates the Linux account FIRST (`agenthub-provision create`):
  `useradd ah-alice` with home `/srv/agenthub/users/alice`, group
  `agenthub-members`; the home is set to `ah-alice:agenthub 0750`; as the
  member it makes `~/.agenthub/{tmux,handoff}` (`0700`) and marks Claude
  Code's onboarding done and the home trusted in `~/.claude.json`. If any of
  that fails, no AgentHub user is made (`502 EPROVISION`, with the helper's own
  words). An account `ah-alice` that already exists outside
  `agenthub-members`, or an existing folder `/srv/agenthub/users/alice` with no
  account, is refused. With a [plugin baseline](#plugin-baseline-opt-in)
  file in place, its plugins are then installed as the member in the
  background.
- **Disable** disables the AgentHub user and ends their logins FIRST, then
  locks the Linux account (`usermod --lock --expiredate 1`) and kills every
  process of `ah-alice` (`pkill -KILL -u`: their tmux server and every pane).
  A failed lock still answers `502 EPROVISION`, but the logins are already
  gone. A user with no Linux account (made before S3, or a name that does not
  fit) is disabled without any sudo. Their files stay.
- **Enable** clears the expiry date (when there is a Linux account) and
  re-enables the AgentHub user.
- **Delete**: AgentHub never deletes a user, an account or a home. To remove
  one for good, disable it, then by hand: `sudo userdel ah-alice`, and move or
  delete `/srv/agenthub/users/alice` yourself. The AgentHub user row stays.
- **A name that does not fit** (upper case, a dot): `provision` refuses it
  (`provision --all` reports it and goes on with the next user, exit code 1).
  Create a new user with a fitting name, move the old workspaces' folders into
  their home (`/srv/agenthub/users/<name>/…`, owned by `ah-<name>`) and add
  them there again. `agenthub users assign <workspace id> <name>` gives a
  workspace only to the member whose home holds it, or — for a folder outside
  the users root, whose panes run as the host — to an admin; anything else
  answers `EOWNERFOLDER`.

### Upgrade

- Update AgentHub as usual (Settings → Updates, `/agenthub-update`). The
  bootstrap repoints the unit by rewriting
  `<data dir>/runtime/agenthub.system.env` (a host-owned file, no root) and
  restarts it with `sudo -n systemctl restart --no-block agenthub.service`.
  Panes survive (`KillMode=process`).
- At start, the server republishes its own scripts to `/opt/agenthub/app`
  and every installed AI tool to `/opt/agenthub/tools`, in the background. An
  install or update from the AI Tools page publishes that tool right after.
- An update never touches a root-owned file. **Re-run `agenthub server
  install-system`** when a release changes anything under `scripts/server/`
  (provision helper, `write-as`, `pane-launch`, the member hook, sudoers, the
  unit; an install from before the plugin baseline needs it once for the
  `claude` sudoers entry), when the port changes (the member hook has it baked in), and when
  `tmux`, `git` or `gh` moves to another path (sudoers allows only the
  recorded ones).
- Keep every other AgentHub server off the same database (a dev server on
  another port). A server without S3 cannot see members' tmux servers, so from
  v2.7.2 on it leaves any run and workspace under the users root (read from
  `/etc/agenthub/server.conf`) alone; a build older than v2.7.2 marks their
  runs interrupted or abandoned within seconds, and the proxy gateway then
  refuses those panes (`401 not a live pane's token`). A dev server that DOES
  run in server mode (it read `AGENTHUB_MODE=server` from
  `~/.config/agenthub/settings.env`) but lacks the unit's
  `CAP_DAC_READ_SEARCH` cannot stat a member's 0700 `.agenthub/tmux`; from
  v2.7.3 on that `EACCES` counts as "maybe running" and the member's list is
  asked through sudo, never read as "no panes". And a run younger than 20 s
  is never ended by any server: only the server that started a pane writes it
  into its own cached tmux list, so another one on the same database had not
  listed it yet and ended a Codex pane seconds after its start.
- **Update now** under the system unit restarts through
  `sudo -n systemctl restart --no-block agenthub.service` (from v2.7.3). An
  older build started a detached replacement instead, which kept the port
  while `Restart=always` started the unit every few seconds, each start exiting
  "already running" — `systemctl status agenthub.service` shows
  `activating (auto-restart)` with `MainPID` 0. Recover with
  `kill -TERM <pid of the server on the port>`; the unit then starts on the
  new code.
- A member's tmux list is re-asked at most every 5 s (one `sudo` each, which
  writes three `auth.log` lines); a pane the dashboard starts or stops shows at
  once, one that exits on its own within those 5 s.
- Claude Code does not carry a folder's trust down from the member's home
  (which provisioning trusts): a member may be asked "Trust this folder" once
  per workspace, in the terminal. AgentHub never answers it for them — it is
  what keeps a cloned repository's own settings and hooks from running unasked.
- A member's first project folder: a workspace named on a folder that does not
  exist yet inside their home is created at its first start, AS the member
  (from v2.7.4; their home itself is no workspace).
- Bypass in a pane's shift+tab cycle (`--allow-dangerously-skip-permissions`)
  is offered only to an account that already accepted Claude Code's bypass
  warning (`skipDangerousModePermissionPrompt` in its `~/.claude/settings.json`)
  or to a run that starts in bypass itself. Before v2.7.4 every new member's
  first session opened on that warning, which the Chat UI cannot answer.
- A developer's System page (`GET /api/settings/info`, `GET /api/updates/status`)
  carries no path: not the database's, the hooks file's, the install folder,
  nor the transcript cache's keys — those are every user's transcript paths.

### Undo and uninstall

```bash
agenthub server install-system --uninstall   # back to the systemd --user unit
```

As root it stops, disables and deletes the system unit (only if it is
AgentHub's), deletes the sudoers file, the managed settings (only if
unchanged since it wrote them), the `/etc/fstab` line (backup:
`/etc/fstab.agenthub-bak`; `/proc` is remounted `hidepid=off`),
`/opt/agenthub/{sbin,libexec}` and `server.conf`. Then, as the host account,
it repoints the `systemd --user` unit at this install and runs `systemctl
--user enable --now agenthub`.

It keeps `/srv/agenthub/users` and every home, the `ah-*` accounts, the groups
and `/opt/agenthub/{tools,app}`. It does not stop members' running tmux
servers: disable those users first if they should end. With `server.conf`
gone, S3 is off and server mode works as without it.

To remove everything:

1. `agenthub server install-system --uninstall`
2. `agenthub server off`, then `agenthub uninstall --yes` (the Uninstall
   button refuses in server mode, `ESERVERMODE`).
3. By hand, as root, only if you want them gone: `userdel` each `ah-*`
   account, `groupdel agenthub agenthub-members agenthub-proc`, and delete
   `/srv/agenthub`, `/opt/agenthub` and `/etc/agenthub`. These hold members'
   work: AgentHub never deletes them.

### Troubleshooting

| Symptom | Cause, and what to do |
| --- | --- |
| `install-system: generated sudoers did not validate` | Nothing was installed past that step. Check the tool paths it printed (`--dry-run`). |
| `install-system: …/managed-settings.json exists and is not ours` | Members' Claude hooks are not installed; add them by hand (above). |
| `install-system` stops at the `/proc` remount | The kernel does not support `hidepid=invisible` (< 5.8). The `/etc/fstab` line is already added: remove it or restore `/etc/fstab.agenthub-bak` before rebooting. |
| `a dashboard is still running — stop it first` | A dashboard outside both units holds the port: `agenthub stop`, then run it again. |
| `agenthub.service did not stay up` | `sudo journalctl -u agenthub -n 50`. The unit reads `<data dir>/runtime/agenthub.system.env`. |
| `502 EPROVISION` when adding a user | The helper's own message says why (name, existing account or folder). No AgentHub user was made. |
| `400 EBADUSER` | The username does not fit the Linux rule (above). |
| `409 EMEMBERGONE` on a start or a git action | The workspace's folder is under the users root but its user is disabled, unknown or not provisioned: `agenthub users enable <name>` or `agenthub users provision <name>`. |
| `409 ENOTSHARED` on a member's start | That tool is not published to `/opt/agenthub/tools` yet: `agenthub server tools`. |
| `403 EMEMBERLANE` | Worktree mode and the runtime layer are not offered in a member's folder; use Repo mode and the prompt box's New worktree. |
| `400 EMEMBERCRED` | Older builds only: a member's typed git login was refused. Now it goes into their own store (Settings → Git → **Your git identity**). |
| `409 EOWNERFOLDER` (on a write to an existing workspace) | Its owner may no longer own that folder by their CURRENT role — typically an admin demoted to developer whose workspace is outside the users root (it runs as the host). Reads still work; an admin reassigns the workspace or restores the role. |
| `403 EGATEWAY` | A request relayed by the AI Proxy gateway reached the dashboard itself (a proxy profile whose base URL points back at AgentHub). Point the profile at the real proxy. |
| `409 ESTOREUNREADABLE` | The dashboard cannot read the member's `~/.config/agenthub/git-credentials` (the unit lost `CAP_DAC_READ_SEARCH`, or the file is not the member's own regular file); nothing is rewritten. Check `grep CapAmb /proc/<pid>/status` shows `0000000000000004`. |
| `403 EPEER` on a hook | The hook came from an account that is neither the host nor an enabled member, or the uid could not be read. |
| `403 EFOREIGN` on a hook | A member's hook named a folder, session or pane outside their own home. |
| `403 EFOREIGN` on a file view or save | The workspace's folder is now a link into another account's files. Put the real folder back, or make a workspace on the real one. |
| A member's runs show as abandoned or interrupted, or Claude in a member's pane gets `401 not a live pane's token` | Another AgentHub server older than v2.7.2 on the same database (see Upgrade). |
| `sudo: a password is required` in the server log | The sudoers file is missing, or a tool moved off its recorded path: re-run `install-system`. |
| `[drop-caps] setpriv is missing` | Install util-linux's `setpriv` at `/usr/bin/setpriv`; until then the dashboard's children inherit `CAP_DAC_READ_SEARCH`. |
| `[plugin-baseline] <name>: claude is not in the shared tools yet` | `claude` is not published to `/opt/agenthub/tools` yet: `agenthub server tools`, then `agenthub users provision <name>`. |
| `[plugin-baseline] <name>: … failed` with `sudo: a password is required` | The sudoers file predates the baseline: re-run `install-system`. |
| A member's first Claude run asks about Bypass Permissions | Expected: that consent stays the member's own. |

## Turn it off

```bash
agenthub server off
systemctl --user restart agenthub
```

Accounts are kept, so `agenthub server setup` turns server mode back on with
the same users.