diff --git a/docs/SERVER-MODE.md b/docs/SERVER-MODE.md index db9c9a64..41825a1f 100644 --- a/docs/SERVER-MODE.md +++ b/docs/SERVER-MODE.md @@ -1156,6 +1156,27 @@ The same re-run installs `agenthub-provision delete`, behind the Users page's **Delete user**: until then a delete answers `502 EPROVISION` ("unknown action: delete") and changes nothing. +The same re-run (v2.8.68) keeps a member's test run from freezing the box — +on 11-02 some 36 open sessions plus test runs filled RAM and the 4 GB swap on +a slow disk, and only a hard reset brought it back: + +- `pane-launch` starts every member pane at `nice 10`, best-effort IO + priority 7 and `oom_score_adj 500`, and defaults `VITEST_MAX_THREADS` / + `VITEST_MAX_FORKS` to 2 (Vitest ≤ 3; a member may set their own). Only + panes started after the re-run get these; the nice cannot be taken back + without root. +- `agenthub.service` gets `MemoryMax=85%`, `MemorySwapMax=1G`, + `OOMPolicy=continue` and `OOMScoreAdjust=-100`. The cgroup holds the server + AND every member pane, so memory runs out inside the unit first, and the + kernel kills the biggest pane process there — not the dashboard, tmux, or + the rest of the machine. `daemon-reload` (which the re-run does) applies + the memory caps to the running unit; `OOMScoreAdjust` takes effect at the + next restart (`sudo -n /usr/bin/systemctl restart --no-block + agenthub.service` — panes survive, `KillMode=process`). + +Check with `systemctl show agenthub.service -p MemoryMax,MemorySwapMax,OOMPolicy` +and, in a new member pane, `ps -o ni= -p $$; cat /proc/self/oom_score_adj`. + ### Deleting a user Users page → Edit → **Delete user** (the admin's own password, then a red diff --git a/scripts/server/install-system.sh b/scripts/server/install-system.sh index bf84ee1f..2c7e01c0 100644 --- a/scripts/server/install-system.sh +++ b/scripts/server/install-system.sh @@ -20,7 +20,8 @@ # /etc/sudoers.d/agenthub checked with visudo before it is installed # /etc/claude-code/managed-settings.json the member hooks (never over a foreign file) # /proc hidepid=invisible,gid=agenthub-proc (now and in /etc/fstab) -# /etc/systemd/system/agenthub.service reads the host-owned env file +# /etc/systemd/system/agenthub.service reads the host-owned env file; caps +# the cgroup (server + every pane) at 85% of RAM, 1G swap # @author Nguyễn Ngọc Trí Vĩ set -euo pipefail @@ -203,6 +204,14 @@ ExecStart=/usr/bin/env \${AGENTHUB_NODE} \${AGENTHUB_ROOT}/server/index.js Restart=always RestartSec=3 KillMode=process +# Every member pane lives in this cgroup. Capped below RAM with little swap +# (swap on a slow disk froze the whole box on 11-02), a runaway test run is +# OOM-killed inside the unit — pane-launch makes panes the first pick — and +# "continue" keeps the dashboard up when that happens. +MemoryMax=85% +MemorySwapMax=1G +OOMPolicy=continue +OOMScoreAdjust=-100 [Install] WantedBy=multi-user.target diff --git a/scripts/server/pane-launch.sh b/scripts/server/pane-launch.sh index 747c71b9..f2608f2e 100644 --- a/scripts/server/pane-launch.sh +++ b/scripts/server/pane-launch.sh @@ -10,7 +10,8 @@ # the environment and execs the command; no shell ever parses a value. # # The file is NUL-separated: the number of environment entries, the -# NAME=value entries, then the argv (none = a login shell). +# NAME=value entries, then the argv (none = a login shell). Every pane starts +# at a lower CPU/IO priority and as the kernel's first OOM pick (see below). # @author Nguyễn Ngọc Trí Vĩ set -eu f=${1:?usage: pane-launch } @@ -22,6 +23,20 @@ for kv in "${parts[@]:1:n}"; do [[ "$kv" =~ ^[A-Za-z_][A-Za-z0-9_]*= ]] || { echo "pane-launch: bad variable" >&2; exit 2; } export "$kv" done +# A member's pane shares the 4-core box with the dashboard and every other +# member: its test run, build or dev server must slow down before anything +# else does, and die first if memory runs out (11-02: 36 sessions plus test +# runs filled RAM and swap, and the box froze until a hard reset). A member +# cannot take the nice back (that needs root); oom_score_adj they can lower +# only to the unit's own -100, and even then the biggest process dies. The unit +# caps the whole cgroup below RAM (install-system.sh), so the kernel's pick +# inside it is the biggest pane process, never the dashboard or tmux. +renice -n 10 -p $$ >/dev/null 2>&1 || true +ionice -c 2 -n 7 -p $$ >/dev/null 2>&1 || true +echo 500 >/proc/self/oom_score_adj 2>/dev/null || true +# Vitest ≤ 3 starts one worker per core by default and these override the +# config; a member may set their own in the pane. +export VITEST_MAX_THREADS="${VITEST_MAX_THREADS:-2}" VITEST_MAX_FORKS="${VITEST_MAX_FORKS:-2}" argv=("${parts[@]:n+1}") if [ ${#argv[@]} -eq 0 ]; then exec -l "${SHELL:-/bin/bash}"; fi exec "${argv[@]}"