fix(drop): stream every dropped file to disk; no local size cap

- POST /api/run/drop-file pipes the body to its file as it arrives in every
  case — a member's (S3) through `write-as … replace` with the bytes on
  stdin — instead of buffering up to 20 MB in memory for a remote browser.
- Remote limit (server mode, a LAN browser) 20 MB → 500 MB, as a Files upload.
- A loopback browser in local mode has no cap any more, only the data dir's
  free space less a 1 GB reserve; ETOOBIG's max_bytes is what still fits, so
  the UI's "at most N MB" stays true.
- drop-image keeps its 20 MB in-memory limit, now checked on its own.
- fake-sudo gains a spawn for run-as.__setSpawnImpl.
This commit is contained in:
2026-10-26 16:36:29 +07:00
parent a0cd808456
commit dce32b96b7
6 changed files with 98 additions and 73 deletions
+1 -1
View File
@@ -2963,7 +2963,7 @@ POST /api/run/drop-image Raw image body (png/jpeg/gif/webp, ≤20 M
POST /api/run/clipboard-files → {files:[{path,size,dir}]}: the files a file manager's Copy put on THIS machine's clipboard (Windows GetFileDropList, macOS NSPasteboard, Linux text/uri-list via wl-paste/xclip), each that exists here (`size` null for a folder) — local mode + loopback only, else 403 `EREMOTE`; a pasted file whose name and size match is attached by that path, no upload and no size limit POST /api/run/clipboard-files → {files:[{path,size,dir}]}: the files a file manager's Copy put on THIS machine's clipboard (Windows GetFileDropList, macOS NSPasteboard, Linux text/uri-list via wl-paste/xclip), each that exists here (`size` null for a folder) — local mode + loopback only, else 403 `EREMOTE`; a pasted file whose name and size match is attached by that path, no upload and no size limit
POST /api/run/selected-files → {files:[{path,size,dir}]}: the same for a DROP — the items selected in THIS machine's file manager, which is what a drag from it carries (Windows Explorer windows and the desktop via Shell.Application, macOS Finder's selection — macOS asks once to let AgentHub control Finder; Linux file managers publish none, so a Linux drop uses the browser's own `file://` list or uploads). Nothing is searched for on disk POST /api/run/selected-files → {files:[{path,size,dir}]}: the same for a DROP — the items selected in THIS machine's file manager, which is what a drag from it carries (Windows Explorer windows and the desktop via Shell.Application, macOS Finder's selection — macOS asks once to let AgentHub control Finder; Linux file managers publish none, so a Linux drop uses the browser's own `file://` list or uploads). Nothing is searched for on disk
GET /api/run/native-paste → {native, key}: whether a `claude` pane can read this machine's clipboard itself (see drop-file below), and `key` — `ctrl+v`, or `alt+v` on Windows — the chord Claude Code binds its image paste to there, which the terminal sends GET /api/run/native-paste → {native, key}: whether a `claude` pane can read this machine's clipboard itself (see drop-file below), and `key` — `ctrl+v`, or `alt+v` on Windows — the chord Claude Code binds its image paste to there, which the terminal sends
POST /api/run/drop-file?name=&laneId= Any raw file body (≤20 MB; ≤2 GB from a loopback browser outside server mode, streamed to disk — Chrome hands a dropped file no `file://` path) → {path}: same dir (S3: the handoff dir of the account workspace `laneId`'s panes run as, else the requester's), 0600, named <random>-<name reduced to a [A-Za-z0-9._-] basename>; what the terminal (drop or paste, Ctrl+V included), the prompt box and its paste use for every dropped file or image — except a browser on a loopback host dropping or pasting a file manager's files, whose `file://` paths are attached as they are, uploading nothing, once `POST /api/run/local-paths` finds the same files here, and an image PASTED into a `claude` pane by a loopback browser of a local-mode dashboard, which is sent as ^V (Alt+V on Windows, Claude Code's binding there) so `claude` reads the desktop clipboard itself (Settings → Appearance → **Paste images natively**, on by default, per browser — off for an SSH tunnel) and only while `GET /api/run/native-paste` answers `{native:true}`: local mode, a loopback request, and on Linux a reachable display plus `wl-paste` (Wayland) or `xclip` (X11) — without one the `^V` pasted nothing, so the image is uploaded instead. Both drop routes answer 413 `ETOOBIG` (with `max_bytes`, the limit the UI shows) to a `Content-Length` over their limit before reading the body, and close the connection; of several files only the refused ones are lost POST /api/run/drop-file?name=&laneId= Any raw file body, streamed to disk as it arrives (S3: through `write-as` as the member) — ≤500 MB, like a Files upload; from a loopback browser outside server mode no cap, only the data dir's free space less a 1 GB reserve (Chrome hands a dropped file no `file://` path) → {path}: same dir (S3: the handoff dir of the account workspace `laneId`'s panes run as, else the requester's), 0600, named <random>-<name reduced to a [A-Za-z0-9._-] basename>; what the terminal (drop or paste, Ctrl+V included), the prompt box and its paste use for every dropped file or image — except a browser on a loopback host dropping or pasting a file manager's files, whose `file://` paths are attached as they are, uploading nothing, once `POST /api/run/local-paths` finds the same files here, and an image PASTED into a `claude` pane by a loopback browser of a local-mode dashboard, which is sent as ^V (Alt+V on Windows, Claude Code's binding there) so `claude` reads the desktop clipboard itself (Settings → Appearance → **Paste images natively**, on by default, per browser — off for an SSH tunnel) and only while `GET /api/run/native-paste` answers `{native:true}`: local mode, a loopback request, and on Linux a reachable display plus `wl-paste` (Wayland) or `xclip` (X11) — without one the `^V` pasted nothing, so the image is uploaded instead. Both drop routes answer 413 `ETOOBIG` (with `max_bytes`, the limit the UI shows) to a `Content-Length` over their limit before reading the body (a loopback drop: when it would not fit on the disk), and close the connection; of several files only the refused ones are lost
POST /api/run/local-paths {paths:[<absolute>…]} (1–100) → {sizes:[bytes|null…]}: each path's size when it is a regular file on this machine, else null. The composer and the terminal use a dropped file's own `file://` path only when every size matches the dropped bytes' — "localhost" in the address bar is no proof (an SSH tunnel's browser is on another machine). 403 EREMOTE in server mode (a pane runs as an account that cannot read the host's files) and to anything but a loopback socket without X-Forwarded-For/Forwarded; 400 EBADFIELD otherwise POST /api/run/local-paths {paths:[<absolute>…]} (1–100) → {sizes:[bytes|null…]}: each path's size when it is a regular file on this machine, else null. The composer and the terminal use a dropped file's own `file://` path only when every size matches the dropped bytes' — "localhost" in the address bar is no proof (an SSH tunnel's browser is on another machine). 403 EREMOTE in server mode (a pane runs as an account that cannot read the host's files) and to anything but a loopback socket without X-Forwarded-For/Forwarded; 400 EBADFIELD otherwise
GET /api/run/drop/:name A dropped png/jpg/gif/webp, or txt/md/csv/tsv/log/json/xml/yaml as text/plain, served back (nosniff, sandbox CSP) from the requester's own drop dir (S3: their member account's, owner-checked — for an admin, then the host's, where their host-folder workspaces drop); only a name drop-image/drop-file made → 404 otherwise GET /api/run/drop/:name A dropped png/jpg/gif/webp, or txt/md/csv/tsv/log/json/xml/yaml as text/plain, served back (nosniff, sandbox CSP) from the requester's own drop dir (S3: their member account's, owner-checked — for an admin, then the host's, where their host-folder workspaces drop); only a name drop-image/drop-file made → 404 otherwise
GET /api/run/:id/permission-mode A live claude pane's mode, read off its mode line → {mode} (null while a menu hides it) GET /api/run/:id/permission-mode A live claude pane's mode, read off its mode line → {mode} (null while a menu hides it)
+1 -1
View File
@@ -65,7 +65,7 @@ The proxy has to:
- forward WebSocket upgrades (`/ws` for live updates, `/ws-pty/*` for terminals); - forward WebSocket upgrades (`/ws` for live updates, `/ws-pty/*` for terminals);
- set `X-Forwarded-For` and `X-Forwarded-Proto`; - set `X-Forwarded-For` and `X-Forwarded-Proto`;
- accept request bodies as large as AgentHub's own limits (Files uploads up to - accept request bodies as large as AgentHub's own limits (Files uploads up to
500 MB, prompt drops 20 MB). nginx's default `client_max_body_size` is 1 MB, 500 MB, files dropped on a terminal or a prompt box 500 MB, dropped images 20 MB). nginx's default `client_max_body_size` is 1 MB,
and its own 413 is an HTML page the UI can only report as a failed upload. and its own 413 is an HTML page the UI can only report as a failed upload.
nginx: nginx:
+12 -10
View File
@@ -1,32 +1,34 @@
/** /**
* @file fake-sudo.js * @file fake-sudo.js
* @description Test helper for the S3 suites: an `execImpl` for * @description Test helper for the S3 suites: an `execImpl` for
* `run-as.__setExecImpl` that plays `sudo -n -u <login> [VAR=v…] <bin> …` * `run-as.__setExecImpl` (and a `spawn` for `__setSpawnImpl`) that plays
* without root — it records the argv (so a test can assert nothing secret * `sudo -n -u <login> [VAR=v…] <bin> …` without root — it records the argv (so a test can assert nothing secret
* crossed sudo) and runs the real `scripts/server/write-as.sh` for * crossed sudo) and runs the real `scripts/server/write-as.sh` for
* `/opt/agenthub/libexec/write-as`, as this account. * `/opt/agenthub/libexec/write-as`, as this account.
* @author Nguyễn Ngọc Trí Vĩ <vinnt@smartgift.vn> * @author Nguyễn Ngọc Trí Vĩ <vinnt@smartgift.vn>
*/ */
const path = require("node:path"); const path = require("node:path");
const { execFileSync } = require("node:child_process"); const { execFileSync, spawn: spawnChild } = require("node:child_process");
const WRITE_AS = path.join(__dirname, "..", "..", "..", "scripts", "server", "write-as.sh"); const WRITE_AS = path.join(__dirname, "..", "..", "..", "scripts", "server", "write-as.sh");
/** `{ exec, calls }`: pass `exec` to `runAs.__setExecImpl`; `calls` = every sudo argv. */ /** `{ exec, spawn, calls }`: pass `exec` to `runAs.__setExecImpl` and `spawn`
* to `runAs.__setSpawnImpl`; `calls` = every sudo argv. */
function fakeSudo() { function fakeSudo() {
const calls = []; const calls = [];
const exec = (cmd, argv, opts = {}) => { // The command sudo would run, as [file, args].
const unwrap = (cmd, argv) => {
calls.push([cmd, ...argv]); calls.push([cmd, ...argv]);
if (cmd !== "sudo") return execFileSync(cmd, argv, opts); if (cmd !== "sudo") return [cmd, argv];
let i = argv.indexOf("-u") + 2; // past the login let i = argv.indexOf("-u") + 2; // past the login
while (/^[A-Z_][A-Z0-9_]*=/.test(argv[i])) i++; while (/^[A-Z_][A-Z0-9_]*=/.test(argv[i])) i++;
const [bin, ...rest] = argv.slice(i); const [bin, ...rest] = argv.slice(i);
if (bin === "/opt/agenthub/libexec/write-as") return bin === "/opt/agenthub/libexec/write-as" ? ["sh", [WRITE_AS, ...rest]] : [bin, rest];
return execFileSync("sh", [WRITE_AS, ...rest], opts);
return execFileSync(bin, rest, opts);
}; };
return { exec, calls }; const exec = (cmd, argv, opts = {}) => execFileSync(...unwrap(cmd, argv), opts);
const spawn = (cmd, argv, opts = {}) => spawnChild(...unwrap(cmd, argv), opts);
return { exec, spawn, calls };
} }
module.exports = { fakeSudo }; module.exports = { fakeSudo };
+6 -4
View File
@@ -334,12 +334,12 @@ describe("POST /api/run/drop-image", () => {
}); });
it("refuses a file declared over the limit before its body arrives", async () => { it("refuses a file declared over the limit before its body arrives", async () => {
// 4 GB announced, 1 KB sent: the answer must come without the rest. // A petabyte announced, 1 KB sent: the answer must come without the rest.
const { status, body } = await new Promise((resolve, reject) => { const { status, body } = await new Promise((resolve, reject) => {
const url = new URL("/api/run/drop-file?name=big.mp4", BASE); const url = new URL("/api/run/drop-file?name=big.mp4", BASE);
const req = http.request(url, { const req = http.request(url, {
method: "POST", method: "POST",
headers: { "Content-Type": "video/mp4", "Content-Length": 4 * 1024 ** 3 }, headers: { "Content-Type": "video/mp4", "Content-Length": 1024 ** 5 },
}); });
req.on("response", (res) => { req.on("response", (res) => {
let text = ""; let text = "";
@@ -354,8 +354,10 @@ describe("POST /api/run/drop-image", () => {
}); });
assert.equal(status, 413); assert.equal(status, 413);
assert.equal(body.error.code, "ETOOBIG"); assert.equal(body.error.code, "ETOOBIG");
// The UI shows the server's limit: a loopback browser's, here. // A loopback browser has no cap, only the disk's free space (less a
assert.equal(body.error.max_bytes, 2 * 1024 ** 3); // reserve) — and that is the limit the UI shows.
const st = fs.statfsSync(process.env.DASHBOARD_DATA_DIR);
assert.ok(body.error.max_bytes <= st.bavail * st.bsize, String(body.error.max_bytes));
}); });
it("streams a loopback drop over the remote limit to disk", async () => { it("streams a loopback drop over the remote limit to disk", async () => {
+5 -1
View File
@@ -97,7 +97,9 @@ before(async () => {
); );
linux.__resetConfCache(); linux.__resetConfCache();
linux.__setRunImpl(() => "ok"); linux.__setRunImpl(() => "ok");
runAs.__setExecImpl(fakeSudo().exec); const sudo = fakeSudo();
runAs.__setExecImpl(sudo.exec);
runAs.__setSpawnImpl(sudo.spawn);
tmux.__setExecImpl(fakeTmux); tmux.__setExecImpl(fakeTmux);
server = await startServer(createApp(), 0); server = await startServer(createApp(), 0);
BASE = `http://127.0.0.1:${server.address().port}`; BASE = `http://127.0.0.1:${server.address().port}`;
@@ -708,6 +710,8 @@ describe("a dropped file", () => {
byAdmin.body.path.startsWith(path.join(MIA, ".agenthub", "handoff", "drops") + path.sep), byAdmin.body.path.startsWith(path.join(MIA, ".agenthub", "handoff", "drops") + path.sep),
byAdmin.body.path byAdmin.body.path
); );
// Streamed to the member's file through write-as, not held whole.
assert.equal(fs.readFileSync(byAdmin.body.path, "utf8"), "hello");
// A host workspace's pane runs as the host: the host's own 0700 drops dir. // A host workspace's pane runs as the host: the host's own 0700 drops dir.
const hostLane = addLane(path.join(TMP, "hostdir", "d"), "boss"); const hostLane = addLane(path.join(TMP, "hostdir", "d"), "boss");
const host = await drop("boss", hostLane); const host = await drop("boss", hostLane);
+73 -56
View File
@@ -25,6 +25,8 @@ const express = require("express");
const { Router } = express; const { Router } = express;
const fs = require("node:fs"); const fs = require("node:fs");
const crypto = require("node:crypto"); const crypto = require("node:crypto");
const { Transform } = require("node:stream");
const { pipeline } = require("node:stream/promises");
const path = require("node:path"); const path = require("node:path");
const runs = require("../lib/pty-run"); const runs = require("../lib/pty-run");
const tmux = require("../lib/tmux"); const tmux = require("../lib/tmux");
@@ -114,21 +116,35 @@ const DROP_TYPES = {
"image/webp": "webp", "image/webp": "webp",
}; };
const DROP_MAX_BYTES = 20 * 1024 * 1024; const DROP_MAX_BYTES = 20 * 1024 * 1024;
// A browser on this machine (loopback, never server mode) crosses no network: // drop-file streams every body to disk, never holding it in memory: from
// its drop is streamed to disk, so a video or a dump is no reason to refuse. // another machine (server mode, a LAN browser) up to what a Files upload may
// Chrome hands a dropped file no `file://` path; the client first asks for // carry, so a video or a dump dropped on a pane is no reason to refuse.
// the file manager's own (`/selected-files`, `/clipboard-files`) and only const DROP_REMOTE_MAX_BYTES = 500 * 1024 * 1024;
// uploads what that cannot place. // A browser on this machine (loopback, never server mode) crosses no network,
const DROP_LOCAL_MAX_BYTES = 2 * 1024 ** 3; // so its drop has no size cap — only what the disk can take: the copy must
// leave DROP_LOCAL_RESERVE_BYTES free. Chrome hands a dropped file no
// `file://` path; the client first asks for the file manager's own
// (`/selected-files`, `/clipboard-files`) and only uploads what that cannot
// place.
const DROP_LOCAL_RESERVE_BYTES = 1024 ** 3;
const DROP_TTL_MS = 24 * 60 * 60_000; const DROP_TTL_MS = 24 * 60 * 60_000;
const localDrop = (req) => !require("../lib/auth").isServerMode() && isLocalRequest(req); const localDrop = (req) => !require("../lib/auth").isServerMode() && isLocalRequest(req);
const dropLimit = (req) => (localDrop(req) ? DROP_LOCAL_MAX_BYTES : DROP_MAX_BYTES); /** The free space a local drop may fill (a local drop lands in the data dir). */
function localRoom() {
try {
const st = fs.statfsSync(require("../lib/claude-home").getDataDir());
return Math.max(0, st.bavail * st.bsize - DROP_LOCAL_RESERVE_BYTES);
} catch {
return Infinity; // unknown: a full disk still fails the write, and leaves no file
}
}
const dropLimit = (req) => (localDrop(req) ? localRoom() : DROP_REMOTE_MAX_BYTES);
function tooBig(res, max) { function tooBig(res, max) {
res.set("Connection", "close"); res.set("Connection", "close");
res.status(413).json({ res.status(413).json({
error: { error: {
code: "ETOOBIG", code: "ETOOBIG",
message: `a dropped file may be at most ${max / 1024 / 1024} MB`, message: `a dropped file may be at most ${Math.floor(max / 1024 / 1024)} MB`,
max_bytes: max, max_bytes: max,
}, },
}); });
@@ -136,11 +152,11 @@ function tooBig(res, max) {
/** A body declared over the limit is refused before a byte of it is read, and /** A body declared over the limit is refused before a byte of it is read, and
* the connection is closed: express.raw would read (and discard) all of it * the connection is closed: express.raw would read (and discard) all of it
* first, so a 4 GB drop crossed the network in full just to be told 413. */ * first, so a 4 GB drop crossed the network in full just to be told 413. */
function refuseOversize(req, res, next) { const refuseOversize = (limit) => (req, res, next) => {
const max = dropLimit(req); const max = limit(req);
if (!(Number(req.headers["content-length"]) > max)) return next(); if (!(Number(req.headers["content-length"]) > max)) return next();
tooBig(res, max); tooBig(res, max);
} };
/** Where a drop goes: the handoff dir of the account the PANE runs as — the /** Where a drop goes: the handoff dir of the account the PANE runs as — the
* workspace `?laneId=` names (scopeGuard already refused one this user may * workspace `?laneId=` names (scopeGuard already refused one this user may
* not see), else the requester's own — written AS that member under S3, so * not see), else the requester's own — written AS that member under S3, so
@@ -195,7 +211,8 @@ function saveDrop(req, res, name, body) {
router.post( router.post(
"/drop-image", "/drop-image",
refuseOversize, // An image is held in memory: 20 MB, wherever the browser is.
refuseOversize(() => DROP_MAX_BYTES),
express.raw({ type: Object.keys(DROP_TYPES), limit: DROP_MAX_BYTES }), express.raw({ type: Object.keys(DROP_TYPES), limit: DROP_MAX_BYTES }),
(req, res) => { (req, res) => {
const ext = const ext =
@@ -222,63 +239,63 @@ const dropName = (req) =>
.replace(/^\.+/, "") .replace(/^\.+/, "")
.slice(-100); .slice(-100);
/** A local browser's drop is piped to its file as it arrives, never held in /** A drop is piped to its file as it arrives, never held in memory whole —
* memory whole; a body that runs past the limit (or is empty) leaves no file. */ * a member's (S3) through `write-as` as them, its bytes on stdin; a body that
function streamLocalDrop(req, res, next) { * runs past the limit (or is empty, or is cut off) leaves no file. */
if (!localDrop(req)) return next(); async function streamDrop(req, res) {
const safe = dropName(req); const safe = dropName(req);
if (!safe) return res.status(400).json({ error: "?name= and a non-empty file body required" }); if (!safe) return res.status(400).json({ error: "?name= and a non-empty file body required" });
const runAs = require("../lib/run-as");
const max = dropLimit(req);
let file; let file;
let out; let member;
try { try {
const { dir, member } = prepareDropDir(req); const dropped = prepareDropDir(req);
if (member) return next(); // a member's file is written as them, by saveDrop member = dropped.member;
file = path.join(dir, `${crypto.randomBytes(8).toString("hex")}-${safe}`); file = path.join(dropped.dir, `${crypto.randomBytes(8).toString("hex")}-${safe}`);
out = fs.createWriteStream(file, { mode: 0o600, flags: "wx" });
} catch (err) { } catch (err) {
return res.status(500).json({ error: err.message }); return res.status(500).json({ error: err.message });
} }
let bytes = 0; let bytes = 0;
let failed = null; const count = new Transform({
const fail = (status, send) => { transform(chunk, _enc, cb) {
if (failed) return; bytes += chunk.length;
failed = status; if (bytes > max) return cb(Object.assign(new Error("too big"), { code: "ETOOBIG" }));
req.unpipe(out); cb(null, chunk);
out.destroy(); },
fs.rm(file, { force: true }, () => {});
if (!res.headersSent) send();
};
req.on("data", (c) => {
bytes += c.length;
if (bytes > DROP_LOCAL_MAX_BYTES) fail(413, () => tooBig(res, DROP_LOCAL_MAX_BYTES));
}); });
req.on("aborted", () => fail(400, () => {})); try {
out.on("error", (err) => fail(500, () => res.status(500).json({ error: err.message }))); if (member) {
out.on("finish", () => { await Promise.all([
if (failed) return; pipeline(req, count).catch((e) => {
if (bytes === 0) { count.destroy(e);
return fail(400, () => throw e;
res.status(400).json({ error: "?name= and a non-empty file body required" }) }),
); runAs.writeAsAsync(member, file, ["replace"], { input: count }),
]);
} else {
await pipeline(req, count, fs.createWriteStream(file, { mode: 0o600, flags: "wx" }));
} }
res.json({ path: file }); if (bytes === 0) throw Object.assign(new Error("empty"), { code: "EEMPTY" });
}); } catch (err) {
req.pipe(out); try {
} if (member) runAs.removeAs(member, file);
else fs.rmSync(file, { force: true });
router.post( } catch {
"/drop-file", /* never written */
refuseOversize, }
streamLocalDrop, // Past the limit mid-body, pipeline has already closed the socket.
express.raw({ type: () => true, limit: DROP_MAX_BYTES }), if (res.headersSent) return;
(req, res) => { if (err.code === "ETOOBIG") return tooBig(res, max);
const safe = dropName(req); if (err.code === "EEMPTY") {
if (!safe || !Buffer.isBuffer(req.body) || req.body.length === 0) {
return res.status(400).json({ error: "?name= and a non-empty file body required" }); return res.status(400).json({ error: "?name= and a non-empty file body required" });
} }
saveDrop(req, res, `${crypto.randomBytes(8).toString("hex")}-${safe}`, req.body); return res.status(500).json({ error: err.message });
} }
); res.json({ path: file });
}
router.post("/drop-file", refuseOversize(dropLimit), streamDrop);
// A browser on this machine is handed a file manager's files as `file://` // A browser on this machine is handed a file manager's files as `file://`
// paths beside their bytes; the composer attaches such a path instead of // paths beside their bytes; the composer attaches such a path instead of