feat(files): uploads survive a dashboard restart
An upload through the File Explorer is now sent in 8 MB parts that the
server stages on the host (<data dir>/upload-staging/<user>/<id>/,
server/lib/upload-staging.js) and writes only once every byte is there,
through the same upload/uploadBatch code and guards (?staged=<id>). A
restart, an update or a crash pauses it: the browser waits up to 3 min,
asks how much arrived (GET /fs/upload-part) and sends the rest; the dock
says the dashboard is restarting meanwhile. A commit's answer is stored
and replayed, never written twice; a refused one (EEXISTS) keeps the
bytes for a retry with another conflict policy.
A file uploaded into a member's folder is now written to a hidden temp
name and moved onto its name only when whole: write-as cannot tell a
dropped connection from the end of a file, and `replace` left a truncated
file under the real name when the server died mid-upload.
New: GET|POST|DELETE /api/{lanes/:id,homes/:who}/fs/upload-part; 409
EOFFSET/EINCOMPLETE/EBUSY, 429 ETOOMANY. Verified on a sandbox server:
SIGTERM mid-upload of a 60 MB file in a real browser, resumed and
sha256-identical.
This commit is contained in:
+33
@@ -290,6 +290,9 @@ POST /api/lanes/:id/fs/trash/purge { id | "all" }
|
||||
POST /api/lanes/:id/fs/upload?path=&conflict= raw body, streamed → 201 { path } | 200 { skipped: true }
|
||||
POST /api/lanes/:id/fs/upload-check { dir?, paths } → { taken: [paths], batch }
|
||||
POST /api/lanes/:id/fs/upload-batch?dir=&conflict= length + JSON list + bytes → 201 { saved, skipped, paths }
|
||||
POST /api/lanes/:id/fs/upload-part?upload=&offset=&total= raw bytes (≤ 16 MB) → { received, total }
|
||||
GET /api/lanes/:id/fs/upload-part?upload= → { received, total, done }
|
||||
DELETE /api/lanes/:id/fs/upload-part?upload= → { dropped }
|
||||
GET /api/lanes/:id/fs/zip?p=<path>&p=… application/zip, streamed
|
||||
```
|
||||
|
||||
@@ -328,6 +331,33 @@ install-system` is re-run, and the page then falls back to one request per
|
||||
file. Measured on loopback (host folder): 1,000 files of 2 KB took 4.3 s one
|
||||
by one, 0.5 s as one batch.
|
||||
|
||||
The Files page sends every upload — a file or a batch — **resumably**
|
||||
(`server/lib/upload-staging.js`), so a dashboard restart (an update, a manual
|
||||
`systemctl restart`, a crash) only pauses it. The browser names the upload
|
||||
with a random id (`upload`, 16–64 of `[A-Za-z0-9_-]`) and sends its bytes in
|
||||
8 MB parts to `upload-part`, which appends each at `offset` to
|
||||
`<data dir>/upload-staging/<user>/<id>/` on the HOST (never in a member's
|
||||
home) — `409 EOFFSET` with `error.received` when a part does not start where
|
||||
the staged bytes end, `409 EBUSY` while another part of the same id is still
|
||||
arriving, `413 ETOOBIG` past `total` (512 MB at most), `429 ETOOMANY` beyond 8
|
||||
unfinished uploads per user. Then `upload?…&staged=<id>` or
|
||||
`upload-batch?…&staged=<id>` writes the staged bytes through the SAME code
|
||||
and guards as a body would (`409 EINCOMPLETE` before every byte is there); a
|
||||
commit already made answers its stored answer again (a restart may have
|
||||
swallowed the first), and a refused one (`EEXISTS`) keeps the bytes for a
|
||||
retry with another `conflict`. On a network error or a bare 5xx (a proxy
|
||||
answering for a restarting dashboard) the browser waits up to 3 minutes,
|
||||
asks `GET upload-part` how much arrived and sends the rest; the dock shows
|
||||
"The dashboard is restarting…" meanwhile. Uploads are per user — another
|
||||
user's id is never found — and one untouched for a day is swept. A reload
|
||||
of the page still loses an upload: the bytes are the browser's.
|
||||
|
||||
A file uploaded into a member's folder is written to a hidden
|
||||
`.<name>.agenthub-upload-<hex>` by `write-as` and moved onto its name only
|
||||
once every byte arrived, so a dropped connection or a dying server never
|
||||
leaves a truncated file under the real name (a `write-as` older than `move`
|
||||
keeps the old `replace`).
|
||||
|
||||
#### File Explorer on a home
|
||||
|
||||
```
|
||||
@@ -353,6 +383,9 @@ POST /api/homes/:who/fs/trash/purge
|
||||
POST /api/homes/:who/fs/upload?path=&conflict=
|
||||
POST /api/homes/:who/fs/upload-check
|
||||
POST /api/homes/:who/fs/upload-batch?dir=&conflict=
|
||||
POST /api/homes/:who/fs/upload-part?upload=&offset=&total=
|
||||
GET /api/homes/:who/fs/upload-part?upload=
|
||||
DELETE /api/homes/:who/fs/upload-part?upload=
|
||||
```
|
||||
|
||||
The same File Explorer on a user's HOME (`server/lib/homes.js`): under S3 their
|
||||
|
||||
Reference in New Issue
Block a user