Upload session state is not persisted; server restart causes 404 on chunk PUT and orphaned files on disk

[dobby-coder]

Filed as a follow-up to postguard-website#117 — that issue is about the website surfacing the failure better; this one is about the underlying persistence gap on the cryptify side.

What is wrong

Store keeps all upload session state in memory (HashMap<String, Arc<Mutex<FileState>>> in src/store.rs:37). There is no on-disk persistence and no startup hydration. On any cryptify process restart — deploy, panic, OOM, container reschedule — every in-progress upload session is lost.

Two visible symptoms follow:

1. Chunk PUTs return 404 forever after restart. upload_chunk looks up the uuid via store.get(uuid) and returns Ok(None) (HTTP 404) when the session is missing (src/main.rs:284-287). The client has no way to recover; from the client's point of view this is indistinguishable from a typo'd uuid.

2. On-disk files leak. upload_init creates data_dir/<uuid> (src/main.rs:101) and stores a 14-day expiry on the in-memory FileState. After a restart, the on-disk file remains but no FileState references it any more, and the purge task in src/store.rs:152-173 only walks the in-memory expirations map. There is no separate disk reaper, so the file sits there until manually cleaned.

Source: src/store.rs · src/main.rs upload_chunk · src/main.rs upload_init

Why it matters

• For a single-replica deployment (current postguard-ops setup), every cryptify deploy or crash silently kills any user mid-upload. Ruben's report on postguard-website#117 is consistent with a deploy-time eviction.
• It also blocks resume support (suggested fix Perform encryption in a Web/Service Worker #4 on website#117) and horizontal scaling — a multi-replica deployment would need shared state regardless.

Possible designs

In rough order of effort:

1. Sidecar JSON per upload. On init, write data_dir/<uuid>.meta.json with the FileState fields. On chunk PUT, update uploaded + cryptify_token in that file (already syncing the disk file anyway). On startup, scan data_dir/ and rebuild the HashMap. Cheapest change, single-replica only.

2. SQLite file in data_dir. Same shape as Add code from rust backend #1 but transactional and easier to query for the purge task. Still single-replica.

3. PostgreSQL. Reuses the postguard-business stack. Required if cryptify ever runs more than one replica.

The 14-day expires field already exists on FileState, so any of these can drive the purge of both the metadata and the disk file from a single source of truth.

Out of scope here

• Client-side retry / better error message — covered by postguard-website#117.
• Resume support API — depends on this issue being fixed first.

Suggested next step

Confirm the diagnosis in production: pick a 404 chunk PUT from logs and check whether its timestamp lines up with a cryptify restart, and whether data_dir/<that-uuid> still exists on the host. If both are true, scope the fix to design #1 above and I can have a draft PR ready.

[dobby-coder]

Self-correction (cross-linking from the postguard-website#117 thread): I originally framed this as restart-only, but the in-memory eviction is also independently broken on a shorter window.

Store::create schedules a hard 15-minute removal at src/store.rs:80 (Duration::from_secs(60 * 15)), regardless of activity. So even on a single, never-restarted process, an upload that takes longer than 15 minutes from init to its next chunk PUT will 404.

That is a much cheaper fix than this persistence gap (just extend the removal_instant on each chunk PUT, or drop the hard timer entirely and rely on the 14-day FileState.expires field) and worth doing independently. Happy to split it into a sibling issue if a maintainer prefers — leaving it noted here for now since the underlying object (FileState lifecycle in the in-memory Store) is the same.

[rubenhensen]

/dobby

[dobby-coder]

Dobby sees the request, master rubenhensen! Dobby will start with the cheaper, well-scoped fix called out above: the 15-minute hard eviction timer in Store::create. The bigger persistence work (design #1, sidecar JSON) is the right next step and Dobby will scope it as a separate PR or sibling issue once the timer fix is up — keeping changes small and reviewable. On it now.

[dobby-coder]

Draft PR up: #132 — extends the in-memory eviction deadline on each successful chunk PUT (60-min idle window) and adds a reverse index so touch/remove are O(1). Tests: 3 new unit tests covering touch + remove cleanup; cargo test and cargo clippy both clean. Out of scope here: the broader on-disk persistence work (server-restart case). Will scope that as a separate PR.

[dobby-coder]

/dobby please open a PR for this issue.

[rubenhensen]

Updating direction on this: we want to land persistence and multi-replica together rather than as two separate steps, so #116 should be scoped to use Postgres as the shared store and merged with the goal in #35 (Support centralized sessions).

• Skip the sidecar-JSON and SQLite designs from the original write-up. They only solve the single-replica case and we'd throw them away the first time we want to run two cryptify pods.
• Postgres is the chosen backend — reuses the postguard-business stack and gives us the same source of truth across replicas. data_dir chunk files still live on disk, so the deployment will also need either a shared volume (ReadWriteMany PVC) or moving file storage to object storage (S3/MinIO). That's the second decision this issue should resolve.
• The 14-day expires field becomes a row in Postgres; the purge task becomes a DELETE … RETURNING uuid driving on-disk cleanup. Same source-of-truth invariant as before.

Effectively this issue and #35 are now one piece of work. Suggest closing #35 in favor of this one, or vice versa, once we pick which to keep open. The resume protocol from #136 (#148 + #145) is already shipped and unaffected by the choice of backend.