Skip to content

P0: atomic manifest commit helper — clone-mutate-save-publish#27

Open
pbudzik wants to merge 1 commit into
mainfrom
fix/manifest-atomic-commit
Open

P0: atomic manifest commit helper — clone-mutate-save-publish#27
pbudzik wants to merge 1 commit into
mainfrom
fix/manifest-atomic-commit

Conversation

@pbudzik
Copy link
Copy Markdown
Owner

@pbudzik pbudzik commented May 17, 2026

Summary

Fixes the P0 manifest-atomicity finding from external review (the "mutate-then-save" anti-pattern across flusher, rollup, compaction, and period close/reopen).

Before: every mutation looked like

let mut manifest = state.manifest.write().await;
manifest.raw_segments.push(meta);
manifest.last_sealed_wal_id = sealed_wal_id;
manifest.save(&db_root)  // ← if this fails, in-memory is now ahead of disk

If `save` failed, the in-memory state held the mutation but disk did not. Queries would see writes that weren't durable. Worse, every cleanup-on-save-failure path unlinks the segment files it just wrote — leaving the in-memory manifest pointing at deleted files until process restart.

After: `AppStateInner::commit_manifest(|m| …)` clones the manifest, mutates the clone, saves the clone, and only publishes it in-memory after the save succeeds. `commit_manifest_if` is the variant that lets the closure decide under the write lock whether to commit (used by close-period / reopen-period for race-safe re-checks).

Changes

  • New `commit_manifest` / `commit_manifest_if` methods on `AppStateInner` in `src/runtime/state.rs`.
  • Refactored every mutate-then-save site:
    • flusher: `raw_segments` + `last_sealed_wal_id`
    • rollup tick: `rollup_segments` + watermark advance
    • rollup rebuild: drop overlapping rollups + rewind watermark
    • compaction commit: `raw_segments` swap + replacement record
    • compaction sweep: drop finalized replacements
    • close-period: append `ClosedPeriod`
    • reopen-period: remove `ClosedPeriod`
  • The only mutate-then-save sites left are the two startup paths in `main.rs` and `Manifest::migrate_legacy` — both run before the manifest is shared, so there's no in-memory state to keep coherent.

Test plan

  • New `tests/p0_manifest_atomic.rs::commit_manifest_leaves_state_unchanged_on_save_failure` sabotages writes by replacing the `manifest/` directory with a regular file, asserts `commit_manifest` returns Err, and asserts both `current_generation` and `bucket_count` are unchanged in memory.
  • `commit_manifest_if_skips_save_when_closure_returns_none` proves the early-exit path doesn't touch disk or in-memory state.
  • `commit_manifest_publishes_only_after_save_succeeds` covers the happy path.
  • Full `cargo test` passes — 144 tests across the suite.

Notes

This is the second of several P0 fixes from the external review (#26 fixed the flusher shutdown deadlock). Next up:

  1. Watermark race-safe commit (validate `last_sealed_wal_id` and raw-segment generation are unchanged between snapshot and commit)
  2. `Open → Closing → Closed` period lifecycle so close-period can't miss racing ingests
  3. Dedupe TTL on `ingested_at_ms`, not event `timestamp_ms`

🤖 Generated with Claude Code

Before: every manifest mutation took the write lock, mutated the
manifest in place, then called `save`. If `save` failed, the
in-memory state held the mutation but disk did not. Queries saw
writes that weren't durable; cleanup-on-save-failure paths (which
unlink the segment files they just wrote) left the in-memory
manifest pointing at files that no longer existed.

Fix: `AppStateInner::commit_manifest` clones the manifest, runs the
caller's mutation on the clone, saves the clone to disk, and only
publishes the clone in-memory after the save succeeds. On failure
both disk and memory are unchanged. `commit_manifest_if` is the
variant that lets the closure decide under the write lock whether
to commit (used by close-period / reopen-period for race-safe
re-checks).

Refactored every call site to use the helper:
- flusher: raw_segments + last_sealed_wal_id
- rollup tick: rollup_segments + watermark
- rollup rebuild: drop overlapping rollups + rewind watermark
- compaction commit: raw_segments swap + replacement record
- compaction sweep: drop finalized replacements
- close-period: append ClosedPeriod
- reopen-period: remove ClosedPeriod

The only mutate-then-save sites left in the tree are the two
startup paths in main.rs and manifest.rs::migrate_legacy — both run
before the manifest is shared, so the publish-in-memory step is
moot.

Regression test sabotages manifest writes by replacing the
manifest/ directory with a regular file, then verifies the
in-memory manifest is unchanged after a failed save. Two more
tests verify the happy path and the commit_manifest_if early-exit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant