Handle cases when dreyfus checkpoint is out-of-sync with the index#5931
Merged
Handle cases when dreyfus checkpoint is out-of-sync with the index#5931
Conversation
Currently, there are two places where the index purge seq is tracked: in the index and in the db local doc checkpoints. Purge sequence folding should never start below the value in the checkpoint document as that could raise an `invalid_start_purge_seq`. Normally both sequences should match, but if they don't try to be explicit about what should happen: * Index pseq > checkpoint pseq. Index somehow got ahead of the checkpoint. Use the checkpoint seq and re-process some purges through the index. This will do extra work but should be safe. * Index pseq < checkpoint pseq. Index somehow got behind the checkpoint and it looks like it could have skipped purges. For views we reset the index, and arguably that's the most correct solution. However, we never really had a reset facility for clouseau, so instead choose to emit an error log and let the user intervene manually but otherwise keep updating the index. When updating the purge sequence in clouseau, save an rpc call if we're not advancing clouseau's purge sequence. Clouseau as of recently already has a check to return `ok` right away if new purge_seq is somehow less or equal to the current one, but it's still nice not have to do an extra round-trip. It was a bit surprising to discover that we had a bunch of nice dreyfus eunit purge tests around but they never actually ran. The test functions there were not discoverably by EUnit. Switching them to be discoverable still wouldn't work as the test suite would need clouseau running during EUnit tests. Since we don't really have a framework for that, let's switch them to Elixir test and run them alongside other search tests.
rnewson
approved these changes
Mar 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, there are two places where the index purge seq is tracked: in the index and in the db local doc checkpoints. Purge sequence folding should never start below the value in the checkpoint document as that could raise an
invalid_start_purge_seq. Normally both sequences should match, but if they don't try to be explicit about what should happen:Index pseq > checkpoint pseq. Index somehow got ahead of the checkpoint. Use the checkpoint seq and re-process some purges through the index. This will do extra work but should be safe.
Index pseq < checkpoint pseq. Index somehow got behind the checkpoint and it looks like it could have skipped purges. For views we reset the index, and arguably that's the most correct solution. However, we never really had a reset facility for clouseau, so instead choose to emit an error log and let the user intervene manually but otherwise keep updating the index.
When updating the purge sequence in clouseau, save an rpc call if we're not advancing clouseau's purge sequence. Clouseau as of recently already has a check to return
okright away if new purge_seq is somehow less or equal to the current one, but it's still nice not have to do an extra round-trip.It was a bit surprising to discover that we had a bunch of nice dreyfus eunit purge tests around but they never actually ran. The test functions there were not discoverable by EUnit. Switching them to be discoverable still wouldn't work as the test suite would need clouseau running during EUnit tests. Since we don't really have a framework for that, let's switch them to Elixir test and run them alongside other search tests.