bugfix: Create tombstones when needed while doing REPLACE mode with range partitioning plus parallel indexing #18938
+152
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…as been filtered out
Contributor
Author
|
I thought the hashed partitions phase would face the same issue, but it does not. The test I added flexes the code path that would have been buggy if it did, but tombstones still end up being created. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if (getIngestionMode() == IngestionMode.REPLACE) { | ||
| // In REPLACE mode, publish segments (and tombstones, when called for) even when no new data was produced | ||
| publishSegments(toolbox, Collections.emptyMap()); | ||
| TaskStatus taskStatus = TaskStatus.success(getId(), msg); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note
| return taskStatus; | ||
| } else { | ||
| LOG.warn(msg); | ||
| return TaskStatus.success(getId(), msg); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note
Contributor
Author
|
I forgot that embedded tests coverage doesn't count for jacoco. will have to look at what existing test structure there is in the package |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Bug Description
Tombstones are not properly created when in
REPLACEindexing mode and ParallelIndexSupervisorTask range partitioning gets no reports for partition boundaries due to all rows being filtered out of the underlying source data. This results in existing segments remaining available that should have been overshadowed.Reproduction Steps
Compaction with range partitioning and a compaction transform filter that filters all rows out of the segments being compacted.
Expected behavior: tombstones created for the interval, and the old segments being overshadowed.
Actual behavior: Task completes with a warning that there were no valid rows for range partitioning. This leaves the old segments in place and will lead to an infinite loop of compaction
Release note
Key changed/added classes in this PR
ParallelIndexSupervisorTaskThis PR has: