HBASE-29890 WAL tailing reader should resume partial cell reads instead of resetting compression #7741
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the WAL tailing reader hits EOF mid-cell during WAL compression, it currently returns EOF_AND_RESET_COMPRESSION, which forces the reader to re-read the entire WAL file from the beginning to rebuild dictionary state. This is an O(n) operation that becomes increasingly expensive as the WAL grows.
The root cause is that the CompressedKvDecoder eagerly adds entries to the compression dictionaries (ROW, FAMILY, QUALIFIER, and tag dictionaries) as it reads each field of a cell. If an IOException occurs partway through reading a cell, the dictionaries are left in a partially-updated state that no longer matches the actual stream position. The reader has no choice but to throw away the entire compression context and start over.
Proposed Fix is to defer dictionary additions until a cell is fully parsed:
With deferred additions, hitting EOF mid-cell leaves the dictionaries in the state they were after the last fully-read cell. This means the reader can return EOF_AND_RESET (a cheap seek to the saved position) instead of EOF_AND_RESET_COMPRESSION, and resume reading from where it left off once the file grows.