Skip to content

fix(flat): handle multi-byte advance in consClose decoder#7683

Merged
zliu41 merged 4 commits intomasterfrom
yura/issue-7542-flat-enum-bug
Mar 23, 2026
Merged

fix(flat): handle multi-byte advance in consClose decoder#7683
zliu41 merged 4 commits intomasterfrom
yura/issue-7542-flat-enum-bug

Conversation

@Unisay
Copy link
Contributor

@Unisay Unisay commented Mar 20, 2026

Context

Fix consClose in the Flat decoder so it correctly advances the stream pointer when constructor tag bits + already-used bits span 2+ bytes. Without this, Generic-derived Flat instances of large enums (>256 constructors) corrupt the decoder state during deserialization, consuming all available memory.

Fixes #7542

What went wrong

The Generic-derived decoder uses a ConsState machine for constructor tags. After decoding, consClose advances the byte stream by the number of bits consumed. The problem: it only handled advancing by at most 1 byte. When constructor_bits + usedBits >= 16 (say, a 9-bit constructor tag after 7 bits already consumed in the current byte), usedBits overflowed to 8+, outside the valid range of [0, 7].

Once corrupted, dBool computes 128 >> usedBits as 0, so it always returns False without moving the stream pointer. The Filler decoder reads 0-bits as FillerBit and a 1-bit as FillerEnd, so it just keeps building an infinite chain of FillerBit constructors until memory runs out.

The fix swaps the hand-rolled if/else for divMod, the same pattern dropBits_ already uses 80 lines below in the same file.

Changes

Bug fix:

  • consClose in Decoder/Prim.hs: use divMod instead of single-byte advance logic, so any number of bytes is handled correctly

Reproduction test (two new cases in testLargeEnum in Spec.hs):

  • Control: 7 Bool fields + E258_256 (8-bit tag), 8+7=15 < 16, works fine
  • Bug trigger: 7 Bool fields + E258_258 (9-bit tag), 9+7=16, previously caused infinite loop; protected with 5-second timeout

@Unisay Unisay self-assigned this Mar 20, 2026
@Unisay Unisay requested a review from zliu41 March 20, 2026 15:14
Unisay added 2 commits March 20, 2026 17:05
consClose only advanced currPtr by at most 1 byte, so when
constructor_bits + usedBits >= 16 (e.g., 9-bit tag after 7 used
bits), usedBits overflowed to 8+. This corrupted the decoder state,
causing the Filler decoder to loop infinitely and consume all memory.

Use divMod to correctly advance by any number of bytes, matching the
pattern already used by dropBits_.

Fixes #7542
@Unisay Unisay force-pushed the yura/issue-7542-flat-enum-bug branch from 929cfd0 to 59688d5 Compare March 20, 2026 16:05
@effectfully
Copy link
Contributor

Fixing dead code instead of removing it is moronic.

else return $ GetResult (s {currPtr = currPtr s `plusPtr` 1, usedBits = u' - 8}) ()
let (bytes, bits) = (n + usedBits s) `divMod` 8
s' = s {currPtr = currPtr s `plusPtr` bytes, usedBits = bits}
if currPtr s' > endPtr
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem correct. I don't know whether endPtr points to end of buffer, or end of buffer + 1, but either way it doesn't seem correct.

In the former case, currPtr s' > endPtr is not an error if usedBits = 0.

In the latter case, currPtr s' == endPtr is an error if usedBits /= 0.

Copy link
Contributor Author

@Unisay Unisay Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, you're right. currPtr > endPtr misses the case where currPtr == endPtr with usedBits > 0, since that state points past the buffer but claims bits remain.

Rewrote consClose to use ensureBits + dropBits_, same pattern dropBits already uses. ensureBits checks (endPtr - currPtr) * 8 - usedBits < n, so the pointer and bit offset are both accounted for.

The previous divMod fix handled multi-byte advances correctly but
the bounds check (currPtr > endPtr) missed the case where
currPtr == endPtr with usedBits > 0. Rewrite consClose to use
ensureBits + dropBits_, matching the pattern dropBits already uses.

This ensures consClose itself rejects overruns rather than relying
on downstream decoders to catch the corrupted state.
Copy link
Contributor Author

@Unisay Unisay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also added a test for this: decoding a 9-bit constructor tag from a 1-byte buffer. It asserts the error is NotEnoughSpace (from consClose via ensureBits), not TooMuchSpace (from strictDecoder catching the bad state after the fact). See 1fed628.

@Unisay Unisay requested a review from zliu41 March 23, 2026 10:41
@Unisay Unisay requested a review from zliu41 March 23, 2026 18:48
Copy link
Member

@zliu41 zliu41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be upstreamed.

@zliu41 zliu41 merged commit c592a8d into master Mar 23, 2026
8 of 9 checks passed
@zliu41 zliu41 deleted the yura/issue-7542-flat-enum-bug branch March 23, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dead flat code has bugs in it

3 participants