[SPARK-56739][SQL] Normalize CTE ids of orphan CTERelationRef in NormalizeCTEIds#56083
Draft
shrirangmhalgi wants to merge 1 commit into
Draft
[SPARK-56739][SQL] Normalize CTE ids of orphan CTERelationRef in NormalizeCTEIds#56083shrirangmhalgi wants to merge 1 commit into
shrirangmhalgi wants to merge 1 commit into
Conversation
…alizeCTEIds CTERelationRef nodes that exist outside any WithCTE node (e.g., after InlineCTE or MergeSubplans removes the parent WithCTE) were not getting their IDs normalized. This causes plan comparison and caching to fail because the same logical plan gets different CTE IDs across sessions. Add a case in applyInternal to normalize orphan CTERelationRefs whose cteId is already in the mapping.
shrirangmhalgi
force-pushed
the
SPARK-56739-normalize-orphan-cte-refs
branch
from
4d36b6e to
59ebea5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Normalize CTE IDs of orphan
CTERelationRefnodes inNormalizeCTEIds. Previously, onlyCTERelationRefnodes insideWithCTEwere normalized viacanonicalizeCTE. Refs that exist outside anyWithCTE(orphans) kept their original IDs.Why are the changes needed?
After
InlineCTEorMergeSubplansruns, someCTERelationRefnodes can end up outside their parentWithCTEnode. WhenNormalizeCTEIdsprocesses the plan, these orphan refs are skipped, leaving non-normalized IDs. This breaks plan comparison and caching because the same logical plan gets different CTE IDs across sessions (sinceCTERelationDefuses a global monotonically increasing counter).Does this PR introduce any user-facing change?
No. This is an internal plan normalization fix that affects plan caching correctness.
How was this patch tested?
Added
NormalizeCTEIdsSuitewith a test that constructs a plan with aCTERelationRefoutsideWithCTEand verifies all ref IDs are normalized. Without the fix, the orphan ref retains its original ID (100); with the fix, it's normalized to 0.Was this patch authored or co-authored using generative AI tooling?
Yes.