Add join commutativity rule for reordering fact dimension join#18551
Open
siddharthteotia wants to merge 1 commit into
Open
Add join commutativity rule for reordering fact dimension join#18551siddharthteotia wants to merge 1 commit into
siddharthteotia wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18551 +/- ##
============================================
+ Coverage 63.75% 64.25% +0.50%
+ Complexity 1932 1120 -812
============================================
Files 3292 3312 +20
Lines 201470 203871 +2401
Branches 31316 31736 +420
============================================
+ Hits 128442 130996 +2554
+ Misses 62735 62358 -377
- Partials 10293 10517 +224
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improvement Summary
PinotJoinCommuteRuleto the MSE logical planner.dim JOIN fact, the rule swaps the inputs so the dim ends up on the right — fixing the build-side memory pressure that the syntactic-default planner gets wrong todayProblem
PinotJoinExchangeNodeInsertRuledecides distribution purely syntactically, and the runtime always builds the hash table from the right input (BaseJoinOperatorinvariant).dim JOIN fact:fact_tbl— the wrong side, memory-wise. For large fact tables this can exceedmax_rows_in_joinand fail queries that would otherwise succeed.Both problems share the same root: the right side is special (build side always, broadcast side sometimes), and the user's syntactic ordering puts the wrong table there.
Solution
dim INNER/LEFT/RIGHT/FULL JOIN fact— auto-commuted so dim ends up on the right.dim JOIN factwith/*+ join_strategy='lookup' */— previously a runtime error (LookupJoinOperatorrequires dim on the right); now auto-corrected. Lookup join no longer requires users to remember table order when writing the query.ResourceBasedQueriesTestnow propagatesisDimTable=trueto the planner, so dim-aware rules can be exercised end-to-end (was previously a silent gap).Testing