[Bench] Avoid concat_batches in hash join#20944
Draft
Dandandan wants to merge 27 commits intoapache:mainfrom
Draft
[Bench] Avoid concat_batches in hash join#20944Dandandan wants to merge 27 commits intoapache:mainfrom
Dandandan wants to merge 27 commits intoapache:mainfrom
Conversation
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpch — base (merge-base)
tpch — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpch — base (merge-base)
tpch — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpch — base (merge-base)
tpch — branch
|
- Remove left_index_to_row closure parameter (only single-batch uses element-wise path, where packed == row index) - Hoist fallback row_indices and il_indices outside the key loop (lazy-allocated, computed once, reused across keys) - Replace scalar bitmap AND loop with word-level byte-slice AND via and_bitmap_with_boolean_buffer helper Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmarks |
|
Benchmark for this request failed. Last 20 lines of output: Click to expand |
2 similar comments
|
Benchmark for this request failed. Last 20 lines of output: Click to expand |
|
Benchmark for this request failed. Last 20 lines of output: Click to expand |
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpch — base (merge-base)
tpch — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
The multi-batch path in equal_rows_arr was using interleave() to materialize intermediate arrays, which is slower than the old take()-on-concatenated-keys approach. This adds compare_rows_elementwise_multi which decodes packed indices (batch_idx << 32 | row_idx) and compares values directly from per-batch arrays, avoiding all intermediate allocation for common key types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…atch API - Extract duplicated DataType match block into dispatch_elementwise! macro used by both single-batch and multi-batch comparison functions - Change compare_rows_elementwise_multi to take &[&ArrayRef] instead of leaking (left_arrays_per_batch, key_idx) internal structure - Pre-compute left null buffers per batch outside the hot loop in do_compare_elementwise_multi Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmarks |
When merging multiple partitions, CoalescePartitionsExec now coalesces small batches into larger ones (target_batch_size from session config). This benefits downstream operators like hash join whose build side receives fewer, larger batches — enabling the fast single-batch take() path instead of the slower multi-batch interleave() path in build_batch_from_indices. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpch — base (merge-base)
tpch — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
Contributor
Author
|
run benchmarks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Hash join (and other joins/operators as well - those can be refactored in a similar way ) currently concatenates the full build side.
When it is relatively large (and perhaps larger than the probe side), this has a few downsides
What changes are included in this PR?
This PR changes it to use
Vec<RecordBatch>and use a composite (u64) index to index into both batches / batch index.Are these changes tested?
Are there any user-facing changes?