Skip to content

Bulk take in ListView naive_rebuild#6492

Open
palaska wants to merge 2 commits intodevelopfrom
bp/listview-rebuild
Open

Bulk take in ListView naive_rebuild#6492
palaska wants to merge 2 commits intodevelopfrom
bp/listview-rebuild

Conversation

@palaska
Copy link
Contributor

@palaska palaska commented Feb 13, 2026

Replace per-list extend_from_array(slice(...)) with a single take in naive_rebuild.

The old approach canonicalized elements upfront, then called extend_from_array per list — each call triggered buffer cloning for VarBinView. The new approach collects element indices into a BufferMut<u64> and
does one bulk take.

Benchmarks (10k lists, overlapping views):

  • Primitives: 2,006 µs → 79 µs (~25x)
  • VarBinView: 3,972 µs → 74 µs (~52x)

@palaska palaska added the changelog/performance A performance improvement label Feb 13, 2026
Signed-off-by: Baris Palaska <barispalaska@gmail.com>
@palaska palaska force-pushed the bp/listview-rebuild branch from 06ce5c2 to e5d6ea0 Compare February 13, 2026 16:19
@codspeed-hq
Copy link

codspeed-hq bot commented Feb 13, 2026

Merging this PR will improve performance by ×8.5

⚡ 1 improved benchmark
✅ 1134 untouched benchmarks
🆕 6 new benchmarks
⏩ 1268 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Simulation rebuild_varbinview[1000] N/A 338.8 µs N/A
🆕 Simulation rebuild_primitive[10000] N/A 2.5 ms N/A
🆕 Simulation rebuild_primitive[1000] N/A 295.5 µs N/A
🆕 Simulation rebuild_primitive[100] N/A 71.1 µs N/A
🆕 Simulation rebuild_varbinview[100] N/A 102.2 µs N/A
🆕 Simulation rebuild_varbinview[10000] N/A 4.4 ms N/A
Simulation rebuild_naive 2,718 µs 321 µs ×8.5

Comparing bp/listview-rebuild (dd49507) with develop (5de9826)

Open in CodSpeed

Footnotes

  1. 1268 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Copy link
Contributor

@connortsui20 connortsui20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice! LGTM

let offsets = new_offsets.into_array();
let sizes = new_sizes.into_array();
let elements = new_elements_builder.finish();
let elements = self.elements().take(take_indices.into_array())?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this builds a lazy take plan, do you want this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you want a execute here?

let offsets = new_offsets.into_array();
let sizes = new_sizes.into_array();
let elements = new_elements_builder.finish();
let elements = self.elements().take(take_indices.into_array())?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you want a execute here?

@joseph-isaacs
Copy link
Contributor

Its faster if you don't do the work 🙈

@connortsui20
Copy link
Contributor

connortsui20 commented Feb 13, 2026

oh right all of our compute is lazy now haha

But I'll be surprised if this is not significantly better than the old naive version for primitive types

Edit: Actually are we sure we want to execute here? I don't think it is absolutely necessary to do so, especially if we want to filter/take after a rebuild has occurred (in which case we can optimize that plan)

Signed-off-by: Baris Palaska <barispalaska@gmail.com>
@palaska
Copy link
Contributor Author

palaska commented Feb 13, 2026

Yeah my bad, didn't know take was lazy. It's still a lot faster than previous impl though

Comment on lines +155 to +159
let elements = self
.elements()
.take(take_indices.into_array())?
.to_canonical()?
.into_array();
Copy link
Contributor

@connortsui20 connortsui20 Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case that we read ListViewArray from disk and we want to export to datafusion (basically if we want to export as a ListArray to Arrow), I believe that not executing here might be more beneficial.

If we want to push down a filter/take into a ListView from Datafusion/arrow, then we definitely do not want to execute here since it would be better to combine the filter/take with this take here.

I also think that this is the more common (perhaps only?) path that this naive_rebuild exists in.

cc @joseph-isaacs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, not sure what the caller expects to get here. I think it makes sense to keep it lazy and leave canonicalization up to the caller, wdyt @joseph-isaacs ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants