Conversation
Signed-off-by: Baris Palaska <barispalaska@gmail.com>
06ce5c2 to
e5d6ea0
Compare
Merging this PR will improve performance by ×8.5
Performance Changes
Comparing Footnotes
|
| let offsets = new_offsets.into_array(); | ||
| let sizes = new_sizes.into_array(); | ||
| let elements = new_elements_builder.finish(); | ||
| let elements = self.elements().take(take_indices.into_array())?; |
There was a problem hiding this comment.
this builds a lazy take plan, do you want this?
There was a problem hiding this comment.
I think you want a execute here?
| let offsets = new_offsets.into_array(); | ||
| let sizes = new_sizes.into_array(); | ||
| let elements = new_elements_builder.finish(); | ||
| let elements = self.elements().take(take_indices.into_array())?; |
There was a problem hiding this comment.
I think you want a execute here?
|
Its faster if you don't do the work 🙈 |
|
oh right all of our compute is lazy now haha But I'll be surprised if this is not significantly better than the old naive version for primitive types Edit: Actually are we sure we want to execute here? I don't think it is absolutely necessary to do so, especially if we want to filter/take after a rebuild has occurred (in which case we can optimize that plan) |
Signed-off-by: Baris Palaska <barispalaska@gmail.com>
|
Yeah my bad, didn't know take was lazy. It's still a lot faster than previous impl though |
| let elements = self | ||
| .elements() | ||
| .take(take_indices.into_array())? | ||
| .to_canonical()? | ||
| .into_array(); |
There was a problem hiding this comment.
In the case that we read ListViewArray from disk and we want to export to datafusion (basically if we want to export as a ListArray to Arrow), I believe that not executing here might be more beneficial.
If we want to push down a filter/take into a ListView from Datafusion/arrow, then we definitely do not want to execute here since it would be better to combine the filter/take with this take here.
I also think that this is the more common (perhaps only?) path that this naive_rebuild exists in.
There was a problem hiding this comment.
Yeah, not sure what the caller expects to get here. I think it makes sense to keep it lazy and leave canonicalization up to the caller, wdyt @joseph-isaacs ?
Replace per-list
extend_from_array(slice(...))with a singletakeinnaive_rebuild.The old approach canonicalized elements upfront, then called
extend_from_arrayper list — each call triggered buffer cloning for VarBinView. The new approach collects element indices into aBufferMut<u64>anddoes one bulk
take.Benchmarks (10k lists, overlapping views):