feat: introduce stack-allocated PyBuffer

[winstxnhdw]

Summary

This PR implements a pinned stack-allocated PyUntypedBuffer variant, PyUntypedBufferView.

Closes #5836

[winstxnhdw]

I've left this incomplete for now because I want to hear more opinions on how to handle this.

Because we are using MaybeUninit to initialise Py_buffer, MaybeUninit never drops its contents, so we still need a way to call PyBuffer_Release to prevent a reference leak.

Here are some options I can think of:

1. Use a local drop guard holding a raw pointer to the Py_buffer, released on scope exit/panic
2. Construct an owned PyUntypedBufferView with Py_buffer::new(), letting Drop handle release, but this is not zero-cost.
3. Call assume_init_drop() or ptr::drop_in_place() at the end of with()

[davidhewitt]

I think a local drop guard is probably the correct approach, because it'll clean up properly on panics as you note. We have a lot of those patterns inside PyO3.

[winstxnhdw]

I didn't really like how the local drop guard approached looked, so I am trying to wrap MaybeUninit onto Py_buffer instead and then implementing Drop outside the trait. AFAICT, it seems like a sound alternative with no resource leaks or loss in performance. Let me know what you think.

[davidhewitt]

Thanks very much for this. Various thoughts around the accessor methods.

Also, out of scope for this PR, but I keep wondering if we should provide iterators for these structures. Especially with the strides / suboffsets etc, it's not necessarily trivial to get this right.

[davidhewitt]

I think a local drop guard is probably the correct approach, because it'll clean up properly on panics as you note. We have a lot of those patterns inside PyO3.

[davidhewitt]

Interesting note from the Python docs for itemsize

If shape is NULL as a result of a PyBUF_SIMPLE or a PyBUF_WRITABLE request, the consumer must disregard itemsize and assume itemsize == 1.

I am unsure whether it's better to have a corresponding check for shape here and return 1 if needed, or to return None if shape is None, or to just keep the implementation as-is and document this for users.

[winstxnhdw]

I am in favour of returning 1. Also, instead of doing the NULL check inside the method, we could probably do it in with() instead so that the check is only done once.

[winstxnhdw]

Also, out of scope for this PR, but I keep wondering if we should provide iterators for these structures. Especially with the strides / suboffsets etc, it's not necessarily trivial to get this right.

Yes, I think it would definitely be useful as well. I am not yet sure how this would look like though.