-
Notifications
You must be signed in to change notification settings - Fork 483
DPL: add support for decompressing directly to shared memory #13763
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
REQUEST FOR PRODUCTION RELEASES: This will add The following labels are available |
|
First step towards being able to read and write RNTuple and TTree seamleassly. |
|
Tested on hyperloop. Not OK. One leak fixed, one more to go. |
|
Error while checking build/O2/fullCI for 8c955dd at 2024-12-14 04:53: Full log here. |
|
Error while checking build/O2/fullCI for 84ca389 at 2025-01-01 00:35: Full log here. |
|
Error while checking build/O2/fullCI for 70b4151 at 2025-01-29 23:28: Full log here. |
|
Error while checking build/O2/fullCI_slc9 for 70b4151 at 2025-01-28 00:44: Full log here. |
1519b94 to
05f2abc
Compare
|
Error while checking build/O2/fullCI_slc9 for 1fe4d36 at 2025-02-03 22:41: Full log here. |
|
Error while checking build/O2/fullCI for 1fe4d36 at 2025-02-04 07:17: Full log here. |
78d334e to
e9b282d
Compare
|
Error while checking build/O2/fullCI for e9b282d at 2025-02-11 22:07: Full log here. |
|
Error while checking build/O2/fullCI_slc9 for e9b282d at 2025-02-05 10:26: Full log here. |
This PR postpones the read operations which would usually populate an intermediate RecordBatch and it performs them directly on its subsequent shared memory serialization. Doing so avoids having the intermediate representation allocate most of the memory. For the moment this is only done for TTree. RNtuple support will come in a subsequent PR.
|
Tested on hyperloop, including using a workflows from a few usual suspects and actually submitting on the Grid. Merging to get it in tomorrow tag. |
This PR postpones the read operations which would usually populate an intermediate
RecordBatch and it performs them directly on its subsequent shared memory
serialization. Doing so avoids having the intermediate representation allocate most
of the memory.
For the moment this is only done for TTree. RNtuple support will come in a subsequent PR.