Proposal on n-σ lossy compression for RNTuple / AO2D #14289
Description
Motivation as desribed in the e-mail and presented during the OFFLINE week
After yesterday’s discussion with Andreas about ROOT RNTuple and generic lossy compression (for new AO2D), I have prepared a short presentation (3 slides to show) that I’d like to review right after the ROOT presentation this afternoon.
These slides continue the work already shown in several WP4/WP14 meetings on RootInteractive and the AO2D-derived data set (see code fragment [L2481-L2603]¹). They summarise the n-σ residual codec (Expectation & scale derived from other columns in the table) we use—partially in RootInteractive and fully in the AO2D prototype (slide Motivation) —and outline the plan to bring it into production (slide Proposal):
* AO2D C++ writer
* RootInteractive reader (Python → JavaScript)
* Full lazy evaluation
With ROOT I/O experts present, I’d appreciate feedback on whether
* ALICE should ship this solely as a custom layer on top of RNTuple, or
* parts of it ought to become first-class support within RNTuple.
The codec compresses “redundant” information by a factor 4–10 × while retaining intrinsic resolution, thus avoiding the physics-detail losses we encountered when Run-3 reconstruction dropped variables to save space.
Draft slides:
https://docs.google.com/presentation/d/16w_3Q6caWlLZKMi0QRidRRmO3mC5SQvo2WfJnqlk12w/edit?usp=sharing