GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification by prtkgaur · Pull Request #557 · apache/parquet-format

prtkgaur · 2026-03-11T04:03:57Z

Add the encoding specification for ALP (encoding value 10) to Encodings.md. ALP compresses FLOAT and DOUBLE columns by converting values to integers via decimal scaling, then applying Frame of Reference encoding and bit-packing. Values that cannot be losslessly round-tripped are stored as exceptions.

Closes [Proposal] Add ALP encoding support in parquet file format #533

See rendered preview here: https://github.com/prtkgaur/parquet-format/blob/alpEncoding/Encodings.md#adaptive-lossless-floating-point-alp--10

The spec covers:

Page layout: 7-byte header, offset array, compressed vectors
Vector format: AlpInfo, ForInfo, packed values, exception data
Encoding math: two-step multiplication for cross-language consistency
Parameter selection, exception detection, and decoding steps

Based on the paper "ALP: Adaptive Lossless floating-Point Compression" (Afroozeh and Boncz, SIGMOD 2024). Wire format matches the C++ Arrow and Java parquet-java implementations.

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Add the encoding specification for ALP (encoding value 10) to Encodings.md. ALP compresses FLOAT and DOUBLE columns by converting values to integers via decimal scaling, then applying Frame of Reference encoding and bit-packing. Values that cannot be losslessly round-tripped are stored as exceptions. The spec covers: - Page layout: 7-byte header, offset array, compressed vectors - Vector format: AlpInfo, ForInfo, packed values, exception data - Encoding math: two-step multiplication for cross-language consistency - Parameter selection, exception detection, and decoding steps Based on the paper "ALP: Adaptive Lossless floating-Point Compression" (Afroozeh and Boncz, SIGMOD 2024). Wire format matches the C++ Arrow and Java parquet-java implementations.

emkornfield · 2026-03-24T23:22:32Z

Encodings.md

 ```
 Bytes  AA 00 A3 BB 11 B4 CC 22 C5 DD 33 D6
 ```
+


I think there are now links at the top of file summarizing applicability of encoding.

emkornfield · 2026-03-24T23:24:34Z

Encodings.md

+
+##### Header (7 bytes)
+
+All multi-byte values are little-endian.


Suggested change

All multi-byte values are little-endian.

All multi-byte values are stored in little-endian order.

emkornfield · 2026-03-24T23:25:47Z

Encodings.md

+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) |
+| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) |
+| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) |


emkornfield · 2026-03-24T23:26:59Z

Encodings.md

+
+| Offset | Field | Size | Type | Description |
+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) |


we think ALP-RD fits in well here? I forget what the extension point is and why we were OK keeping this field, but not version.

emkornfield · 2026-03-24T23:27:36Z

Encodings.md

+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) |
+| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) |
+| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) |


emkornfield · 2026-03-24T23:28:58Z

Encodings.md

+**Note:** The number of elements per vector and the packed data size are NOT stored
+in the header. They are derived:
+* Elements per vector: `vector_size` for all vectors except the last, which may be smaller.
+* Packed data size: `ceil(num_elements_in_vector * bit_width / 8)`.


bit_width isn't in the header either, so it is a little strange to call ths out here?

emkornfield · 2026-03-24T23:30:01Z

Encodings.md

+
+**Note:** The number of elements per vector and the packed data size are NOT stored
+in the header. They are derived:
+* Elements per vector: `vector_size` for all vectors except the last, which may be smaller.


This seems like a little bit of a strange callout since it is covered on line 457 explicitly. and log_vector_size is stored in the header?

emkornfield · 2026-03-24T23:31:06Z

Encodings.md

+values. Each offset gives the byte position of the corresponding vector's data,
+measured from the start of the offset array itself.
+
+The first offset equals `num_vectors * 4` (pointing just past the offset array).


Suggested change

The first offset equals `num_vectors * 4` (pointing just past the offset array).

The first offset always equals `num_vectors * 4` (pointing just past the offset array).

Lets be explicitly here that we don't support padding.

emkornfield · 2026-03-24T23:33:04Z

Encodings.md

+Data section sizes:
+| Section             | Size Formula                | Description                  |
+|---------------------|-----------------------------|------------------------------|
+| PackedValues        | ceil(N * bit\_width / 8)    | Bit-packed delta values      |


emkornfield · 2026-03-24T23:35:09Z

Encodings.md

+|---------------------|-----------------------------|------------------------------|
+| PackedValues        | ceil(N * bit\_width / 8)    | Bit-packed delta values      |
+| ExceptionPositions  | num\_exceptions * 2 bytes   | uint16 indices of exceptions |
+| ExceptionValues     | num\_exceptions * sizeof(T) | Original float/double values |


emkornfield · 2026-03-24T23:37:48Z

Encodings.md

+
+The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes.
+Values are packed from the least significant bit of each byte to the most significant bit,
+in groups of 8 values, using the same bit-packing order as the


Where does the group of 8 values come in? Wouldn't this messup the number of bytes math?

emkornfield · 2026-03-24T23:38:55Z

Encodings.md

+The encoding uses two separate multiplications (not a single multiplication by
+`10^(e-f)`, and not division) to ensure that implementations produce identical
+floating-point rounding across languages. The powers of 10 MUST be stored as
+precomputed floating-point constants (i.e., literal values like `1e-3f`), not


Why can't they be precomputed at runtime?

emkornfield · 2026-03-24T23:39:39Z

Encodings.md

+| Type   | Magic Number                      | Formula                          |
+|--------|-----------------------------------|----------------------------------|
+| FLOAT  | 2^22 + 2^23 = 12,582,912         | `(int)((value + magic) - magic)` |
+| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(long)((value + magic) - magic)` |


emkornfield · 2026-03-24T23:39:46Z

Encodings.md

+
+| Type   | Magic Number                      | Formula                          |
+|--------|-----------------------------------|----------------------------------|
+| FLOAT  | 2^22 + 2^23 = 12,582,912         | `(int)((value + magic) - magic)` |


emkornfield · 2026-03-24T23:40:13Z

Encodings.md

+```
+-------------------------------------------------------------------+
+|                                                                   |
+|   encoded = round( value  *  10^e  *  10^(-f) )                  |


Suggested change

| encoded = round( value * 10^e * 10^(-f) ) |

| encoded = fast_round( value * 10^e * 10^(-f) ) |

alamb changed the title ~~Add ALP (Adaptive Lossless floating-Point) encoding specification~~ GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification Mar 11, 2026

This was referenced Mar 11, 2026

GH-533: Adaptive Lossless Floating-Point (ALP) Encoding #548

Closed

[WIp] Alp encoding support apache/arrow-rs#9372

Draft

emkornfield reviewed Mar 24, 2026

View reviewed changes


		##### Header (7 bytes)

		All multi-byte values are little-endian.

	All multi-byte values are little-endian.
	All multi-byte values are stored in little-endian order.

	\| 2 \| log_vector_size \| 1 byte \| uint8 \| log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) \|
	\| 2 \| log_vector_size \| 1 byte \| uint8 \| log2(vector\_size). Must be in \[3, 15\]. Recommended default: 10 (vector size 1024) \|

	The first offset equals `num_vectors * 4` (pointing just past the offset array).
	The first offset always equals `num_vectors * 4` (pointing just past the offset array).

	\| PackedValues \| ceil(N * bit\_width / 8) \| Bit-packed delta values \|
	\| PackedValues \| ceil(`vector_size` * bit\_width / 8) \| Bit-packed delta values \|

	\| ExceptionValues \| num\_exceptions * sizeof(T) \| Original float/double values \|
	\| ExceptionValues \| num\_exceptions * sizeof(encoded type) (float=4 and double=8) \| Original float/double values \|

	\| DOUBLE \| 2^51 + 2^52 = 6,755,399,441,055,744 \| `(long)((value + magic) - magic)` \|
	\| DOUBLE \| 2^51 + 2^52 = 6,755,399,441,055,744 \| `(int64_t)((value + magic) - magic)` \|

	\| FLOAT \| 2^22 + 2^23 = 12,582,912 \| `(int)((value + magic) - magic)` \|
	\| FLOAT \| 2^22 + 2^23 = 12,582,912 \| `(int32_t)((value + magic) - magic)` \|

	\| encoded = round( value * 10^e * 10^(-f) ) \|
	\| encoded = fast_round( value * 10^e * 10^(-f) ) \|

Conversation

prtkgaur commented Mar 11, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prtkgaur commented Mar 11, 2026 •

edited by alamb

Loading