Skip to content

chore(audit): audit bitwise expressions across Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 #4479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andygrove wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore(audit): audit bitwise expressions across Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 #4479

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/source/contributor-guide/spark_expressions_support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,17 +163,61 @@
### bitwise_funcs

- [x] `&`
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseAnd(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseAnd` to the proto's `bitwise_and` binary expression.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] `<<`
- Spark 3.4.3 (audited 2026-05-27): only the function form `shiftleft(...)` exists; the `<<` operator alias is added in Spark 4.0.
- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- Spark 4.0.1 (audited 2026-05-27): `<<` parses to `ShiftLeft` (the same `case class` as `shiftleft`) via a parser alias. `ShiftLeft` now extends a shared `BitShiftOperation` trait; eval semantics unchanged. Comet `CometShiftLeft` casts the `right` operand to `LongType` when `left.dataType == LongType` because DataFusion's bitwise shift requires matching operand types.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] `>>`
- Spark 3.4.3 (audited 2026-05-27): same as `<<` history — only the function form exists pre-4.0.
- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- Spark 4.0.1 (audited 2026-05-27): `>>` parses to `ShiftRight`. Comet `CometShiftRight` mirrors the same operand-cast logic as `CometShiftLeft`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [ ] `>>>`
- [x] `^`
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseXor(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseXor` to the proto's `bitwise_xor` binary expression.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] bit_count
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseCount(child) extends UnaryExpression`; accepts `IntegralType` or `BooleanType` and returns the population count as `IntegerType`. Wired as `CometScalarFunction("bit_count")`.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] bit_get
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseGet(left, right) extends BinaryExpression`; `inputTypes = (IntegralType, IntegerType) -> ByteType`. The position is bounds-checked at runtime via `BitwiseGetUtil.checkPosition`. Comet routes via `CometBitwiseGet` to the native `bit_get` scalar.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] getbit
- Spark 3.4.3 (audited 2026-05-27): registry alias of `BitwiseGet`. Same support as `bit_get`.
- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
- [x] shiftright
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. Function name for `ShiftRight`. Same support as the `>>` operator alias added in 4.0.
- Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8 (the `>>` operator is the only addition).
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] shiftrightunsigned
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `ShiftRightUnsigned(left, right) extends BinaryExpression`; uses Java's `>>>` (zero-fill) semantics. Comet wires as `CometScalarFunction("shiftrightunsigned")` (added in #4375).
- Spark 4.0.1 (audited 2026-05-27): now extends the shared `BitShiftOperation` trait; eval semantics unchanged. The `>>>` operator alias is added at the parser level.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] `|`
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseOr(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseOr` to the proto's `bitwise_or` binary expression.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- [x] `~`
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseNot(child) extends UnaryExpression`; accepts `IntegralType` and returns the bitwise complement of the same type. Comet routes via `CometBitwiseNot` to the native `bitwise_not` scalar.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.

### collection_funcs

Expand Down