chore(audit): audit struct expressions across Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1#4469
Open
andygrove wants to merge 1 commit into
Open
chore(audit): audit struct expressions across Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1#4469andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
….1, 4.1.1 Add per-version audit sub-bullets to `named_struct` and `struct` in `docs/source/contributor-guide/spark_expressions_support.md`. The Spark `CreateNamedStruct` class is byte-for-byte identical for behaviour across all four versions; only internal optimizer flags (`stateful` on 4.0, `contextIndependentFoldable` on 4.1) were added. Both `named_struct` and `struct` SQL functions lower to the same `CreateNamedStruct` node, so Comet handles them through one serde. Apply the one support-level consistency fix surfaced by the audit: - `CometCreateNamedStruct`: lift the duplicate-field-names fallback out of `convert` and into `getSupportLevel`, so the dispatcher handles the fallback uniformly and `getUnsupportedReasons()` documents the restriction for the compatibility guide. No correctness divergences were found, so no tracking issues are filed for this category.
This was referenced May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
Following the same pattern as #4436 (
any), #4437 (bit_and), and #4461 (string expressions), this PR audits thestruct_funcscategory in Comet against Spark 3.4.3, 3.5.8, 4.0.1, and now 4.1.1 (per the updated audit skill in #4468), records the findings inline in the support doc, and applies the one support-level consistency fix surfaced.What changes are included in this PR?
Support-doc audit notes
Add per-version audit sub-bullets to
named_structandstructindocs/source/contributor-guide/spark_expressions_support.md. The SparkCreateNamedStructclass is byte-for-byte identical for behaviour across all four versions; only internal optimizer flags (statefulon 4.0,contextIndependentFoldableon 4.1) were added. Bothnamed_structandstructSQL functions lower to the sameCreateNamedStructnode, so Comet handles them through one serde.Support-level consistency fix (in
structs.scala)CometCreateNamedStruct: lift the duplicate-field-names fallback out ofconvertand intogetSupportLevelvia a sharedprivate valso the dispatcher handles the fallback uniformly andgetUnsupportedReasons()documents the restriction for the compatibility guide.Tracking issues filed for follow-up
None. No correctness divergences were found for this category.
Audit process
Audited directly using the
audit-comet-expressionskill (4 Spark versions per the update in #4468). The category is small (one backing serde), so it did not need parallel subagents.How are these changes tested?
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite expressions/struct/" -Dtest=none(5 tests pass)make coresucceeds with the serde change.