[GLUTEN-10215][VL] Delta write: Native top-level NOT NULL checks#12030
Open
malinjawi wants to merge 1 commit intoapache:mainfrom
Open
[GLUTEN-10215][VL] Delta write: Native top-level NOT NULL checks#12030malinjawi wants to merge 1 commit intoapache:mainfrom
malinjawi wants to merge 1 commit intoapache:mainfrom
Conversation
025cebc to
6ce1a2b
Compare
6ce1a2b to
e702792
Compare
zhztheplayer
reviewed
May 8, 2026
Member
zhztheplayer
left a comment
There was a problem hiding this comment.
Just a initial design question atm.
Comment on lines
+32
to
+34
| private[delta] case class GlutenDeltaInvariantChecker private ( | ||
| notNullConstraints: Seq[(Int, NotNull)]) | ||
| extends Serializable { |
Member
There was a problem hiding this comment.
Compared to adding a runtime check, can we offload https://github.com/delta-io/delta/blob/ff715542a7b5b3aa6f0809b7a2c9b1760566ef60/spark/src/main/scala/org/apache/spark/sql/delta/constraints/CheckDeltaInvariant.scala#L35 to Velox instead?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This patch keeps simple Delta write invariants native in the Velox backend.
The change:
GlutenDeltaInvariantCheckerfor Delta 3.3 and Delta 4.0 sources.NOT NULLconstraints directly on Velox columnar batches before write.DeltaInvariantCheckerExecandColumnarToRowfor supported NOT NULL-only writes.CHECKconstraints and nested NOT NULL constraints still use Delta's row checker.nullCountwhen available.This is independent from #12024 and follows the Delta write C2R reduction work in #11419 and #12016.
Why are the changes needed?
Delta writes with table invariants currently go through
DeltaInvariantCheckerExec, which is row-based. Even simple top-levelNOT NULLconstraints can introduce a C2R transition in an otherwise native Delta write path.This patch handles the safe common case natively while preserving Delta's existing fallback behavior for unsupported constraints.
Does this PR introduce any user-facing change?
No public API change. Delta constraint behavior is preserved; supported NOT NULL-only writes can stay native.
How was this patch tested?
Built locally and ran:
DeltaNativeWriteInvariantSuite: passedDeltaNativeWriteSuite: passedlibvelox.dylibgit diff --check: passedCovered cases:
DeltaInvariantCheckerExecColumnarToRowDeltaInvariantCheckerExecInvariantViolationExceptionI also ran a targeted local benchmark for append workloads comparing the native top-level NOT NULL path with an equivalent unsupported CHECK fallback path.
Workload: 2,000,000 rows, 14 columns, 3 append iterations.
Excluding the first warm-up append:
The benchmark is effectively neutral because Delta write setup, Parquet output, and commit/log work dominate this microbenchmark. This PR is mainly native write correctness/posture work: it removes a row invariant operator and C2R transition from a common constrained Delta write path.
Related issue: #10215
Tracked by #12025