[SPARK-56969][SQL] Enhance the SQL config spark.sql.timestampNanosTypes.enabled#56094
Open
MaxGekk wants to merge 4 commits into
Open
[SPARK-56969][SQL] Enhance the SQL config spark.sql.timestampNanosTypes.enabled#56094MaxGekk wants to merge 4 commits into
spark.sql.timestampNanosTypes.enabled#56094MaxGekk wants to merge 4 commits into
Conversation
…cument preview conf Extend TypeUtils.failUnsupportedDataType to reject TimestampNTZNanosType and TimestampLTZNanosType when spark.sql.timestampNanosTypes.enabled is off, add UNSUPPORTED_TIMESTAMP_NANOS_TYPE, align the conf default with Utils.isTesting, and add a short enablement note in sql-ref-datatypes.md.
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Do we still need [WIP] in the PR title, @MaxGekk ?
MaxGekk
changed the title
spark.sql.timestampNanosTypes.enabledspark.sql.timestampNanosTypes.enabled
Member
Author
|
@dongjoon-hyun The PR is ready for review. I removed the |
MaxGekk
commented
May 25, 2026
| "types. Enabling this flag does not guarantee full SQL support: casts, Parquet read, " + | ||
| "typed literals, and other operations may still fail until their respective features " + | ||
| "are implemented.") | ||
| .version("4.2.0") |
Member
Author
There was a problem hiding this comment.
@HyukjinKwon @cloud-fan @dongjoon-hyun I merged this config to master, will it be released in 4.2.0? Should I merge new features to branch-4.x?
Mirror the SQLConf default (Utils.isTesting) in DefaultSqlApiConf so no-session-bound paths (Spark Connect parsing without a session) match session-bound behavior. Drop the link to "Runtime SQL Configuration" in sql-ref-datatypes.md since the conf is .internal() and is not listed there. Co-authored-by: Isaac
Member
Author
|
@stevomitric Please, have a look at the PR. It is related to your changes in #56041 |
stevomitric
reviewed
May 25, 2026
Contributor
stevomitric
left a comment
stevomitric
left a comment
There was a problem hiding this comment.
Latest commit says "Co-authored-by: Isaac" but the description states "Generated-by: Cursor Auto".
Maybe squash the latest commits?
| } | ||
|
|
||
| def unsupportedTimestampNanosTypeError(): Throwable = { | ||
| new AnalysisException( |
Contributor
There was a problem hiding this comment.
Why not reuse FEATURE_NOT_ENABLED ?
Drop the dedicated UNSUPPORTED_TIMESTAMP_NANOS_TYPE error condition and route the analysis-time gate through the same FEATURE_NOT_ENABLED throw used by the parser path. Extract the throw into DataTypeErrors.timestampNanosTypesNotEnabledError so both gates share one source of truth. Short-circuit on the conf flag before traversing the data type recursively. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR completes SPARK-56969 on top of the parser gating added in SPARK-56965 / #56041.
TypeUtils.failUnsupportedDataTypeto recursively rejectTimestampNTZNanosTypeandTimestampLTZNanosTypewhenspark.sql.timestampNanosTypes.enabledis off.UNSUPPORTED_TIMESTAMP_NANOS_TYPEwith a message naming the conf key.spark.sql.timestampNanosTypes.enabledand align its default withUtils.isTesting(mirroringspark.sql.timeType.enabled).docs/sql-ref-datatypes.md.Part of SPIP SPARK-56822.
Why are the changes needed?
Parser and JSON entry points already gate parameterized nanos timestamp types behind
spark.sql.timestampNanosTypes.enabled, but analyzed schemas/plans could still surface these types through other paths (for exampleCREATE TABLE, connectors, or programmatic schemas) before downstream execution support is ready.Analysis-time gating closes that gap and keeps behavior consistent with the existing preview flag.
Does this PR introduce any user-facing change?
Yes.
spark.sql.timestampNanosTypes.enabledis off, analyzed schemas/plans containingTIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p)withpin[7, 9]now fail withUNSUPPORTED_TIMESTAMP_NANOS_TYPE.Utils.isTestinginstead of alwaysfalse, so tests enable the preview by default while production remains off.docs/sql-ref-datatypes.mddocuments how to enable the preview feature.Unparameterized
TIMESTAMP,TIMESTAMP_NTZ, andTIMESTAMP_LTZbehavior is unchanged.Example:
How was this patch tested?
Added/updated unit tests:
Existing nanos parser/JSON tests continue to pass with the conf enabled.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Auto