[SPARK-56756][SQL] Add error class for recursiveFileLookup conflict with partitioned data source#55721
Open
markj-db wants to merge 2 commits intoapache:masterfrom
Open
[SPARK-56756][SQL] Add error class for recursiveFileLookup conflict with partitioned data source#55721markj-db wants to merge 2 commits intoapache:masterfrom
markj-db wants to merge 2 commits intoapache:masterfrom
Conversation
…ith partitioned data source Replace the raw IllegalArgumentException thrown when recursiveFileLookup=true is combined with a partitioned data source with a tagged AnalysisException using the new RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE error class, so the failure is correctly classified as a user error.
sandip-db
approved these changes
May 6, 2026
| }, | ||
| "RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE" : { | ||
| "message" : [ | ||
| "Recursive file loading is not supported when the data source has explicit partition columns. Either remove the option \"recursiveFileLookup\", or read the data without supplying partition columns (for example, do not read a partitioned table or set partition-column options such as \"cloudFiles.partitionColumns\")." |
Contributor
There was a problem hiding this comment.
Suggested change
| "Recursive file loading is not supported when the data source has explicit partition columns. Either remove the option \"recursiveFileLookup\", or read the data without supplying partition columns (for example, do not read a partitioned table or set partition-column options such as \"cloudFiles.partitionColumns\")." | |
| "Recursive file loading is not supported when the data source has explicit partition columns. Either remove the option \"recursiveFileLookup\", or read the data without supplying partition columns (for example, do not read a partitioned table)." |
Strip the internal cloudFiles.partitionColumns example from the error message for RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE as this is OSS code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
PartitioningAwareFileIndex.listFilesrejects the combination ofrecursiveFileLookup=trueand a non-emptypartitionSpec().partitionColumnsby throwing a rawjava.lang.IllegalArgumentExceptionwith the message "Datasource with partition do not allow recursive file loading."This PR replaces that with a tagged
AnalysisExceptionusing a new error class:RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE(sqlState 0A000) inerror-conditions.json.QueryCompilationErrors.recursiveFileLookupNotSupportedForPartitionedDataSourceError().PartitioningAwareFileIndex.scalaupdated to use the helper.Why are the changes needed?
The raw
IllegalArgumentExceptionis unclassified and does not surface as a user-facing error with a clear message. Replacing it with anAnalysisExceptionusing a proper error class ensures it is correctly classified as a user error with an actionable message.Does this PR introduce any user-facing change?
Yes. Users who hit this error will now see a clearer message:
Previously the error was a raw
IllegalArgumentExceptionwith the message "Datasource with partition do not allow recursive file loading."How was this patch tested?
Added
"recursiveFileLookup with a partitioned catalog table is rejected"inFileBasedDataSourceSuite, which creates a partitioned Parquet catalog table, then asserts that reading it withrecursiveFileLookup=truethrows anAnalysisExceptionwith conditionRECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (claude-sonnet-4-6)