[AURON #2172] Add parquet suite wrappers for Spark versions by myandpr · Pull Request #2281 · apache/auron

myandpr · 2026-05-19T19:58:52Z

Which issue does this PR close?

Rationale for this change

Spark 3.3 already has Parquet correctness test wrappers. This PR extends the same Parquet correctness coverage to the other supported Spark test modules requested by the issue.

What changes are included in this PR?

Add Parquet correctness suite wrappers for:
- Spark 3.1
- Spark 3.2
- Spark 3.4
- Spark 3.5
- Spark 4.0
- Spark 4.1
Enable the Parquet suites in each corresponding AuronSparkTestSettings.
Enable Spark correctness tests in CI for the affected Spark versions.
Add version-specific exclusions for unsupported or failing Parquet correctness cases.
Skip suites that are not available in the corresponding Spark version.

Are there any user-facing changes?

No. This PR only adds test coverage.

How was this patch tested?

Ran affected Spark correctness tests locally.
Verified GitHub CI is green.

Copilot

Pull request overview

Extends Auron’s Spark Parquet correctness coverage beyond the existing Spark 3.3 module by adding version-specific wrapper suites and enabling them in each Spark test module’s AuronSparkTestSettings, while omitting suites that do not exist in older Spark versions.

Changes:

Added AuronParquet* wrapper test suites for Spark 3.1, 3.2, 3.4, 3.5, 4.0, and 4.1 test modules.
Enabled the new Parquet wrapper suites in each module’s AuronSparkTestSettings.
Version-gated suite coverage by only adding/enabling wrappers for suites present in that Spark version.

Reviewed changes

Copilot reviewed 188 out of 188 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
auron-spark-tests/spark31/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Import parquet package + enable Spark 3.1 Parquet wrapper suites
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala	Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Import parquet package + enable Spark 3.2 Parquet wrapper suites
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala	Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Import parquet package + enable Spark 3.4 Parquet wrapper suites
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala	Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Import parquet package + enable Spark 3.5 Parquet wrapper suites
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala	Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Import parquet package + enable Spark 4.0 Parquet wrapper suites
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala	Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Import parquet package + enable Spark 4.1 Parquet wrapper suites
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala	Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala	Spark 4.1 Parquet suite wrapper

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ShreyeshArangath

@myandpr I think the tests right now are not running, before we merge more PRs do you mind enabling the correctness tests for the other Spark versions?

auron-spark-tests/pom.xml needs six new profiles (spark-3.1, spark-3.2, spark-3.4, spark-3.5, spark-4.0, spark-4.1) each wiring their sparkXX submodule, mirroring the existing spark-3.3 profile.
github/workflows/tpcds.yml needs sparktests: true added to the test-spark-31, test-spark-32, test-spark-34, test-spark-35-, test-spark-40-, test-spark-41-* jobs. Currently only test-spark-33 has it

You can file a new issue if you'd like

myandpr · 2026-05-21T16:25:53Z

@myandpr I think the tests right now are not running, before we merge more PRs do you mind enabling the correctness tests for the other Spark versions?

auron-spark-tests/pom.xml needs six new profiles (spark-3.1, spark-3.2, spark-3.4, spark-3.5, spark-4.0, spark-4.1) each wiring their sparkXX submodule, mirroring the existing spark-3.3 profile.

github/workflows/tpcds.yml needs sparktests: true added to the test-spark-31, test-spark-32, test-spark-34, test-spark-35-, test-spark-40-, test-spark-41-* jobs. Currently only test-spark-33 has it

You can file a new issue if you'd like

Hi @ShreyeshArangath Thanks for pointing this out.

Updated the TPC-DS workflow to enable spark correctness tests for the affected Spark versions. The spark test profiles already exist in auron-spark-tests/pom.xml.

Could you please take another look? Thanks!

ShreyeshArangath · 2026-05-21T20:48:38Z

Thanks for fixing it! Change looks fine, do you mind looking at the failures?

myandpr · 2026-05-22T17:00:10Z

Thanks for fixing it! Change looks fine, do you mind looking at the failures?

@ShreyeshArangath Thanks. The failures have been fixed and CI is green now. Could you please take another look?

ShreyeshArangath

LGTM, but there are a lot of exclusions added here. Can we make sure to add relevant issues so we dont miss any actual issues before merging this in?

ShreyeshArangath · 2026-05-23T00:52:46Z

    .disable("Native execution can crash in Spark 4")
  enableSuite[AuronParquetPartitionDiscoverySuite]
    .exclude("read partitioned table - normal case")
+    .exclude("Resolve type conflicts - decimals, dates and timestamps in partition column")


Can we file an issue for this?

@ShreyeshArangath make sense. I filed a follow-up issue #2287 to track the newly added exclusions, including this one, so we can re-validate and remove them where possible.

Could you please take another look?

@ShreyeshArangath LGTM. A few follow-ups remain — happy to help get those addressed post-merge. Nice work! @myandpr

Make sense, feel free to take! @merrily01

merrily01 · 2026-05-27T07:39:46Z

Merged.

@myandpr Thanks for the contribution!
@ShreyeshArangath @slfan1989 @yew1eb Thanks for the review!

myandpr · 2026-05-27T07:43:42Z

@ShreyeshArangath @merrily01 @slfan1989 @yew1eb Thanks for your review!

[AURON apache#2172] Add parquet suite wrappers for Spark versions

721318f

github-actions Bot added the spark-tests label May 19, 2026

myandpr mentioned this pull request May 20, 2026

[Correctness Testing] All Spark Versions - Add Parquet related tests #2172

Closed

6 tasks

slfan1989 requested a review from Copilot May 20, 2026 15:42

Copilot started reviewing on behalf of slfan1989 May 20, 2026 15:42 View session

slfan1989 approved these changes May 20, 2026

View reviewed changes

Copilot AI reviewed May 20, 2026

View reviewed changes

ShreyeshArangath requested changes May 20, 2026

View reviewed changes

github-actions Bot added the infra label May 21, 2026

github-actions Bot added the build label May 21, 2026

myandpr force-pushed the add-parquet-suite-wrappers branch 2 times, most recently from 14874aa to 110425d Compare May 22, 2026 14:12

myandpr added 2 commits May 22, 2026 22:24

[AURON apache#2172] Enable Spark correctness tests in CI

526e230

[AURON apache#2172] Exclude unsupported correctness cases

5745ba5

myandpr force-pushed the add-parquet-suite-wrappers branch from 110425d to 5745ba5 Compare May 22, 2026 14:24

myandpr mentioned this pull request May 22, 2026

[WIP] Verify Spark 4 parquet exclusions #2282

Closed

myandpr added 2 commits May 22, 2026 23:15

[AURON apache#2172] Exclude Spark 4.1 partition discovery case

90e070d

[AURON apache#2172] Exclude Spark 4.0 partition discovery case

8907616

myandpr requested a review from ShreyeshArangath May 22, 2026 17:00

yew1eb approved these changes May 26, 2026

View reviewed changes

ShreyeshArangath approved these changes May 26, 2026

View reviewed changes

myandpr mentioned this pull request May 26, 2026

Track version-specific correctness test exclusions added for Spark suites #2287

Open

merrily01 approved these changes May 27, 2026

View reviewed changes

yew1eb approved these changes May 27, 2026

View reviewed changes

merrily01 merged commit f894f53 into apache:master May 27, 2026
125 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AURON #2172] Add parquet suite wrappers for Spark versions#2281

[AURON #2172] Add parquet suite wrappers for Spark versions#2281
merrily01 merged 5 commits into
apache:masterfrom
myandpr:add-parquet-suite-wrappers

myandpr commented May 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

ShreyeshArangath left a comment

Uh oh!

myandpr commented May 21, 2026 •

edited

Loading

Uh oh!

ShreyeshArangath commented May 21, 2026

Uh oh!

myandpr commented May 22, 2026

Uh oh!

ShreyeshArangath left a comment

Uh oh!

ShreyeshArangath May 23, 2026

Uh oh!

myandpr May 26, 2026 •

edited

Loading

Uh oh!

merrily01 May 27, 2026 •

edited

Loading

Uh oh!

myandpr May 27, 2026

Uh oh!

Uh oh!

merrily01 commented May 27, 2026

Uh oh!

myandpr commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

myandpr commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

ShreyeshArangath left a comment

Choose a reason for hiding this comment

Uh oh!

myandpr commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShreyeshArangath commented May 21, 2026

Uh oh!

myandpr commented May 22, 2026

Uh oh!

ShreyeshArangath left a comment

Choose a reason for hiding this comment

Uh oh!

ShreyeshArangath May 23, 2026

Choose a reason for hiding this comment

Uh oh!

myandpr May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

merrily01 May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

myandpr May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

merrily01 commented May 27, 2026

Uh oh!

myandpr commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

myandpr commented May 19, 2026 •

edited

Loading

myandpr commented May 21, 2026 •

edited

Loading

myandpr May 26, 2026 •

edited

Loading

merrily01 May 27, 2026 •

edited

Loading