[native_datafusion] Enable native_datafusion in auto scan mode — test failure tracking

[andygrove]

Summary

Tracking issue for all test failures discovered when enabling native_datafusion in auto scan mode (#3307).

• [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully #3311
• [native_datafusion] [Spark SQL Tests] input_file_name() returns empty string #3312
• [native_datafusion] [Spark SQL Tests] Dynamic Partition Pruning (DPP) not working correctly #3313
• [native_datafusion] [Spark SQL Tests] Missing files error handling differs from Spark #3314
• [native_datafusion] [Spark SQL Tests] Plan structure differences cause test failures #3315
• [native_datafusion] [Spark SQL Tests] Parquet field ID matching not supported #3316
• [native_datafusion] [Spark SQL Tests] _tmp_metadata_row_index not populated #3317
• [native_datafusion] [Spark SQL Tests] Nested schema pruning tests fail — CometNativeScan not recognized as file source #3318
• [native_datafusion] [Spark SQL Tests] Bucketing not exposed by CometNativeScan #3319
• [native_datafusion] [Spark SQL Tests] CometNativeExec crash — executed without serialized plan #3320
• [native_datafusion] [Spark SQL Tests] Streaming self-union returns no results #3401
• Add DPP support to native_datafusion/CometNativeScan #3442
• native_datafusion: no error thrown for schema mismatch when reading Parquet with incompatible types #3720
• native_datafusion: case-insensitive mode doesn't detect duplicate/ambiguous Parquet fields #3760

[andygrove]

Once the above issues are all resolved, we can make this code change to CometScanRule:

COMET_NATIVE_SCAN_IMPL.get() match {
          case SCAN_AUTO =>
            nativeDataFusionScan(plan, session, scanExec, r, hadoopConf)
              .orElse {
                // Clear explain info tags from the failed nativeDataFusionScan
                // attempt so they don't leak into the fallback path
                scanExec.unsetTagValue(CometExplainInfo.EXTENSION_INFO)
                nativeIcebergCompatScan(session, scanExec, r, hadoopConf)
              }
              .getOrElse(scanExec)

[andygrove]

Current status:

#3311 — Schema/type mismatch errors (7 tests)

• Spark native readers should respect spark.sql.caseSensitive (FileBasedDataSourceSuite)
• SPARK-35640: read binary as timestamp should throw schema incompatible error (ParquetSchemaSuite)
• SPARK-35640: int as long should throw schema incompatible error (ParquetSchemaSuite)
• SPARK-36182: can't read TimestampLTZ as TimestampNTZ (ParquetSchemaSuite)
• SPARK-34212 Parquet should read decimals correctly (ParquetSchemaSuite)
• row group skipping doesn't overflow when reading into larger type (ParquetSchemaSuite)
• schema mismatch failure error message for parquet vectorized reader (ParquetSchemaSuite)
• SPARK-45604: schema mismatch failure error on timestamp_ntz to array<timestamp_ntz> (ParquetSchemaSuite)

#3320 — Parquet filter pushdown behavior (2 tests)

• SPARK-31026: Parquet predicate pushdown for fields having dots in the names (ParquetFilterSuite)
• Filters should be pushed down for Parquet readers at row group level (ParquetFilterSuite)

#3401 — Self-union streaming reads (2 tests)

• self-union, DSv1, read via DataStreamReader API (StreamSuite)
• self-union, DSv1, read via table API (StreamSuite)

[andygrove]

also need to look at:

-  test("filter pushdown - StringPredicate") {
+  test("filter pushdown - StringPredicate",
+    IgnoreCometNativeDataFusion("cannot be pushed down")) {

need to file separate issue

[andygrove]

also need to look at:

-  test("filter pushdown - StringPredicate") {
+  test("filter pushdown - StringPredicate",
+    IgnoreCometNativeDataFusion("cannot be pushed down")) {

need to file separate issue

PR to fix this issue - #3830