Skip to content

[SPARK-51549][BUILD][3.5] Bump Parquet 1.15.1 #50528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 9 commits into from
Closed

[SPARK-51549][BUILD][3.5] Bump Parquet 1.15.1 #50528

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
cd744f3
[SPARK-48177][BUILD] Upgrade `Apache Parquet` to 1.14.1
Fokko Jul 2, 2024
0a44ebb
[SPARK-49310][BUILD] Upgrade `Parquet` to 1.14.2
Fokko Sep 11, 2024
7a24a64
[SPARK-49903][BUILD] Upgrade `Parquet` to 1.14.3
panbingkun Oct 8, 2024
a91ae40
[SPARK-50259][BUILD] Update Parquet to 1.14.4
Fokko Nov 12, 2024
49c0707
[SPARK-50425][BUILD] Bump Apache Parquet to 1.15.0
Fokko Dec 3, 2024
aaa6fc6
[SPARK-51549][BUILD] Bump Parquet 1.15.1
pan3793 Mar 21, 2025
0936ead
Fix build and UT
wangyum Apr 7, 2025
ad82597
Exclude com.h2database:h2
wangyum Apr 8, 2025
4dc47ea
fix
wangyum Apr 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,12 +218,12 @@ orc-shims/1.9.5//orc-shims-1.9.5.jar
oro/2.0.8//oro-2.0.8.jar
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
paranamer/2.8//paranamer-2.8.jar
parquet-column/1.13.1//parquet-column-1.13.1.jar
parquet-common/1.13.1//parquet-common-1.13.1.jar
parquet-encoding/1.13.1//parquet-encoding-1.13.1.jar
parquet-format-structures/1.13.1//parquet-format-structures-1.13.1.jar
parquet-hadoop/1.13.1//parquet-hadoop-1.13.1.jar
parquet-jackson/1.13.1//parquet-jackson-1.13.1.jar
parquet-column/1.15.1//parquet-column-1.15.1.jar
parquet-common/1.15.1//parquet-common-1.15.1.jar
parquet-encoding/1.15.1//parquet-encoding-1.15.1.jar
parquet-format-structures/1.15.1//parquet-format-structures-1.15.1.jar
parquet-hadoop/1.15.1//parquet-hadoop-1.15.1.jar
parquet-jackson/1.15.1//parquet-jackson-1.15.1.jar
pickle/1.3//pickle-1.3.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
Expand Down
8 changes: 7 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@
<kafka.version>3.4.1</kafka.version>
<!-- After 10.15.1.3, the minimum required version is JDK9 -->
<derby.version>10.14.2.0</derby.version>
<parquet.version>1.13.1</parquet.version>
<parquet.version>1.15.1</parquet.version>
<orc.version>1.9.5</orc.version>
<orc.classifier>shaded-protobuf</orc.classifier>
<jetty.version>9.4.56.v20240826</jetty.version>
Expand Down Expand Up @@ -2663,6 +2663,12 @@
<version>${parquet.version}</version>
<scope>${parquet.test.deps.scope}</scope>
<classifier>tests</classifier>
<exclusions>
<exclusion>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
</exclusion>
</exclusions>
Comment on lines +2666 to +2671
Copy link
Member Author

@wangyum wangyum Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To fix:

 build/sbt "sql/testOnly *.JDBCSuite"

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.UnsupportedClassVersionError: org/h2/Driver has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)

</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
Expand Down
70 changes: 35 additions & 35 deletions sql/core/benchmarks/BuiltInDataSourceWriteBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,69 +2,69 @@
Parquet writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1031-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_442-b06 on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
Parquet(PARQUET_1_0) writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 2724 2758 49 5.8 173.2 1.0X
Output Single Double Column 2816 2829 20 5.6 179.0 1.0X
Output Int and String Column 8999 9080 115 1.7 572.1 0.3X
Output Partitions 5003 5086 117 3.1 318.1 0.5X
Output Buckets 6911 6956 64 2.3 439.4 0.4X
Output Single Int Column 1685 1742 81 9.3 107.1 1.0X
Output Single Double Column 1675 1774 139 9.4 106.5 1.0X
Output Int and String Column 5038 5126 125 3.1 320.3 0.3X
Output Partitions 2904 2927 33 5.4 184.6 0.6X
Output Buckets 4051 4058 10 3.9 257.6 0.4X

OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1031-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_442-b06 on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
Parquet(PARQUET_2_0) writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 2761 2806 64 5.7 175.5 1.0X
Output Single Double Column 2652 2678 37 5.9 168.6 1.0X
Output Int and String Column 8377 8518 199 1.9 532.6 0.3X
Output Partitions 4865 4914 70 3.2 309.3 0.6X
Output Buckets 6622 6664 59 2.4 421.0 0.4X
Output Single Int Column 1545 1551 9 10.2 98.2 1.0X
Output Single Double Column 1605 1629 34 9.8 102.0 1.0X
Output Int and String Column 5077 5107 42 3.1 322.8 0.3X
Output Partitions 2819 2822 3 5.6 179.2 0.5X
Output Buckets 3911 3911 0 4.0 248.7 0.4X


================================================================================================
ORC writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1031-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_442-b06 on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
ORC writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1575 1627 74 10.0 100.1 1.0X
Output Single Double Column 2021 2087 94 7.8 128.5 0.8X
Output Int and String Column 6533 6800 377 2.4 415.4 0.2X
Output Partitions 3577 3635 82 4.4 227.4 0.4X
Output Buckets 4895 4923 41 3.2 311.2 0.3X
Output Single Int Column 944 974 32 16.7 60.0 1.0X
Output Single Double Column 1514 1518 6 10.4 96.3 0.6X
Output Int and String Column 4797 4801 6 3.3 305.0 0.2X
Output Partitions 2270 2272 3 6.9 144.3 0.4X
Output Buckets 3201 3222 30 4.9 203.5 0.3X


================================================================================================
JSON writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1031-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_442-b06 on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
JSON writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 2415 2465 71 6.5 153.6 1.0X
Output Single Double Column 3690 3856 236 4.3 234.6 0.7X
Output Int and String Column 6922 6930 12 2.3 440.1 0.3X
Output Partitions 4619 4622 4 3.4 293.7 0.5X
Output Buckets 6674 6756 116 2.4 424.3 0.4X
Output Single Int Column 1659 1671 17 9.5 105.4 1.0X
Output Single Double Column 2260 2262 4 7.0 143.7 0.7X
Output Int and String Column 4963 4964 2 3.2 315.5 0.3X
Output Partitions 2912 2915 3 5.4 185.2 0.6X
Output Buckets 3868 3870 3 4.1 245.9 0.4X


================================================================================================
CSV writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1031-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 1.8.0_442-b06 on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
CSV writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 4276 4368 130 3.7 271.8 1.0X
Output Single Double Column 5273 5346 104 3.0 335.2 0.8X
Output Int and String Column 8999 9139 199 1.7 572.1 0.5X
Output Partitions 6466 6526 85 2.4 411.1 0.7X
Output Buckets 8844 8878 48 1.8 562.3 0.5X
Output Single Int Column 2603 2606 4 6.0 165.5 1.0X
Output Single Double Column 2887 2888 1 5.4 183.6 0.9X
Output Int and String Column 6464 6492 40 2.4 411.0 0.4X
Output Partitions 3844 3896 73 4.1 244.4 0.7X
Output Buckets 5662 5671 13 2.8 360.0 0.5X


Loading