[SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor to mitigate perf regression by pan3793 · Pull Request #53454 · apache/spark

pan3793 · 2025-12-12T08:44:57Z

What changes were proposed in this pull request?

Previously, lz4-java was upgraded to 1.10.x to address CVEs,

while this casues significant performance drop, see the benchmark report at

[SPARK-54693][CORE][TESTS] Add LZ4TPCDSDataBenchmark #53453

this PR follows the suggestion to migrate to safeDecompressor.

Why are the changes needed?

Mitigate performance regression.

Does this PR introduce any user-facing change?

No, except for performance.

How was this patch tested?

GHA for functionality, benchmark for performance.

TL;DR - my test results show lz4-java 1.10.1 is about 10~15% slower on lz4 compression than 1.8.0, and is about ~5% slower on lz4 decompression even with migrating to suggested safeDecompressor

Was this patch authored or co-authored using generative AI tooling?

No.

pan3793 · 2025-12-14T19:30:39Z

the test failure is caused by com.ibm.db2:jcc includes unshaded old lz4 classes

pan3793 · 2025-12-15T02:39:33Z

cc @dbtsai @huaxingao, com.ibm.db2:jcc includes unshaded old lz4 classes, which causes sql/hive/docker-it modules test failure after this patch

java.lang.NoSuchMethodError: 'net.jpountz.lz4.LZ4BlockInputStream$Builder net.jpountz.lz4.LZ4BlockInputStream.newBuilder()'
 	at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:156)
 	...

I checked all versions available in Maven Central, all of them have the same issue.
https://mvnrepository.com/artifact/com.ibm.db2/jcc

~~I don't find the public contact info of the IBM DB2 JDBC driver team, not sure what's the next step, temporarily purge the dependency and disable DB2 tests? Or any better ideas?~~

Update: contacted the DB2 JDBC driver's author, new release that bundles the latest lz4-java is working in progress

Update: DB2 team provides a special JDBC driver 12.1.3.0_special_74723 that bundles lz4-java 1.10.1 which addressed NoSuchMethodError issue.

github-actions · 2026-01-22T06:30:25Z

JIRA Issue Information

=== Improvement SPARK-54571 ===
Summary: Use LZ4 safeDecompressor
Assignee: None
Status: Open
Affected: ["4.2.0"]

This comment was automatically generated by GitHub Actions

pan3793 · 2026-01-22T07:59:49Z

pom.xml

    <postgresql.version>42.7.7</postgresql.version>
-    <db2.jcc.version>11.5.9.0</db2.jcc.version>
+    <!-- A special version that bundles lz4-java 1.10.1 -->
+    <db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>


this dep is only used for testing

pan3793 · 2026-01-22T12:34:41Z

cc @dbtsai @mridulm @dongjoon-hyun @LuciferYang

LuciferYang · 2026-01-22T12:38:25Z

pom.xml

    <postgresql.version>42.7.7</postgresql.version>
-    <db2.jcc.version>11.5.9.0</db2.jcc.version>
+    <!-- A special version that bundles lz4-java 1.10.1 -->
+    <db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>


Can we upgrade it first in a separate pr？

okay, let me upgrade it first

@LuciferYang FYI, I opened SPARK-55136 (#53920) for it

dbtsai · 2026-02-11T06:18:17Z

@viirya can you take a look? Thanks.

viirya · 2026-02-11T07:42:15Z

Hmm, so that's said, in lz4-java 1.10.1, fastDecompressor is paradoxically much slower than safeDecompressor?

pan3793 · 2026-02-11T07:48:11Z

@viirya yeah, it was mentioned in https://sites.google.com/sonatype.com/vulnerabilities/cve-2025-12183

viirya · 2026-02-11T08:23:47Z

core/src/main/scala/org/apache/spark/io/CompressionCodec.scala

+    LZ4BlockInputStream.newBuilder()
+      .withDecompressor(lz4Factory.safeDecompressor())
+      .withChecksum(xxHashFactory.newStreamingHash32(defaultSeed).asChecksum)
+      .withStopOnEmptyBlock(disableConcatenationOfByteStream)


Is this .withStopOnEmptyBlock(disableConcatenationOfByteStream) a correct mapping to previous behavior?

yes, exactly, you can check the source code at

https://github.com/yawkat/lz4-java/blob/33d180cb70c4d93c80fb0dc3ab3002f457e93840/src/java/net/jpountz/lz4/LZ4BlockInputStream.java#L69

public LZ4BlockInputStream(InputStream in, LZ4FastDecompressor fastDecompressor, Checksum checksum, boolean stopOnEmptyBlock) { this(in, fastDecompressor, null, checksum, stopOnEmptyBlock); }

viirya · 2026-02-11T08:25:17Z

pom.xml

+    <!-- A special version that bundles lz4-java 1.10.1 -->
+    <db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>


Does it mean we need to update this dependency when upgrading lz4-java?

@viirya, ideally, we should, but I'm afraid we can't, we can discuss the Db2 JDBC issue in the dedicated PR #53920

Yicong-Huang · 2026-02-26T00:49:00Z

@pan3793 thanks for the great effort! I was not aware of this PR and opened the other one due to performance regression. Do we know what are the current blockers from merging this PR?

cc @dbtsai @ueshin @dongjoon-hyun

pan3793 · 2026-02-26T00:57:33Z

@Yicong-Huang, you can find all related issues/background in the PR description

dongjoon-hyun · 2026-02-26T04:58:21Z

core/benchmarks/LZ4TPCDSDataBenchmark-results.txt

 Compression:                              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-Compression 4 times                                2605           2611           8          0.0   651243236.8       1.0X
+Compression 4 times                                2613           2621          12          0.0   653232793.3       1.0X


~~Although safeDecompressor shows slower performance, but it's in the marginal scope. There might be some improvements at lz4-java 1.10.3.~~

~~Given the above, could you update the PR description?~~

nvm. I forgot that this PR is about Decompression (lz4Factory.safeDecompressor).

dongjoon-hyun · 2026-02-26T05:06:31Z

BTW, I updated the PR description to include the following because the previous benchmark result was generated based on that.

[SPARK-55189][BUILD] Upgrade lz4-java to 1.10.3 #53971

pan3793 · 2026-02-26T05:39:19Z

@dongjoon-hyun, the benchmark results in this PR are generated by lz4-java 1.10.1 a few months ago, I will regenerate that once we resolve the Db2 JDBC driver issue.

dongjoon-hyun · 2026-02-26T05:44:02Z

What I meant (by the following) was that the result in the current master branch. The - part.

the previous benchmark result was generated based on that.

- Decompression 4 times                              2219           2220           1          0.0   554762743.5       1.0X

For the new result, yes. We need to update it, of course.

pan3793 · 2026-02-26T05:47:47Z

I see

…17, Scala 2.13, split 1 of 1)

…21, Scala 2.13, split 1 of 1)

pan3793 · 2026-02-26T11:25:36Z

After SPARK-55706, we don't need to worry about the Db2 JDBC driver issue.

I refreshed the benchmark results, now it's ready for review. @Yicong-Huang @dongjoon-hyun @viirya @dbtsai

Yicong-Huang

LGTM! Many thanks for the great effort!

pan3793 · 2026-02-27T12:28:08Z

thanks all! merged to master

… CVE‐2025‐12183 and CVE-2025-66566 ### What changes were proposed in this pull request? - Bump lz4-java version from 1.8.0 to 1.10.4 to resolve CVE‐2025‐12183 and CVE-2025-66566. - `Lz4Decompressor` follows the [suggestion](apache/spark#53290 (comment)) to move from `fastDecompressor` to `safeDecompressor` to mitigate the performance. Backport: - apache/spark#53327 - apache/spark#53347 - apache/spark#53971 - apache/spark#53454 - apache/spark#54585 ### Why are the changes needed? - [CVE‐2025‐12183](https://sites.google.com/sonatype.com/vulnerabilities/cve-2025-12183): Various lz4-java compression and decompression implementations do not guard against out-of-bounds memory access. Untrusted input may lead to denial of service and information disclosure. Vulnerable Maven coordinates: org.lz4:lz4-java up to and including 1.8.0. - [CVE-2025-66566](GHSA-cmp6-m4wj-q63q): Insufficient clearing of the output buffer in Java-based decompressor implementations in lz4-java 1.10.0 and earlier allows remote attackers to read previous buffer contents via crafted compressed input. In applications where the output buffer is reused without being cleared, this may lead to disclosure of sensitive data. JNI-based implementations are not affected. Therefore, lz4-java version should upgrade to 1.10.4. ### Does this PR resolve a correctness bug? No. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI. Closes #3555 from SteNicholas/CELEBORN-2218. Lead-authored-by: SteNicholas <programgeek@163.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: SteNicholas <programgeek@163.com>

… CVE‐2025‐12183 and CVE-2025-66566 - Bump lz4-java version from 1.8.0 to 1.10.4 to resolve CVE‐2025‐12183 and CVE-2025-66566. - `Lz4Decompressor` follows the [suggestion](apache/spark#53290 (comment)) to move from `fastDecompressor` to `safeDecompressor` to mitigate the performance. Backport: - apache/spark#53327 - apache/spark#53347 - apache/spark#53971 - apache/spark#53454 - apache/spark#54585 - [CVE‐2025‐12183](https://sites.google.com/sonatype.com/vulnerabilities/cve-2025-12183): Various lz4-java compression and decompression implementations do not guard against out-of-bounds memory access. Untrusted input may lead to denial of service and information disclosure. Vulnerable Maven coordinates: org.lz4:lz4-java up to and including 1.8.0. - [CVE-2025-66566](GHSA-cmp6-m4wj-q63q): Insufficient clearing of the output buffer in Java-based decompressor implementations in lz4-java 1.10.0 and earlier allows remote attackers to read previous buffer contents via crafted compressed input. In applications where the output buffer is reused without being cleared, this may lead to disclosure of sensitive data. JNI-based implementations are not affected. Therefore, lz4-java version should upgrade to 1.10.4. No. No. CI. Closes #3555 from SteNicholas/CELEBORN-2218. Lead-authored-by: SteNicholas <programgeek@163.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: SteNicholas <programgeek@163.com> (cherry picked from commit dca3749) Signed-off-by: SteNicholas <programgeek@163.com>

github-actions bot added the CORE label Dec 12, 2025

This was referenced Dec 12, 2025

[SPARK-54693][CORE][TESTS] Add LZ4TPCDSDataBenchmark #53453

Closed

[WIP][SPARK-54571][CORE] Use LZ4 safeDecompressor #53290

Closed

pan3793 changed the title ~~WIP [SPARK-54571] Use safeDecompressor~~ WIP [SPARK-54571] Use LZ4 safeDecompressor Dec 12, 2025

github-actions bot added the SQL label Dec 17, 2025

pan3793 force-pushed the SPARK-54571 branch from 8d20d9f to 6bf027c Compare December 17, 2025 07:11

pan3793 force-pushed the SPARK-54571 branch from c91c457 to 101a1ca Compare January 22, 2026 06:30

pan3793 changed the title ~~WIP [SPARK-54571] Use LZ4 safeDecompressor~~ [WIP][SPARK-54571] Use LZ4 safeDecompressor Jan 22, 2026

github-actions bot added the BUILD label Jan 22, 2026

pan3793 changed the title ~~[WIP][SPARK-54571] Use LZ4 safeDecompressor~~ [SPARK-54571] Use LZ4 safeDecompressor Jan 22, 2026

pan3793 changed the title ~~[SPARK-54571] Use LZ4 safeDecompressor~~ [SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor Jan 22, 2026

pan3793 commented Jan 22, 2026

View reviewed changes

pan3793 marked this pull request as ready for review January 22, 2026 08:00

pan3793 force-pushed the SPARK-54571 branch from a04e58b to d293139 Compare January 22, 2026 08:04

LuciferYang reviewed Jan 22, 2026

View reviewed changes

pan3793 mentioned this pull request Jan 22, 2026

[SPARK-55136][BUILD] Bump Db2 JDBC driver 12.1.3.0_special_74723 #53920

Closed

pan3793 requested a review from LuciferYang January 23, 2026 03:10

viirya reviewed Feb 11, 2026

View reviewed changes

pan3793 mentioned this pull request Feb 26, 2026

[SPARK-54571][CORE] Use LZ4 safeDecompressor #54496

Closed

pan3793 force-pushed the SPARK-54571 branch from d293139 to 63d6c50 Compare February 26, 2026 01:53

dongjoon-hyun reviewed Feb 26, 2026

View reviewed changes

pan3793 force-pushed the SPARK-54571 branch from 63d6c50 to e3518b8 Compare February 26, 2026 07:19

pan3793 mentioned this pull request Feb 26, 2026

[SPARK-55706][SQL][TESTS] Disable DB2 JDBC Driver tests #54505

Closed

pan3793 force-pushed the SPARK-54571 branch 3 times, most recently from d9e4e55 to 453e2ca Compare February 26, 2026 10:19

pan3793 and others added 4 commits February 26, 2026 19:19

safeDecompressor

67d63f7

Switch to LZ4 safeDecompressor in Statistics

7369326

Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …

def6e39

…17, Scala 2.13, split 1 of 1)

Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …

c051508

…21, Scala 2.13, split 1 of 1)

pan3793 force-pushed the SPARK-54571 branch from bbfc3c5 to c051508 Compare February 26, 2026 11:20

pan3793 changed the title ~~[SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor~~ [SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor to mitigate perf regression Feb 26, 2026

inline

0d1bde0

Yicong-Huang approved these changes Feb 26, 2026

View reviewed changes

dongjoon-hyun approved these changes Feb 26, 2026

View reviewed changes

viirya approved these changes Feb 26, 2026

View reviewed changes

LuciferYang approved these changes Feb 27, 2026

View reviewed changes

pan3793 closed this in dc57455 Feb 27, 2026

pan3793 deleted the SPARK-54571 branch February 27, 2026 12:41

SteNicholas mentioned this pull request Feb 28, 2026

[CELEBORN-2218] Bump lz4-java version from 1.8.0 to 1.10.4 to resolve CVE‐2025‐12183 and CVE-2025-66566 apache/celeborn#3555

Closed

		<!-- A special version that bundles lz4-java 1.10.1 -->
		<db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>

Conversation

pan3793 commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793 commented Dec 14, 2025

Uh oh!

pan3793 commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 22, 2026

JIRA Issue Information

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbtsai commented Feb 11, 2026

Uh oh!

viirya commented Feb 11, 2026

Uh oh!

pan3793 commented Feb 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yicong-Huang commented Feb 26, 2026

Uh oh!

pan3793 commented Feb 26, 2026

Uh oh!

dongjoon-hyun Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Feb 26, 2026

Uh oh!

dongjoon-hyun commented Feb 26, 2026

Uh oh!

pan3793 commented Feb 26, 2026

Uh oh!

pan3793 commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yicong-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pan3793 commented Dec 12, 2025 •

edited

Loading

pan3793 commented Dec 15, 2025 •

edited

Loading

dongjoon-hyun Feb 26, 2026 •

edited

Loading

dongjoon-hyun commented Feb 26, 2026 •

edited

Loading

pan3793 commented Feb 26, 2026 •

edited

Loading