Skip to content

Conversation

@SteNicholas
Copy link
Member

What changes were proposed in this pull request?

Support Spark 4.1.

Why are the changes needed?

Spark 4.1.0 has already released, which release notes refer to Spark Release 4.1.0.

Does this PR resolve a correctness bug?

No.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

@SteNicholas
Copy link
Member Author

Ping @pan3793, @turboFei.

metrics-jvm/4.2.25//metrics-jvm-4.2.25.jar
netty-all/4.1.118.Final//netty-all-4.1.118.Final.jar
netty-buffer/4.1.118.Final//netty-buffer-4.1.118.Final.jar
netty-buffer/4.2.7.Final//netty-buffer-4.2.7.Final.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two versions of netty co-exist does not make sense

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pan3793, I have excluded netty dependencies of spark-sql, which produces 4.2.7.Final version netty dependencies:

[INFO] org.apache.celeborn:celeborn-client-spark-3_2.13:jar:0.7.0-SNAPSHOT
[INFO] +- org.apache.celeborn:celeborn-common_2.13:jar:0.7.0-SNAPSHOT:compile
[INFO] |  +- org.apache.celeborn:celeborn-spi:jar:0.7.0-SNAPSHOT:compile
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:4.2.25:compile
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:4.2.25:compile
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:4.2.25:compile
[INFO] |  +- org.yaml:snakeyaml:jar:2.2:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.10.5:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.36:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.36:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.36:compile
[INFO] |  +- commons-io:commons-io:jar:2.17.0:compile
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.17.0:compile
[INFO] |  +- io.netty:netty-all:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-haproxy:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-memcache:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-mqtt:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-redis:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-smtp:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-stomp:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-xml:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-transport-rxtx:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-transport-sctp:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-transport-udt:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.118.Final:runtime
[INFO] |  |  \- io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.118.Final:runtime
[INFO] |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- com.google.guava:guava:jar:33.1.0-jre:compile
[INFO] |  |  \- com.google.guava:failureaccess:jar:1.0.2:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.25.5:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.13.17:compile
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.13.17:compile
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:1.0.6:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.15.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.3:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.15.3:compile
[INFO] +- org.apache.celeborn:celeborn-client_2.13:jar:0.7.0-SNAPSHOT:compile
[INFO] |  +- org.lz4:lz4-java:jar:1.8.0:compile
[INFO] |  \- com.github.luben:zstd-jni:jar:1.5.7-6:compile
[INFO] +- org.apache.celeborn:celeborn-client-spark-common_2.13:jar:0.7.0-SNAPSHOT:compile
[INFO] +- org.apache.spark:spark-core_2.13:jar:4.1.0:provided
[INFO] |  +- org.scala-lang.modules:scala-parallel-collections_2.13:jar:1.2.0:provided
[INFO] |  +- org.apache.avro:avro:jar:1.12.1:provided
[INFO] |  +- org.apache.avro:avro-mapred:jar:1.12.1:provided
[INFO] |  +- com.twitter:chill_2.13:jar:0.10.0:provided
[INFO] |  +- com.twitter:chill-java:jar:0.10.0:provided
[INFO] |  +- com.esotericsoftware:kryo-shaded:jar:4.0.3:provided
[INFO] |  +- org.objenesis:objenesis:jar:3.4:provided
[INFO] |  +- org.apache.spark:spark-launcher_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-kvstore_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-common_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-unsafe_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-common-utils_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.curator:curator-recipes:jar:5.9.0:provided
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.9.4:provided
[INFO] |  +- io.jsonwebtoken:jjwt-api:jar:0.12.6:provided
[INFO] |  +- io.jsonwebtoken:jjwt-impl:jar:0.12.6:provided
[INFO] |  +- io.jsonwebtoken:jjwt-jackson:jar:0.12.6:provided
[INFO] |  +- commons-codec:commons-codec:jar:1.19.0:provided
[INFO] |  +- org.apache.commons:commons-compress:jar:1.28.0:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.6.1:provided
[INFO] |  +- org.apache.commons:commons-text:jar:1.14.0:provided
[INFO] |  +- org.apache.commons:commons-collections4:jar:4.5.0:provided
[INFO] |  +- com.ning:compress-lzf:jar:1.1.2:provided
[INFO] |  +- org.scala-lang.modules:scala-xml_2.13:jar:2.4.0:provided
[INFO] |  +- org.json4s:json4s-jackson_2.13:jar:4.0.7:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:3.0.18:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:3.0.18:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.47:provided
[INFO] |  |  \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:3.0.18:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.47:provided
[INFO] |  +- org.glassfish.jersey.inject:jersey-hk2:jar:2.47:provided
[INFO] |  +- com.clearspring.analytics:stream:jar:2.9.8:provided
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:4.2.37:provided
[INFO] |  +- io.dropwizard.metrics:metrics-jmx:jar:4.2.37:provided
[INFO] |  +- net.razorvine:pickle:jar:1.5:provided
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.9.9:provided
[INFO] |  \- org.apache.spark:spark-tags_2.13:jar:4.1.0:provided
[INFO] +- org.apache.spark:spark-sql_2.13:jar:4.1.0:provided
[INFO] |  +- org.rocksdb:rocksdbjni:jar:9.10.0:provided
[INFO] |  +- com.univocity:univocity-parsers:jar:2.9.1:provided
[INFO] |  +- org.apache.spark:spark-sketch_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-common-utils-java_2.13:jar:4.1.0:provided
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.8.3:compile
[INFO] |  +- org.apache.ivy:ivy:jar:2.5.3:provided
[INFO] |  +- oro:oro:jar:2.0.8:provided
[INFO] |  +- org.apache.logging.log4j:log4j-slf4j2-impl:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-api:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-core:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-1.2-api:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-layout-template-json:jar:2.24.3:provided
[INFO] |  +- org.apache.avro:avro-ipc:jar:1.12.1:provided
[INFO] |  +- org.tukaani:xz:jar:1.10:provided
[INFO] |  +- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-classes:jar:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.74.Final:provided
[INFO] |  +- com.google.crypto.tink:tink:jar:1.16.0:provided
[INFO] |  +- com.google.code.gson:gson:jar:2.11.0:provided
[INFO] |  +- org.apache.curator:curator-framework:jar:5.9.0:provided
[INFO] |  +- org.apache.curator:curator-client:jar:5.9.0:provided
[INFO] |  +- org.apache.zookeeper:zookeeper-jute:jar:3.9.4:provided
[INFO] |  +- org.apache.yetus:audience-annotations:jar:0.12.0:provided
[INFO] |  +- org.json4s:json4s-core_2.13:jar:4.0.7:provided
[INFO] |  +- org.json4s:json4s-ast_2.13:jar:4.0.7:provided
[INFO] |  +- org.json4s:json4s-scalap_2.13:jar:4.0.7:provided
[INFO] |  +- org.json4s:json4s-jackson-core_2.13:jar:4.0.7:provided
[INFO] |  +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:provided
[INFO] |  +- jakarta.inject:jakarta.inject-api:jar:2.0.1:provided
[INFO] |  +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
[INFO] |  +- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
[INFO] |  +- jakarta.validation:jakarta.validation-api:jar:3.0.2:provided
[INFO] |  +- org.glassfish.hk2:hk2-locator:jar:3.0.6:provided
[INFO] |  +- org.glassfish.hk2.external:aopalliance-repackaged:jar:3.0.6:provided
[INFO] |  +- org.glassfish.hk2:hk2-api:jar:3.0.6:provided
[INFO] |  +- org.glassfish.hk2:hk2-utils:jar:3.0.6:provided
[INFO] |  +- org.javassist:javassist:jar:3.30.2-GA:provided
[INFO] |  +- io.netty:netty-buffer:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-base:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-dns:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-compression:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-http:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-http3:jar:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-unix-common:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-socks:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-protobuf:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-marshalling:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-common:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-handler:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-handler-proxy:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-resolver:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-resolver-dns:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-epoll:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-kqueue:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-io_uring:jar:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-classes-quic:jar:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-riscv64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-io_uring:jar:linux-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-io_uring:jar:linux-aarch_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-io_uring:jar:linux-riscv64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:linux-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:linux-aarch_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:osx-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:osx-aarch_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:windows-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.2.7.Final:runtime
[INFO] |  +- org.apache.spark:spark-catalyst_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-sql-api_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-variant_2.13:jar:4.1.0:provided
[INFO] |  +- org.codehaus.janino:janino:jar:3.1.9:provided
[INFO] |  +- org.codehaus.janino:commons-compiler:jar:3.1.9:provided
[INFO] |  +- org.apache.datasketches:datasketches-java:jar:6.2.0:provided
[INFO] |  +- org.apache.datasketches:datasketches-memory:jar:3.0.2:provided
[INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.13:jar:2.4.0:provided
[INFO] |  +- org.antlr:antlr4-runtime:jar:4.13.1:provided
[INFO] |  +- org.apache.arrow:arrow-vector:jar:18.3.0:provided
[INFO] |  +- org.apache.arrow:arrow-format:jar:18.3.0:provided
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.3:provided
[INFO] |  +- com.google.flatbuffers:flatbuffers-java:jar:25.2.10:provided
[INFO] |  +- org.apache.arrow:arrow-memory-netty:jar:18.3.0:provided
[INFO] |  +- org.apache.arrow:arrow-memory-netty-buffer-patch:jar:18.3.0:provided
[INFO] |  +- org.apache.orc:orc-format:jar:shaded-protobuf:1.1.1:provided
[INFO] |  +- org.apache.orc:orc-core:jar:shaded-protobuf:2.2.1:provided
[INFO] |  +- org.apache.orc:orc-shims:jar:2.2.1:provided
[INFO] |  +- io.airlift:aircompressor:jar:2.0.2:provided
[INFO] |  +- org.threeten:threeten-extra:jar:1.8.0:provided
[INFO] |  +- org.locationtech.jts:jts-core:jar:1.20.0:provided
[INFO] |  +- org.apache.orc:orc-mapreduce:jar:shaded-protobuf:2.2.1:provided
[INFO] |  +- org.apache.hive:hive-storage-api:jar:2.8.1:provided
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-common:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-encoding:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-format-structures:jar:1.16.0:provided
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:provided
[INFO] |  +- org.apache.parquet:parquet-hadoop:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-jackson:jar:1.16.0:provided
[INFO] |  +- org.apache.ws.xmlschema:xmlschema-core:jar:2.3.1:provided
[INFO] |  +- org.apache.xbean:xbean-asm9-shaded:jar:4.28:provided
[INFO] |  +- com.ibm.icu:icu4j:jar:77.1:provided
[INFO] |  +- org.apache.arrow:arrow-compression:jar:18.3.0:provided
[INFO] |  \- org.apache.arrow:arrow-memory-core:jar:18.3.0:provided
[INFO] +- org.apache.celeborn:celeborn-client_2.13:test-jar:tests:0.7.0-SNAPSHOT:test
[INFO] +- org.apache.spark:spark-core_2.13:test-jar:tests:4.1.0:test
[INFO] +- javax.servlet:javax.servlet-api:jar:4.0.1:test
[INFO] +- jakarta.servlet:jakarta.servlet-api:jar:5.0.0:test
[INFO] +- org.mockito:mockito-core:jar:4.11.0:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.12.19:test
[INFO] |  \- net.bytebuddy:byte-buddy-agent:jar:1.12.19:test
[INFO] +- org.mockito:mockito-inline:jar:4.11.0:test
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.scalatest:scalatest_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-core_2.13:jar:3.2.16:test
[INFO] |  |  +- org.scalatest:scalatest-compatible:jar:3.2.16:test
[INFO] |  |  \- org.scalactic:scalactic_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-featurespec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-flatspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-freespec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-funsuite_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-funspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-propspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-refspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-wordspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-diagrams_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-matchers-core_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-shouldmatchers_2.13:jar:3.2.16:test
[INFO] |  \- org.scalatest:scalatest-mustmatchers_2.13:jar:3.2.16:test
[INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
[INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.3:runtime
[INFO] \- com.github.olivergondza:maven-jdk-tools-wrapper:jar:0.1:compile
[INFO]    \- jdk.tools:jdk.tools:jar:1.6:system

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SteNicholas don't harry to mark unfinished conversations as resolved.

I didn't find where your claimed change is, I also don't expect this work.

I have excluded netty dependencies of spark-sql

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants