Skip to content

[FLINK-38523] Update Pekko to 1.4.0 #27368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mateczagany wants to merge 9 commits into
base: master
Choose a base branch
from
+249 −130
Draft

[FLINK-38523] Update Pekko to 1.4.0 #27368

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bb2c38f
[FLINK-38523] Update Pekko to 1.4.0
mateczagany Dec 26, 2025
35d951e
[FLINK-38523] Remove Netty from flink-rpc-akka shaded JAR
mateczagany Dec 31, 2025
bb5a288
[FLINK-38523] Fix Netty transitive dependency of Arrow
mateczagany Jan 1, 2026
59457bb
[FLINK-38523] Fix ExecNodeBase JsonProperty of type field
mateczagany Jan 1, 2026
81ceb74
[FLINK-38523] Remove deprecated Netty calls
mateczagany Jan 1, 2026
05fde20
[FLINK-38523] Update NettyConnectionManagerTest
mateczagany Jan 1, 2026
d2088ba
[FLINK-38523] Fix hostname verification configuration
mateczagany Jan 2, 2026
b65099b
[FLINK-38523] REST handlers responses should be written from EventLoo…
mateczagany Jan 6, 2026
0a4138e
[FLINK-38523] Update rest_api_v1.snapshot
mateczagany Jan 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/layouts/shortcodes/generated/security_configuration.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@
</tr>
<tr>
<td><h5>security.ssl.verify-hostname</h5></td>
<td style="word-wrap: break-word;">true</td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>Flag to enable peer’s hostname verification during ssl handshake.</td>
</tr>
Expand Down
2 changes: 1 addition & 1 deletion docs/layouts/shortcodes/generated/security_ssl_section.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@
</tr>
<tr>
<td><h5>security.ssl.verify-hostname</h5></td>
<td style="word-wrap: break-word;">true</td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>Flag to enable peer’s hostname verification during ssl handshake.</td>
</tr>
Expand Down
2 changes: 1 addition & 1 deletion flink-core/src/main/java/org/apache/flink/configuration/SecurityOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,7 @@ public static Configuration forProvider(Configuration configuration, String prov
public static final ConfigOption<Boolean> SSL_VERIFY_HOSTNAME =
key("security.ssl.verify-hostname")
.booleanType()
.defaultValue(true)
.defaultValue(false)
Copy link
Contributor Author

@mateczagany mateczagany Jan 2, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This configuration was not used before for a long time, I believe since this commit: 30c4bc8#diff-659981ee1b5263cef2478bd413c1e4b5122c9a636ef76bc0d4aafc9633274644

Before this upgrade, Netty disabled hostname verification by default, but since 4.2.x, it's enabled by default, according to the official migration guide.

I've added the necessary code in SSLUtils to check for this config option when setting up SSLContext, but I felt like it's a good idea to disable hostname verification by default with this change, as that was the behavior before this PR as well.

.withDescription(
"Flag to enable peer’s hostname verification during ssl handshake.");

Expand Down
17 changes: 0 additions & 17 deletions flink-kubernetes/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ under the License.

<properties>
<kubernetes.client.version>7.3.1</kubernetes.client.version>
<netty.override.version>4.1.108.Final</netty.override.version>
<surefire.module.config><!--
CommonTestUtils#setEnv
-->--add-opens=java.base/java.util=ALL-UNNAMED
Expand Down Expand Up @@ -109,22 +108,6 @@ under the License.

</dependencies>

<dependencyManagement>
<dependencies>
<!--
This should be removed once Netty is upgraded to at least 4.1.108.Final.
Important to note that this only affects test scope.
Version 4.1.100.Final does not contain a class that's required by Vert.x, which is used by kubernetes-server-mock.
-->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http</artifactId>
<version>${netty.override.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</dependencyManagement>

<build>
<plugins>
<plugin>
Expand Down
8 changes: 8 additions & 0 deletions flink-python/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,14 @@ under the License.
<version>2.5.1</version>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-bom</artifactId>
<version>4.1.100.Final</version>
<type>pom</type>
<scope>import</scope>
</dependency>
Comment on lines +423 to +428
Copy link
Contributor

@snuyanzin snuyanzin Jan 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably I missed it during first look

can we also bump netty here?

Copy link
Contributor Author

@mateczagany mateczagany Jan 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might not have been there when you've had the first look, I'm still running into issues on the Azure CI.

We currently use Apache Arrow 13.0.0 which was built against Netty 4.1.96. The newest Apache Arrow version is 18.3.0, that uses Netty 4.1.119.

I'll play with this, but we most probably won't be able to use Netty 4.2.x here.

</dependencies>
</dependencyManagement>

Expand Down
10 changes: 0 additions & 10 deletions flink-python/src/main/resources/META-INF/NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,17 +58,7 @@ The bundled Apache Beam dependencies bundle the following dependencies under the
- io.grpc:grpc-stub:1.59.1
- io.grpc:grpc-testing:1.59.1
- io.netty:netty-buffer:4.1.100.Final
- io.netty:netty-codec:4.1.100.Final
- io.netty:netty-codec-http:4.1.100.Final
- io.netty:netty-codec-http2:4.1.100.Final
- io.netty:netty-codec-socks:4.1.100.Final
- io.netty:netty-common:4.1.100.Final
- io.netty:netty-handler:4.1.100.Final
- io.netty:netty-handler-proxy:4.1.100.Final
- io.netty:netty-resolver:4.1.100.Final
- io.netty:netty-transport:4.1.100.Final
- io.netty:netty-transport-native-epoll:4.1.100.Final:linux-x86_64
- io.netty:netty-transport-native-unix-common:4.1.100.Final
- io.opencensus:opencensus-api:0.31.0
- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0
- io.perfmark:perfmark-api:0.26.0
Expand Down
8 changes: 5 additions & 3 deletions ...client-java/src/main/java/org/apache/flink/queryablestate/network/AbstractServerBase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInitializer;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelOption;
import org.apache.flink.shaded.netty4.io.netty.channel.EventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.MultiThreadIoEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioIoHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.SocketChannel;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.LengthFieldBasedFrameDecoder;
Expand Down Expand Up @@ -236,8 +237,9 @@ private boolean attemptToBind(final int port) throws Throwable {
.setNameFormat("Flink " + serverName + " EventLoop Thread %d")
.build();

final NioEventLoopGroup nioGroup =
new NioEventLoopGroup(numEventLoopThreads, threadFactory);
final MultiThreadIoEventLoopGroup nioGroup =
new MultiThreadIoEventLoopGroup(
numEventLoopThreads, threadFactory, NioIoHandler.newFactory());

this.bootstrap =
new ServerBootstrap()
Expand Down
8 changes: 6 additions & 2 deletions ...yable-state-client-java/src/main/java/org/apache/flink/queryablestate/network/Client.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInitializer;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelOption;
import org.apache.flink.shaded.netty4.io.netty.channel.EventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.MultiThreadIoEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioIoHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.SocketChannel;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.LengthFieldBasedFrameDecoder;
Expand Down Expand Up @@ -110,7 +111,10 @@ public Client(
.setNameFormat("Flink " + clientName + " Event Loop Thread %d")
.build();

final EventLoopGroup nioGroup = new NioEventLoopGroup(numEventLoopThreads, threadFactory);
final MultiThreadIoEventLoopGroup nioGroup =
new MultiThreadIoEventLoopGroup(
numEventLoopThreads, threadFactory, NioIoHandler.newFactory());

final ByteBufAllocator bufferPool = new NettyBufferPool(numEventLoopThreads);

this.bootstrap =
Expand Down
7 changes: 4 additions & 3 deletions ...yable-state-runtime/src/test/java/org/apache/flink/queryablestate/network/ClientTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerContext;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInitializer;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.MultiThreadIoEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioIoHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.SocketChannel;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.LengthFieldBasedFrameDecoder;
Expand Down Expand Up @@ -93,11 +94,11 @@ class ClientTest {
private static final Logger LOG = LoggerFactory.getLogger(ClientTest.class);

// Thread pool for client bootstrap (shared between tests)
private NioEventLoopGroup nioGroup;
private MultiThreadIoEventLoopGroup nioGroup;

@BeforeEach
void setUp() {
nioGroup = new NioEventLoopGroup();
nioGroup = new MultiThreadIoEventLoopGroup(NioIoHandler.newFactory());
}

@AfterEach
Expand Down
6 changes: 4 additions & 2 deletions ...tate-runtime/src/test/java/org/apache/flink/queryablestate/network/KvStateServerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInitializer;
import org.apache.flink.shaded.netty4.io.netty.channel.EventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.MultiThreadIoEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioIoHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.SocketChannel;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.LengthFieldBasedFrameDecoder;
Expand All @@ -74,7 +75,8 @@
class KvStateServerTest {

// Thread pool for client bootstrap (shared between tests)
private static final NioEventLoopGroup NIO_GROUP = new NioEventLoopGroup();
private static final MultiThreadIoEventLoopGroup NIO_GROUP =
new MultiThreadIoEventLoopGroup(NioIoHandler.newFactory());

private static final int TIMEOUT_MILLIS = 10000;

Expand Down
9 changes: 2 additions & 7 deletions flink-rpc/flink-rpc-akka/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ under the License.
</description>

<properties>
<pekko.version>1.1.2</pekko.version>
<pekko.version>1.4.0</pekko.version>
</properties>

<dependencies>
Expand Down Expand Up @@ -180,12 +180,7 @@ under the License.
<filter>
<artifact>io.netty:*</artifact>
<excludes>
<!-- Only some of these licenses actually apply to the JAR and have been manually
placed in this module's resources directory. -->
<exclude>META-INF/license/**</exclude>
<!-- Only parts of NOTICE file actually apply to the netty JAR and have been manually
copied into this modules's NOTICE file. -->
<exclude>META-INF/NOTICE.txt</exclude>
<exclude>**/*</exclude>
</excludes>
</filter>
<filter>
Expand Down
39 changes: 32 additions & 7 deletions flink-rpc/flink-rpc-akka/src/main/resources/META-INF/NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,39 @@ This project bundles the following dependencies under the Apache Software Licens
- com.hierynomus:asn-one:0.6.0
- com.typesafe:config:1.4.2
- com.typesafe:ssl-config-core_2.12:0.6.1
- io.netty:netty-all:4.1.100.Final
- org.agrona:agrona:1.22.0
- org.apache.pekko:pekko-actor_2.12:1.1.2
- org.apache.pekko:pekko-remote_2.12:1.1.2
- org.apache.pekko:pekko-pki_2.12:1.1.2
- org.apache.pekko:pekko-protobuf-v3_2.12:1.1.2
- org.apache.pekko:pekko-slf4j_2.12:1.1.2
- org.apache.pekko:pekko-stream_2.12:1.1.2
- org.apache.pekko:pekko-actor_2.12:1.4.0
- org.apache.pekko:pekko-remote_2.12:1.4.0
- org.apache.pekko:pekko-pki_2.12:1.4.0
- org.apache.pekko:pekko-protobuf-v3_2.12:1.4.0
- org.apache.pekko:pekko-slf4j_2.12:1.4.0
- org.apache.pekko:pekko-stream_2.12:1.4.0
- com.typesafe:ssl-config-core_2.12:0.7.1
- io.netty:netty-buffer:4.2.6.Final
- io.netty:netty-transport-native-unix-common:4.2.6.Final
- io.netty:netty-common:4.2.6.Final
- io.netty:netty-transport:4.2.6.Final
- io.netty:netty-transport-classes-epoll:4.2.6.Final
- io.netty:netty-transport-classes-kqueue:4.2.6.Final
- io.netty:netty-resolver-dns-classes-macos:4.2.6.Final
- io.netty:netty-transport-classes-io_uring:4.2.6.Final
- io.netty:netty-codec-classes-quic:4.2.6.Final
- io.netty:netty-transport-native-epoll:linux-x86_64:4.2.6.Final
- io.netty:netty-transport-native-epoll:linux-aarch_64:4.2.6.Final
- io.netty:netty-transport-native-epoll:linux-riscv64:4.2.6.Final
- io.netty:netty-transport-native-io_uring:linux-x86_64:4.2.6.Final
- io.netty:netty-transport-native-io_uring:linux-aarch_64:4.2.6.Final
- io.netty:netty-transport-native-io_uring:linux-riscv64:4.2.6.Final
- io.netty:netty-transport-native-kqueue:osx-x86_64:4.2.6.Final
- io.netty:netty-transport-native-kqueue:osx-aarch_64:4.2.6.Final
- io.netty:netty-resolver-dns-native-macos:osx-x86_64:4.2.6.Final
- io.netty:netty-resolver-dns-native-macos:osx-aarch_64:4.2.6.Final
- io.netty:netty-codec-native-quic:linux-x86_64:4.2.6.Final
- io.netty:netty-codec-native-quic:linux-aarch_64:4.2.6.Final
- io.netty:netty-codec-native-quic:osx-x86_64:4.2.6.Final
- io.netty:netty-codec-native-quic:osx-aarch_64:4.2.6.Final
- io.netty:netty-codec-native-quic:windows-x86_64:4.2.6.Final


The following dependencies all share the same BSD license which you find under licenses/LICENSE.scala.

Expand Down
9 changes: 6 additions & 3 deletions ...ime-web/src/main/java/org/apache/flink/runtime/webmonitor/utils/WebFrontendBootstrap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelFuture;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInitializer;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.MultiThreadIoEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioIoHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.SocketChannel;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpServerCodec;
Expand Down Expand Up @@ -147,8 +148,10 @@ protected void initChannel(SocketChannel ch) throws ConfigurationException {
}
};

NioEventLoopGroup bossGroup = new NioEventLoopGroup(1);
NioEventLoopGroup workerGroup = new NioEventLoopGroup();
MultiThreadIoEventLoopGroup bossGroup =
new MultiThreadIoEventLoopGroup(1, NioIoHandler.newFactory());
MultiThreadIoEventLoopGroup workerGroup =
new MultiThreadIoEventLoopGroup(NioIoHandler.newFactory());

this.bootstrap = new ServerBootstrap();
this.bootstrap
Expand Down
5 changes: 3 additions & 2 deletions ...ntime-web/src/test/java/org/apache/flink/runtime/webmonitor/testutils/HttpTestClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelInitializer;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelPipeline;
import org.apache.flink.shaded.netty4.io.netty.channel.EventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.MultiThreadIoEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup;
import org.apache.flink.shaded.netty4.io.netty.channel.nio.NioIoHandler;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.SocketChannel;
import org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.DefaultFullHttpRequest;
Expand Down Expand Up @@ -102,7 +103,7 @@ public HttpTestClient(String host, int port) {
this.host = host;
this.port = port;

this.group = new NioEventLoopGroup();
this.group = new MultiThreadIoEventLoopGroup(NioIoHandler.newFactory());

this.bootstrap = new Bootstrap();
this.bootstrap
Expand Down
Loading