When using netty 4.2.8.Final and higher - Will throw error for encoded paths

[nischit7]

Describe the bug

The issue is particularly found when one tends to use Netty 4.2.8.Final and higher.
A new change has been added in netty
https://github.com/netty/netty/blob/4.2/codec-http/src/main/java/io/netty/handler/codec/http/DefaultHttpRequest.java#L94

Specifically in DefaultHttpRequest constructor.
It tries to check for invalid characters in the http path.

When using AWS Bedrock SDK, modelId is sent as a path param. The model Id ARN contains colons. AWS SDK by default tries to encode the path.

The issue starts when RequestAdaptor https://github.com/netty/netty/blob/4.2/codec-http/src/main/java/io/netty/handler/codec/http/DefaultHttpRequest.java#L94
initializes Netty object as below -

DefaultHttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, method, uri, headers);

This constructor has been in use for many years.
But the new change in Netty lib now enforces the below code flow within DefaultHttpRequest

if (validateRequestLine) {
    HttpUtil.validateRequestLineTokens(method, uri);
}

Since Bedrock Model ARN is encoded and is part of the path, it ends up throwing an error - "IllegalArgumentException; The URI contain illegal characters".

We had to downgrade netty lib to 4.2.7.final to get around the issue. A temporary approach.
However a fix is needed in AWS SDK to either the call the new constructor of DefaultHttpRequest to not validate the url or fix the encoding logic.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The AWS Bedrock streaming should pass without any such client side issues.

Current Behavior

The client side ends up getting an error thrown by Netty as below -
"IllegalArgumentException; The URI contain illegal characters".

Reproduction Steps

Reproducing is easy.
Use AWS Bedrock streaming.
Use Netty 4.2.8.Final version.
Initiate a call for any model ARN.
It will fail

Possible Solution

Solution suggested in the description

Additional Information/Context

No response

AWS Java SDK version used

2.30.6

JDK version used

21

Operating System and version

Linux

[bhoradc]

Hello @nischit7

Thank you for reporting this issue. Netty 4.2.8.Final contained a bit mask validation bug that incorrectly rejected uppercase 'M' as an illegal character. But this appears to be fixed in Netty 4.2.9.Final.

Below is the sample code I used to validate that character M was throwing the IllegalArgumentException with Netty 4.2.8.Final. Netty version 4.2.9.Final works fine.

Sample code

import software.amazon.awssdk.core.SdkBytes;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeAsyncClient;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamRequest;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamResponseHandler;

public class BedrockSDKTest {

    public static void main(String[] args) {
        BedrockRuntimeAsyncClient client = BedrockRuntimeAsyncClient.builder()
                .region(Region.US_EAST_1)
                .httpClient(NettyNioAsyncHttpClient.builder().build())
                .build();

        InvokeModelWithResponseStreamResponseHandler handler = 
            InvokeModelWithResponseStreamResponseHandler.builder()
                .subscriber(event -> {})
                .build();

        client.invokeModelWithResponseStream(
                InvokeModelWithResponseStreamRequest.builder()
                    .modelId("arn:aws:bedrock:us-east-1:123456789012:provisioned-Model/test")
                    .body(SdkBytes.fromUtf8String("{}"))
                    .build(), 
                handler).join();

        client.close();
    }
}

Underlying exception

Caused by: java.lang.IllegalArgumentException: The URI contain illegal characters: /model/arn%3Aaws%3Abedrock%3Aus-east-1%3A123456789012%3Aprovisioned-Model%2Ftest/invoke-with-response-stream
	at io.netty.handler.codec.http.HttpUtil.validateRequestLineTokens(HttpUtil.java:90)
	at io.netty.handler.codec.http.DefaultHttpRequest.<init>(DefaultHttpRequest.java:95)
	at io.netty.handler.codec.http.DefaultHttpRequest.<init>(DefaultHttpRequest.java:78)
	at software.amazon.awssdk.http.nio.netty.internal.RequestAdapter.adapt(RequestAdapter.java:55)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequest(NettyRequestExecutor.java:236)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$makeRequestListener$10(NettyRequestExecutor.java:181)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
	at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	... 1 more

Note that latest Java SDK v2.41.9 still uses Netty 4.1.130.Final which already seems to have this issue fixed though. Kindly validate at your end or provide a reproducible code sample in case you think ARNs containing colons are a problem.

Regards,
Chaitanya

[nischit7]

Hello @nischit7

Thank you for reporting this issue. Netty 4.2.8.Final contained a bit mask validation bug that incorrectly rejected uppercase 'M' as an illegal character. But this appears to be fixed in Netty 4.2.9.Final.

Below is the sample code I used to validate that character M was throwing the IllegalArgumentException with Netty 4.2.8.Final. Netty version 4.2.9.Final works fine.

Sample code
import software.amazon.awssdk.core.SdkBytes;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeAsyncClient;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamRequest;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamResponseHandler;

public class BedrockSDKTest {

public static void main(String[] args) {
    BedrockRuntimeAsyncClient client = BedrockRuntimeAsyncClient.builder()
            .region(Region.US_EAST_1)
            .httpClient(NettyNioAsyncHttpClient.builder().build())
            .build();

    InvokeModelWithResponseStreamResponseHandler handler = 
        InvokeModelWithResponseStreamResponseHandler.builder()
            .subscriber(event -> {})
            .build();

    client.invokeModelWithResponseStream(
            InvokeModelWithResponseStreamRequest.builder()
                .modelId("arn:aws:bedrock:us-east-1:123456789012:provisioned-Model/test")
                .body(SdkBytes.fromUtf8String("{}"))
                .build(), 
            handler).join();

    client.close();
}

}
Underlying exception

Caused by: java.lang.IllegalArgumentException: The URI contain illegal characters: /model/arn%3Aaws%3Abedrock%3Aus-east-1%3A123456789012%3Aprovisioned-Model%2Ftest/invoke-with-response-stream
	at io.netty.handler.codec.http.HttpUtil.validateRequestLineTokens(HttpUtil.java:90)
	at io.netty.handler.codec.http.DefaultHttpRequest.<init>(DefaultHttpRequest.java:95)
	at io.netty.handler.codec.http.DefaultHttpRequest.<init>(DefaultHttpRequest.java:78)
	at software.amazon.awssdk.http.nio.netty.internal.RequestAdapter.adapt(RequestAdapter.java:55)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequest(NettyRequestExecutor.java:236)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$makeRequestListener$10(NettyRequestExecutor.java:181)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
	at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	... 1 more

Note that latest Java SDK v2.41.9 still uses Netty 4.1.130.Final which already seems to have this issue fixed though. Kindly validate at your end or provide a reproducible code sample in case you think ARNs containing colons are a problem.

Regards, Chaitanya

I am going to validate this fix