SocketException: Unexpected end of file from server

[kronst]

Describe the bug

After upgrading AWS SDK for Java from version 2.35.10 to 2.37.5, I started experiencing intermittent SocketException: Unexpected end of file from server errors during S3 putObject operations.
The issue happens with UrlConnectionHttpClient. We did not change any configuration, only upgraded the SDK version.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

putObject should complete successfully as it did in version 2.35.10

Current Behavior

Intermittent failures with the following exception:

java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:955)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759)
    at sun.net.www.protocol.http.HttpURLConnection.expect100Continue(HttpURLConnection.java:1352)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1488)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1437)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:219)
    at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
    at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
    at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.lambda$tryGetOutputStream$0(UrlConnectionHttpClient.java:331)
    at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.getAndHandle100Bug(UrlConnectionHttpClient.java:367)
    at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.tryGetOutputStream(UrlConnectionHttpClient.java:331)
    at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:308)
    at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:281)
    at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:103)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:88)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:64)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:46)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.executeRequest(RetryableStage.java:93)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:56)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:12535)

Reproduction Steps

// Current S3Client configuration
S3Client.builder()
            .region(Region.of(awsRegion))
            .credentialsProvider(DefaultCredentialsProvider.create())
            .endpointOverride(URI.create(awsS3Endpoint))
            .forcePathStyle(true)
            .build()

// UrlConnectionHttpClient dependency in the project
implementation("software.amazon.awssdk:url-connection-client")

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.37.5

JDK version used

Eclipse Temurin 21.0.5

Operating System and version

Ubuntu 24.04 LTS

[bhoradc]

Hello @kronst

Thank you for reporting the issue. Based on the stacktrace you provided, the error seems to occur during the HTTP 100-continue handshake when using the UrlConnectionHttpClient with S3.

I am unable to reproduce the error with AWS S3 with given details.

S3ClientSample

import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;

import java.net.URI;

public class S3ClientExample {
    public static void main(String[] args) {
        
        String awsRegion = "us-east-1";
        
        S3Client s3Client = S3Client.builder()
                .region(Region.of(awsRegion))
                .httpClient(UrlConnectionHttpClient.builder().build())
                .build();

        PutObjectRequest putRequest = PutObjectRequest.builder()
                .bucket("**-test-bucket")
                .key("***-testhttp-key")
                .build();

        s3Client.putObject(putRequest, RequestBody.fromString("test content"));
        
        s3Client.close();
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>V2_UrlConnectionHttpClient_6670</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>22</maven.compiler.source>
        <maven.compiler.target>22</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <aws.sdk.version>2.37.5</aws.sdk.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>${aws.sdk.version}</version>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>url-connection-client</artifactId>
            <version>${aws.sdk.version}</version>
        </dependency>
    </dependencies>

</project>

Can you kindly provide below additional details to better understand your specific scenario?

1. Are you connecting to AWS S3 directly or using a mock/local S3-compatible service? Whats awsS3Endpoint?
2. Minimal reproducible code sample showing how the RequestBody is created. More details on the size of the request body you are uploading.
3. Are you setting any custom headers or metadata on the PutObjectRequest?
4. SDK Debug logs, if the issue is not reliably reproducible (post redacting sensitive details).

Regards,
Chaitanya

[kronst]

Hello, @bhoradc
Thanks for the answer!

1. I'm connecting directly to AWS S3 (https://s3.eu-central-1.amazonaws.com)

fun upload(
    data: ByteArray,
    objectKey: String,
    contentType: String,
) {
    val request = PutObjectRequest.builder()
        .bucket(bucket)
        .key(objectKey)
        .contentType(contentType)
        .contentLength(data.size.toLong())
        .acl(ObjectCannedACL.PRIVATE)
        .build()

    s3Client.putObject(
        request,
        RequestBody.fromBytes(data)
    )
}

3. No custom headers, no metadata and no additional HTTP client configuration provided to the S3Client builder
4. Unfortunately, I have no debug logs. The issue appears periodically on production. I'll try to get some additional information about files that provokes the issue. Maybe I'll find any dependency.

Just in case, I'll add that application uses Quarkus as web framework. The exception start rising right after we upgraded Quarkus which brought new AWS SDK version. But since we're using SDK directly I don't think it somehow related to Quarkus.

[bhoradc]

Hi @kronst

Thanks for the response. I am still unable to reproduce the issue. Can you provide a self-contained code sample that reproduces the error? Additionally, any context or patterns around when this issue occurs would be helpful.

Have you tried testing with ApacheHttpClient to see if the issue persists?

Regards,
Chaitanya

[bhoradc]

HI @kronst

While we wait for the self-contained code sample, debug logs, and patterns around when this issue occurs, I understand you're using Quarkus which manages the SDK version. If it's feasible in your environment, could you test with different SDK versions to help narrow down when the issue was introduced?

Based on some investigation, this may be related to changes in SDK 2.37.0. If testing different versions is possible, starting with 2.37.0 and testing lower versions like 2.36.3 would help confirm when the issue first appeared.

Additionally, have you had a chance to test with ApacheHttpClient instead of UrlConnectionHttpClient to see if the issue persists?

Regards,
Chaitanya

[kronst]

Hello, @bhoradc
Sorry for the delay with an answer.
Yes, you're totally right about Quarkus. We upgraded version from 3.29.4 to 3.30.2, which brings new version of AWS SDK (2.35.10 -> 2.37.5). Before upgrade there were no issues with handling files. There were no changes in the application except upgrading version. That's why my assumption concerns the AWS SDK update specifically.

Regarding the logs I've added recently. Unfortunately, this didn't give me any patterns or any additional information about possible reasons. Because the exception raises on various files with different content types (images, documents, audio, video files) or sizes.

Answering your question about Apache client. No, we haven't tried it yet. Our application uses many AWS services managed by Quarkus. And this client will be applied to all of them by default, which a bit risky to do at once. But I'll try to force 2.35.10 SDK version which was used before upgrade to investigate if the issue will raise again.