Upgrade to AWS SDK V2 #18891

jtuglu1 · 2026-01-08T03:10:58Z

Description

Upgrade to AWS SDK v2.40.0. Looks like dependencies hdfs-storage + ranger have transitive dependencies on V1. Those will need to be excluded or upgraded as well. Core modules/extensions required for runtime are not reliant on V1.

Migrated to strictly synchronous APIs (to maintain backwards-compatibility with V1). A follow-up PR should be able to make use of async APIs where suitable. S3TransferManager is the only case where an async client may be used.

Hiccups

KinesisAggregatorV2 not available in Maven – Maven via JitPack: amazon-kinesis-aggregator-2.0.2 has v1 code, amazon-kinesis-deaggregator-2.0.2 unavailable awslabs/kinesis-aggregation#120. Doesn't look like anything uses aggregation=true, so added a warning message. Seems like [FLINK-32097][Connectors/Kinesis] Implement support for Kinesis deaggregation flink-connector-aws#188 also ran into similar issues, but implemented their own unmerged approach. Per the warning on the https://github.com/awslabs/kinesis-aggregation repo, looks like data loss can occur anyways when using this mode: https://github.com/awslabs/kinesis-aggregation/blob/master/potential_data_loss.md so don't think that's really a desired feature anyways.

Release note

Upgrade to AWS SDK v2.40.0

This PR has:

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

...ns-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3DataSegmentPullerTest.java

...sions-core/s3-extensions/src/test/java/org/apache/druid/data/input/s3/S3InputSourceTest.java

...-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3StorageConnector.java

extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentMover.java

...exing-service/src/test/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplierTest.java

cryptoe · 2026-01-12T09:04:04Z

pom.xml

        <graaljs.version>22.3.5</graaljs.version>
        <mockito.version>5.14.2</mockito.version>
        <aws.sdk.version>1.12.784</aws.sdk.version>
+        <aws.sdk.v2.version>2.28.28</aws.sdk.v2.version>


Should we upgrade to the latest aws v2 sdk version ?

Yes I will bump it to 2.40 minor release. Wanted to minimize chances of upstream bugs by using a stable version for testing.

Fly-Style · 2026-01-13T10:36:11Z

cloud/aws-common/src/main/java/org/apache/druid/common/aws/AWSCredentialsUtils.java

 public class AWSCredentialsUtils
 {
-  public static AWSCredentialsProviderChain defaultAWSCredentialsProviderChain(final AWSCredentialsConfig config)
+  public static AwsCredentialsProvider defaultAWSCredentialsProviderChain(final AWSCredentialsConfig config)


Nit: there is also a AwsCredentialsProviderChain.of(...) method, it might look cleaner.

cloud/aws-common/src/test/java/org/apache/druid/common/aws/AWSClientUtilTest.java

...ons-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3ServerSideEncryption.java

Fly-Style · 2026-01-13T15:02:50Z

Very good job @jtuglu1! I have no objections, PR looks good from my perspective!

jtuglu1 · 2026-01-14T03:58:29Z

👍 I'd still like to add 1-2 ITs that fully cover the newer S3 changes, but overall think it's basically there.

cryptoe · 2026-01-14T13:58:26Z

Due to the criticality of the change, I think we should get 2 approvals for this PR.
cc @jtuglu1

cryptoe

Left a partial review.
Wanted to ask the following questions:
Did we test this patch on any cluster with the following things with we have s3 as the deep storage.

Kinesis Ingestion.
MSQ with Durable storage.
MSQ select with durable storage as the destination.
Segment getting nuked out.
Task logs being viewable and getting purged out on S3.
Segment ingestion.

Also have you tried this patch in any of the production clusters ?

cryptoe · 2026-01-14T14:01:59Z

cloud/aws-common/src/main/java/org/apache/druid/common/aws/FileSessionCredentialsProvider.java


-  @Override
-  public void refresh()
+  private void refresh()


Who calls the refresh method now ?

cryptoe · 2026-01-14T14:06:10Z

...-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplier.java

+   * Parse region from a Kinesis endpoint URL.
+   * Expected format: https://kinesis.{region}.amazonaws.com
+   */
+  private static Region parseRegionFromEndpoint(String endpoint)


We should probably add tests for this function.

cryptoe · 2026-01-14T15:21:09Z

...s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryWithBucketIterator.java

+    this.maxListingLength = maxListingLength;
+    this.maxRetries = maxRetries;
+
+    prepareNextRequest();


Feels a bit weird that in the constructor we are calling these methods.
Should we call these in hasNext() ?

cryptoe · 2026-01-14T15:22:09Z

...s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryWithBucketIterator.java

+      return true;
+    }
+
+    // Recognize s3sync.rb directory placeholders by MD5/ETag value.


Could you please share relevant docs for these placeholders ?

jtuglu1 · 2026-01-14T17:38:17Z

Hi @cryptoe. Yes, I'm testing this on our production clusters running Kafka + S3. We don't have production clusters using Kinesis. I've only been able to run tests locally. I wonder if you folks might have some Kinesis clusters to test the new logic out on?

cryptoe · 2026-01-20T10:03:10Z

Hi @cryptoe. Yes, I'm testing this on our production clusters running Kafka + S3. We don't have production clusters using Kinesis. I've only been able to run tests locally.
Thanks for this. Do keep us posted about the findings.

I wonder if you folks might have some Kinesis clusters to test the new logic out on?
We also don't have active kinesis clusters but let me get back to you on this.

github-actions bot added Area - Streaming Ingestion Area - Dependencies labels Jan 8, 2026

jtuglu1 added the Release Notes label Jan 8, 2026

jtuglu1 changed the title ~~Upgrade to AWS SDK V2~~ WIP: Upgrade to AWS SDK V2 Jan 8, 2026

github-advanced-security bot found potential problems Jan 8, 2026

View reviewed changes

jtuglu1 force-pushed the upgrade-sdk-v2 branch 4 times, most recently from 12c1550 to 0bde0ea Compare January 8, 2026 17:24

jtuglu1 added 6 commits January 8, 2026 09:41

Migrate s3-extensions

db06e05

Migrate ec2

2283996

Migrate kinesis

5e2ad2a

Migrate everything else

dbaee7a

Update licenses

15f4696

Migrate kinesis

24034e2

jtuglu1 force-pushed the upgrade-sdk-v2 branch 2 times, most recently from 91ee288 to 84db439 Compare January 8, 2026 18:20

Spotbugs

bd6cd75

jtuglu1 force-pushed the upgrade-sdk-v2 branch from 84db439 to bd6cd75 Compare January 8, 2026 18:26

More fixes

6336a9d

github-advanced-security bot found potential problems Jan 9, 2026

View reviewed changes

cryptoe reviewed Jan 12, 2026

View reviewed changes

jtuglu1 force-pushed the upgrade-sdk-v2 branch 2 times, most recently from 3e2092e to e2afbe4 Compare January 12, 2026 20:09

jtuglu1 closed this Jan 12, 2026

jtuglu1 reopened this Jan 12, 2026

jtuglu1 changed the title ~~WIP: Upgrade to AWS SDK V2~~ Upgrade to AWS SDK V2 Jan 13, 2026

Fly-Style reviewed Jan 13, 2026

View reviewed changes

cloud/aws-common/src/test/java/org/apache/druid/common/aws/AWSClientUtilTest.java Show resolved Hide resolved

Fly-Style reviewed Jan 13, 2026

View reviewed changes

...ons-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3ServerSideEncryption.java Show resolved Hide resolved

jtuglu1 marked this pull request as ready for review January 14, 2026 03:46

More fixes

009f34e

jtuglu1 force-pushed the upgrade-sdk-v2 branch from e2afbe4 to 009f34e Compare January 14, 2026 03:59

cryptoe added the Design Review label Jan 14, 2026

cryptoe reviewed Jan 14, 2026

View reviewed changes

Upgrade to AWS SDK V2 #18891

Are you sure you want to change the base?

Upgrade to AWS SDK V2 #18891

Uh oh!

Conversation

jtuglu1 commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Hiccups

Release note

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cryptoe Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

jtuglu1 Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fly-Style Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Fly-Style commented Jan 13, 2026

Uh oh!

jtuglu1 commented Jan 14, 2026

Uh oh!

cryptoe commented Jan 14, 2026

Uh oh!

cryptoe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cryptoe Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

cryptoe Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

cryptoe Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

cryptoe Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

jtuglu1 commented Jan 14, 2026

Uh oh!

cryptoe commented Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jtuglu1 commented Jan 8, 2026 •

edited

Loading

jtuglu1 Jan 12, 2026 •

edited

Loading

cryptoe left a comment •

edited

Loading