Skip to content

perf: Streamline s3 backend#19394

Draft
jtuglu1 wants to merge 1 commit intoapache:masterfrom
jtuglu1:optimize-s3-operations
Draft

perf: Streamline s3 backend#19394
jtuglu1 wants to merge 1 commit intoapache:masterfrom
jtuglu1:optimize-s3-operations

Conversation

@jtuglu1
Copy link
Copy Markdown
Contributor

@jtuglu1 jtuglu1 commented Apr 30, 2026

Description

S3 achieve strong read-after-write consistency in 2020. The current s3 backend architecture assumes a prior consistency model and therefore does some redundant calls which are both slow and costly.

Some other things to look into in a separate PR is parallel download requests using byte ranges for a single file (currently we use a single TCP connection which isn't the fastest and is subject to S3 single-connection bandwidth limitations).

High-level list of changes

  1. Removed isObjectInBucket guards before zip and gzip downloads (and the now-dead private method). The 404 from GetObject propagates as a SegmentLoadingException.
  2. Replaced the doesObjectExist + listObjectsV2 two-call sequence with a single getObjectMetadata request in S3DataSegmentMover.

Release note


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@jtuglu1 jtuglu1 force-pushed the optimize-s3-operations branch 2 times, most recently from ab3c110 to 180aa0f Compare May 1, 2026 16:50
@jtuglu1 jtuglu1 force-pushed the optimize-s3-operations branch from 180aa0f to 4986d81 Compare May 1, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant