Skip to content

feat: S3 transfer manager improvements#3247

Open
yenfryherrerafeliz wants to merge 19 commits intoaws:masterfrom
yenfryherrerafeliz:s3_transfer_manager_improvements
Open

feat: S3 transfer manager improvements#3247
yenfryherrerafeliz wants to merge 19 commits intoaws:masterfrom
yenfryherrerafeliz:s3_transfer_manager_improvements

Conversation

@yenfryherrerafeliz
Copy link
Contributor

Description of changes:

  • Includes resumable capabilities to resume downloads and uploads.
  • Concurrent downloads to enhance download speed.
  • Directory Transfer Improvements.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@yenfryherrerafeliz yenfryherrerafeliz changed the title S3 transfer manager improvements feat: S3 transfer manager improvements Feb 24, 2026
- Address some formatting issues.
- Fixed arguments passed to ResumableDownload
- Add test coverage for:
  - DirectoryProgressTracker
  - DirectoryTransferProgressAggregator
  - DirectoryTransferProgressSnapshot
  - DirectoryDownloader
  - DirectoryUploader
@yenfryherrerafeliz
Copy link
Contributor Author

vendor/bin/behat --format=progress --suite=smoke --tags='~@noassumerole'
...................................................................... 70
...................................................................... 140
...................................................................... 210
............

111 scenarios (111 passed)
222 steps (222 passed)
0m35.66s (96.23Mb)
vendor/bin/behat --format=progress --tags=integ
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
...................................................................... 350
...................................................................... 420
......................................................

136 scenarios (136 passed)
474 steps (474 passed)
27m28.00s (720.09Mb)

Copy link
Member

@stobrien89 stobrien89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few more questions/comments

*
* @return bool
*/
private function writePartToDestinationHandle(array $response): bool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: do part numbers start at 1 or 0? If zero, I think this might be exposed to an off-by-one error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part numbers start at 1 actually.

$this->body->rewind();
}

$partNo = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does resume data store 1-based part numbers? Might be a mismatch during resume skip logic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$partNo is incremented at line 388, which will unlikely cause a resumable
persist mismatch. Is there any other mismatch you are referring to here, that I
may be missing?

$chunk = $body->read(self::READ_BUFFER_SIZE);

if (fwrite($this->handle, $chunk) === false) {
throw new FileDownloadException("Failed to write data to temporary file.");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: where do these exceptions bubble up to?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If part number one fails then it will go through here, otherwise it will go through here.

- Add validation for when resuming an upload and the source file changed
- Add test coverage for the behavior when resuming upload and source file changed
- Address some styling issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants