update azure blob upload support to use azure_storage_blob by demoray · Pull Request #762 · microsoft/avml

demoray · 2026-03-17T19:33:33Z

No description provided.

Copilot

Pull request overview

Updates the Azure Blob Storage upload implementation to use the newer azure_storage_blob crate/API, aligning the blobstore uploader with the updated Azure Rust SDK ecosystem.

Changes:

Replace azure_storage_blobs usage with azure_storage_blob (stage_block + commit_block_list) in the blob uploader.
Refactor the uploader to share a BlockBlobClient via Arc for concurrent block uploads.
Update Cargo dependencies to azure_core = 0.33 and azure_storage_blob = 0.10.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`src/upload/blobstore.rs`	Migrates block upload/finalize calls to `azure_storage_blob` APIs and updates client handling for concurrency.
`Cargo.toml`	Swaps dependency from `azure_storage_blobs` to `azure_storage_blob` and bumps `azure_core`.
`Cargo.lock`	Lockfile refresh reflecting the dependency migration and transitive dependency changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Cargo.toml

Copilot

Pull request overview

Updates the Azure Blob Storage upload implementation to use the newer azure_storage_blob crate (and newer azure_core) rather than azure_storage_blobs, aligning the blob upload path with the current Azure SDK crates.

Changes:

Migrated blob upload/commit logic from put_block/put_block_list to stage_block/commit_block_list using azure_storage_blob.
Updated BlobUploader to share the BlobClient via Arc for concurrent upload workers.
Bumped Azure SDK dependencies and adjusted feature flags accordingly in Cargo.toml (with lockfile updates).

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/upload/blobstore.rs	Ports the uploader to `azure_storage_blob` APIs and updates concurrency/client sharing accordingly.
Cargo.toml	Switches dependencies from `azure_storage_blobs` to `azure_storage_blob` and updates Azure SDK versions/features.
Cargo.lock	Lockfile refresh reflecting updated Azure SDK crates and new transitive dependencies.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Cargo.toml

Copilot

Pull request overview

Updates the Azure Blob upload implementation to use the newer azure_storage_blob crate APIs, replacing the previous manual block upload approach.

Changes:

Replace queued/manual block uploads with BlobClient::upload(...) using SDK upload options.
Add a SeekableStream-backed FileStream adapter to support SDK retries/resets and progress reporting.
Update Cargo features/dependencies and add a unit test for stream reset behavior.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

File	Description
src/upload/status.rs	Adds `Status::reset()` to support rewinding progress on stream reset during blob uploads.
src/upload/blobstore.rs	Reworks blob upload logic to SDK `upload`, introduces `FileStream` implementing `SeekableStream`, and adds a reset test.
Cargo.toml	Switches Azure crate dependencies/features to `azure_storage_blob` + `async-trait`, and adjusts default features.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/upload/blobstore.rs

Cargo.toml

src/upload/blobstore.rs

Copilot

Pull request overview

This PR migrates the Azure Blob upload implementation to the newer azure_storage_blob crate and adapts the upload path to use the SDK’s built-in parallel upload with a seekable body.

Changes:

Replace the previous block-queue uploader with BlobClient::upload using BlobClientUploadOptions (parallelism + partition size).
Introduce a SeekableStream-backed FileStream to support SDK retries/reset while streaming from disk.
Update Cargo features/dependencies to use azure_storage_blob/azure_core versions and remove async-channel.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
`src/upload/status.rs`	Adds a `reset()` hook so progress reporting can be reset when the SDK retries/rewinds the stream.
`src/upload/blobstore.rs`	Reworks blob upload to the new SDK API and adds `FileStream` implementing `SeekableStream` + `AsyncRead`.
`Cargo.toml`	Swaps Azure blob dependencies/features to `azure_storage_blob` and updates TLS feature wiring.
`Cargo.lock`	Updates lockfile for the new Azure SDK dependency graph.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/upload/blobstore.rs

+            match *state {
+                ReadState::Idle => {
+                    *state = ReadState::Locking(Box::pin(this.handle.clone().lock_owned()));
+                }
+                ReadState::Locking(ref mut lock_future) => {
+                    match Future::poll(Pin::as_mut(lock_future), cx) {
+                        Poll::Ready(guard) => *state = ReadState::Locked(guard),
+                        Poll::Pending => return Poll::Pending,
+                    }
+                }
+                ReadState::Locked(ref mut guard) => {


src/upload/blobstore.rs

Copilot

Pull request overview

This PR migrates the Azure Blob upload implementation to the newer azure_storage_blob crate and adapts the upload path to use the SDK’s built-in parallel upload with a seekable request body.

Changes:

Replaced manual block-queue/concurrent uploader logic with BlobClient::upload(...) + BlobClientUploadOptions.
Added a seekable, cloneable FileStream wrapper to support SDK retry/reset semantics.
Updated Cargo features/dependencies to use azure_storage_blob, newer azure_core, and async-trait.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.

File	Description
src/upload/status.rs	Adds a `reset()` hook (real + no-op) used by blob upload stream retry/reset logic.
src/upload/blobstore.rs	Reworks upload implementation to `azure_storage_blob` and introduces `FileStream` for seekable uploads.
Cargo.toml	Swaps Azure SDK crates, updates feature flags, and adds `async-trait` for async trait impl.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/upload/blobstore.rs

+#[derive(Default)]
+enum ReadState {
+    #[default]
+    Idle,
+    Locking(FileLockFuture),
+    Locked(OwnedMutexGuard<File>),
+}
+
+#[async_trait::async_trait]
+impl SeekableStream for FileStream {
+    async fn reset(&mut self) -> azure_core::Result<()> {
+        *self.read_state.lock().await = ReadState::Idle;
+        let mut handle = self.handle.clone().lock_owned().await;
+        handle.seek(SeekFrom::Start(0)).await?;
+        self.status.reset();
+        Ok(())
+    }


src/upload/blobstore.rs

+            // Another task is currently holding `read_state`; yield and try again later
+            cx.waker().wake_by_ref();


src/upload/blobstore.rs

+        let stream = FileStream::new(file, DEFAULT_BUFFER_SIZE).await?;
+        let stream: Box<dyn SeekableStream> = Box::new(stream);
+        let content: RequestContent<Bytes, NoFormat> = Body::from(stream).into();

-        let (block_list, ()) = futures::try_join!(queue_handle, uploaders)?;
+        let options = BlobClientUploadOptions {
+            parallel: NonZeroUsize::new(uploaders_count),
+            partition_size: NonZeroUsize::new(block_size),
+            ..Default::default()
+        };


Copilot

Pull request overview

This PR migrates Azure Blob uploads from azure_storage_blobs to azure_storage_blob, updating the uploader implementation and related CLI/config surfaces accordingly.

Changes:

Replaced the previous block-queue uploader with azure_storage_blob::BlobClient::upload() and a seekable, retry-friendly file stream.
Switched block size/concurrency parameters to NonZeroUsize and adjusted defaulting logic + tests around upload parameter calculation.
Updated feature flags/dependencies and build script to run doc tests with all features.

Reviewed changes

Copilot reviewed 5 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/upload/status.rs	Adds `Status::reset()` for blobstore use and a no-op stub for non-status builds.
src/upload/blobstore.rs	Migrates uploader to `azure_storage_blob`, introduces `FileStream` (seekable/retryable), updates sizing/concurrency logic and tests.
src/bin/avml.rs	Updates CLI arg types for blobstore block size/concurrency to `NonZeroUsize` options.
src/bin/avml-upload.rs	Updates upload CLI arg types for block size/concurrency to `NonZeroUsize` options.
eng/build.sh	Adds `cargo test --doc` run for all features.
README.md	Updates help text to reflect non-zero requirements for blobstore args.
Cargo.toml	Bumps crate version, swaps Azure blob dependency, updates feature flags and TLS wiring.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/upload/blobstore.rs

+            // Another task is currently holding `read_state`; yield and try again later
+            cx.waker().wake_by_ref();


src/upload/blobstore.rs

        let file_size = file.metadata().await?.len().try_into()?;
+        let Some(file_size) = NonZeroUsize::new(file_size) else {
+            return Ok(());
+        };


README.md

-          specify blob upload concurrency
+          specify blob upload concurrency; must be greater than 0

          [default: 10]


demoray added 2 commits March 17, 2026 19:32

update azure blob upload support to use azure_storage_blob

3f08bd5

fmt

f026b1f

demoray requested a review from Copilot March 17, 2026 19:35

Copilot started reviewing on behalf of demoray March 17, 2026 19:36 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

demoray added 2 commits March 17, 2026 20:01

address lint

1e5781f

reduce features used from azure_core

4efbc04

demoray requested a review from Copilot March 17, 2026 20:14

Copilot started reviewing on behalf of demoray March 17, 2026 20:16 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

demoray added 2 commits March 18, 2026 11:19

use the Azure SDK's built-in parallel upload functionality

37892b4

address copilot PR feedback

8af2084

demoray requested a review from Copilot March 18, 2026 11:24

Copilot AI reviewed Mar 18, 2026

View reviewed changes

src/upload/blobstore.rs Show resolved Hide resolved

src/upload/blobstore.rs Outdated Show resolved Hide resolved

src/upload/blobstore.rs Show resolved Hide resolved

Cargo.toml Outdated Show resolved Hide resolved

src/upload/blobstore.rs Outdated Show resolved Hide resolved

Copilot started reviewing on behalf of demoray March 18, 2026 11:37 View session

demoray added 2 commits March 18, 2026 12:00

address feedback

8b54936

remove status as a default feature

8c9bda5

demoray requested a review from Copilot March 18, 2026 12:02

Copilot started reviewing on behalf of demoray March 18, 2026 12:06 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

don't expose setting the size for BlobUploader, as this isn't used

8f29971

demoray requested a review from Copilot March 18, 2026 12:18

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Copilot started reviewing on behalf of demoray March 18, 2026 12:31 View session

demoray added 2 commits March 18, 2026 13:18

bump rev

b1bacec

use NonZeroUsize for sizes

d2615ab

demoray requested a review from Copilot March 19, 2026 14:35

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update azure blob upload support to use azure_storage_blob#762

update azure blob upload support to use azure_storage_blob#762
demoray wants to merge 11 commits intomainfrom
update-azure-sdk

demoray commented Mar 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// Another task is currently holding `read_state`; yield and try again later
		cx.waker().wake_by_ref();

Conversation

demoray commented Mar 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants