S3UTILS-226 MPU orphan cleanup script by jonathan-gramain · Pull Request #380 · scality/s3utils

jonathan-gramain · 2026-03-13T12:52:40Z

Implement cleanupMpuOrphans.js to delete orphaned parts and sproxyd keys.

Orphaned parts are part metadata which do not correspond to an overview key with the same upload ID.

Sproxyd keys are only removed if no completed MPU object has a matching upload ID in the same bucket.

codecov · 2026-03-13T12:58:01Z

Codecov Report

❌ Patch coverage is 61.67401% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.23%. Comparing base (07cb65f) to head (4af1de4).
⚠️ Report is 1 commits behind head on development/1.16.

Files with missing lines	Patch %	Lines
cleanupMpuOrphans.js	61.67%	86 Missing and 1 partial ⚠️

Additional details and impacted files

@@                 Coverage Diff                  @@
##           development/1.16     #380      +/-   ##
====================================================
+ Coverage             44.59%   45.23%   +0.64%     
====================================================
  Files                    84       85       +1     
  Lines                  5819     6046     +227     
  Branches               1237     1282      +45     
====================================================
+ Hits                   2595     2735     +140     
- Misses                 3179     3265      +86     
- Partials                 45       46       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bert-e · 2026-03-25T18:42:20Z

Hello jonathan-gramain,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-25T18:42:23Z

Incorrect fix version

The Fix Version/s in issue S3UTILS-226 contains:

1.17.5

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

1.16.6
1.17.6
1.18.0

Please check the Fix Version/s of S3UTILS-226, or the target
branch of this pull request.

jonathan-gramain · 2026-03-25T18:52:56Z

ping

bert-e · 2026-03-25T18:53:04Z

Incorrect fix version

The Fix Version/s in issue S3UTILS-226 contains:

1.16.6
1.17.5
1.18.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

1.16.6
1.17.6
1.18.0

Please check the Fix Version/s of S3UTILS-226, or the target
branch of this pull request.

arnaudlangloisscality

lgtm

scality-fno

Strictly from a doc perspective.

I'm ok with the documentation. Some clarification could be beneficial. Not a blocker, though.

scality-fno · 2026-03-27T21:15:21Z

cleanupMpuOrphans.js

+    log.info('starting discovery phase: scanning MPU shadow bucket', { bucket, shadowBucket });
+
+    status.bucket = bucket;
+    status.phase = 'discovery';
+    status.orphanedUploadIds = 0;
+    status.versionsScanned = 0;
+    status.orphanPartsDeleted = 0;
+    status.sproxydKeysDeleted = 0;
+
+    const orphanMap = await buildOrphanMap(
+        BUCKETD_HOSTPORT,
+        shadowBucket,
+        { pageSize: LISTING_PAGE_SIZE, retry: RETRY_PARAMS },
+    );
+    const orphanCount = Object.keys(orphanMap).length;
+    status.orphanedUploadIds = orphanCount;
+    logProgress('discovery phase complete');
+    if (orphanCount === 0) {
+        return;
+    }
+
+    for (const [uploadId, info] of Object.entries(orphanMap)) {
+        log.info('orphaned MPU found', {


Don't know if the Usage section is meant to describe the exact sequence of the log messages, but if it is, the code contradicts it: the "orphaned MPU found" message comes last. Is this intentional?

I put the messages in roughly the order in which they should come indeed. I think "orphaned MPU found" is correctly placed since it occurs during the first discovery phase, before the cleanup phase.

cleanupMpuOrphans.js

Implement cleanupMpuOrphans.js to delete orphaned parts and sproxyd keys. Orphaned parts are part metadata which do not correspond to an overview key with the same upload ID. Sproxyd keys are only removed if no completed MPU object has a matching upload ID in the same bucket.

jonathan-gramain · 2026-04-01T18:35:32Z

/approve

bert-e · 2026-04-01T18:35:41Z

Incorrect fix version

The Fix Version/s in issue S3UTILS-226 contains:

1.16.6
1.17.5
1.18.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

1.16.6
1.17.6
1.18.0

Please check the Fix Version/s of S3UTILS-226, or the target
branch of this pull request.

The following options are set: approve

jonathan-gramain · 2026-04-01T18:36:00Z

ping

bert-e · 2026-04-01T18:36:11Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/S3UTILS-226-cleanup-mpu-orphans into
development/1.16
w/1.17/improvement/S3UTILS-226-cleanup-mpu-orphans will be merged into development/1.17
w/1/improvement/S3UTILS-226-cleanup-mpu-orphans will be merged into development/1

The following branches will NOT be impacted:

development/1.13
development/1.14
development/1.15
development/1.4
development/1.7

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e · 2026-04-01T18:49:38Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.16
✔️ development/1.17
✔️ development/1

The following branches have NOT changed:

development/1.13
development/1.14
development/1.15
development/1.4
development/1.7

Please check the status of the associated issue S3UTILS-226.

Goodbye jonathan-gramain.

The following options are set: approve

jonathan-gramain force-pushed the improvement/S3UTILS-225-list-bucketd-versions-full-metadata branch from 8ab54e6 to 07cb65f Compare March 25, 2026 18:14

jonathan-gramain force-pushed the improvement/S3UTILS-226-cleanup-mpu-orphans branch from 3386bfa to e3f5b12 Compare March 25, 2026 18:14

Base automatically changed from improvement/S3UTILS-225-list-bucketd-versions-full-metadata to development/1.16 March 25, 2026 18:42

jonathan-gramain force-pushed the improvement/S3UTILS-226-cleanup-mpu-orphans branch from 0458ae8 to bb4d32c Compare March 25, 2026 22:30

arnaudlangloisscality approved these changes Mar 27, 2026

View reviewed changes

scality-fno approved these changes Mar 27, 2026

View reviewed changes

jonathan-gramain force-pushed the improvement/S3UTILS-226-cleanup-mpu-orphans branch from 747bcf5 to 4af1de4 Compare April 1, 2026 18:34

bert-e merged commit 4af1de4 into development/1.16 Apr 1, 2026
12 checks passed

bert-e deleted the improvement/S3UTILS-226-cleanup-mpu-orphans branch April 1, 2026 18:49

Conversation

jonathan-gramain commented Mar 13, 2026

Uh oh!

codecov bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bert-e commented Mar 25, 2026

Hello jonathan-gramain,

Uh oh!

bert-e commented Mar 25, 2026

Incorrect fix version

Uh oh!

jonathan-gramain commented Mar 25, 2026

Uh oh!

bert-e commented Mar 25, 2026

Incorrect fix version

Uh oh!

arnaudlangloisscality left a comment

Choose a reason for hiding this comment

Uh oh!

scality-fno left a comment

Choose a reason for hiding this comment

Uh oh!

scality-fno Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-gramain Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jonathan-gramain commented Apr 1, 2026

Uh oh!

bert-e commented Apr 1, 2026

Incorrect fix version

Uh oh!

jonathan-gramain commented Apr 1, 2026

Uh oh!

bert-e commented Apr 1, 2026

Integration data created

Uh oh!

bert-e commented Apr 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 13, 2026 •

edited

Loading