file lost during write to S3 mount point

[gremila]

Mountpoint for Amazon S3 version

mount-s3 1.21.0

AWS Region

af-south-1

Describe the running environment

Hello, I am trying to stream backup of database directly to the S3 mount to S3 from an EC2.
My backup is around 26TB split in files of 50GB. At the end of the backup i notice no error, but, in the syslog I see

mount-s3[57948]: [WARN] ThreadId(10) fsync{req=1703526 ino=15 fh=10 datasync=false name=BACKUP_PROD_2.dmp.2}: mountpoint_s3_fs::fuse: fsync failed with errno 5: put fai
led: put request failed: Client error: Unknown response error: MetaRequestResult { response_status: 200, crt_error: Error(14357, "aws-c-s3: AWS_ERROR_S3_NON_RECOVERABLE_ASYNC_ERROR, Async error received from S
3 and not recoverable from retry."), error_response_headers: Some(Headers { inner: 0x7fc984038680 }), error_response_body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n\n\n\n\n<Error><Code>InvalidPart</
Code><Message>One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part\'s entity tag.</Message><UploadId>8YRzhZ_8YY9BmH9XE
6wSLy6uagVbIqhb_HYfJ05ZUuM9bGdf9vUtXzP_fhh_OFv6tJgFIWjoAMgUcwNncLAohbroSNPWfG6YwxQi1sF_6WsjgWgHsLXpbQZLUVLaptOYXMiKKoqkWxH91YcRXal_E2AtkYmtVbm9OkZfx7DLVnM-</UploadId><PartNumber>3001</PartNumber><ETag>725309c2
b0479cc10c7c325e834a0f01</ETag><RequestId>4XERWR9QR7J94R6E</RequestId><HostId>edHExP89EWRhQr0yxDObtudNmJPfS2Qk6/BPuairnNWV3fewcR06K/OnniBNXa3lLkS8epGUZwM=</HostId></Error>") }
2025-11-04T23:03:35.973527+02:00 p1iqiq1 mount-s3[57948]: [WARN] ThreadId(12) flush{req=1742978 ino=15 fh=10 pid=8053 name=BACKUP_PROD_2.dmp.2}: mountpoint_s3_fs::fuse: flush failed with errno 5: upload alread
y aborted for key ValidKey { key: "SAPIQ/BACKUP/SAPNLSDB_PROD/20251104-230001/bck2/BACKUP_PROD_2.dmp.2", name_offset: 48 }

which make my backup impossible to use.
Backup takes around 3:20 hours

How Can I ensure all my files was send or at lease if one file faile to write it somewhere localy to send it after

Credentials are the one of the EC2 role and os is SUSE Linux Enterprise Server 15 SP6"

Mountpoint options

mount-s3 s3://backup-bucket/DATABSE/ /mnt/s3

What happened?

Some files where lost during intensiv write to s3

Relevant log output

mount-s3[57948]: [WARN] ThreadId(10) fsync{req=1703526 ino=15 fh=10 datasync=false name=BACKUP_PROD_2.dmp.2}: mountpoint_s3_fs::fuse: fsync failed with errno 5: put fai
led: put request failed: Client error: Unknown response error: MetaRequestResult { response_status: 200, crt_error: Error(14357, "aws-c-s3: AWS_ERROR_S3_NON_RECOVERABLE_ASYNC_ERROR, Async error received from S
3 and not recoverable from retry."), error_response_headers: Some(Headers { inner: 0x7fc984038680 }), error_response_body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n\n\n\n\n<Error><Code>InvalidPart</
Code><Message>One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part\'s entity tag.</Message><UploadId>8YRzhZ_8YY9BmH9XE
6wSLy6uagVbIqhb_HYfJ05ZUuM9bGdf9vUtXzP_fhh_OFv6tJgFIWjoAMgUcwNncLAohbroSNPWfG6YwxQi1sF_6WsjgWgHsLXpbQZLUVLaptOYXMiKKoqkWxH91YcRXal_E2AtkYmtVbm9OkZfx7DLVnM-</UploadId><PartNumber>3001</PartNumber><ETag>725309c2
b0479cc10c7c325e834a0f01</ETag><RequestId>4XERWR9QR7J94R6E</RequestId><HostId>edHExP89EWRhQr0yxDObtudNmJPfS2Qk6/BPuairnNWV3fewcR06K/OnniBNXa3lLkS8epGUZwM=</HostId></Error>") }
2025-11-04T23:03:35.973527+02:00 p1iqiq1 mount-s3[57948]: [WARN] ThreadId(12) flush{req=1742978 ino=15 fh=10 pid=8053 name=BACKUP_PROD_2.dmp.2}: mountpoint_s3_fs::fuse: flush failed with errno 5: upload alread
y aborted for key ValidKey { key: "SAPIQ/BACKUP/SAPNLSDB_PROD/20251104-230001/bck2/BACKUP_PROD_2.dmp.2", name_offset: 48 }

[tadiwa-aizen]

Thank you for reporting this issue with Mountpoint-S3. We're actively investigating this problem and understand how concerning data loss can be, especially with large backup operations.

Our initial analysis indicates that Mountpoint correctly passed the write error to your application when the S3 multipart upload failed due to a checksum mismatch. Which means the data loss was properly signaled rather than silently occurring.

We're working to understand the root cause and we'll keep you updated on our progress.

[gremila]

Hello, as per investiguation, it seems, that, my EC2 had some spic throughput above its max, at the corresponding time, so, It can be possible that the ENA driver has drop some packet, not sure, but that can explain the issue. I don't know if it exist some possibility to resend some packet event with 2xx http return code, if the reponse contains error

[muddyfish]

Hi, we're investigating why Mountpoint issued multiple UploadPart requests which contained different checksums. From our investigation, it appears that although this didn't result in an immediate error, it was instead caught during the CompleteMultipartUpload request. In this particular case where the CompleteMultipartUpload request fails due to bad part checksums, Mountpoint is unable to retry the whole upload process as it discards the data after uploading the part to S3.

Regardless of this, your application should be able to handle errors returned by Mountpoint in case it's unable to write to files.

We're still continuing to investigate why Mountpoint issued multiple UploadPart requests which contained different checksums.