Fix S3 cache race that breaks restores with "did not match expected ETag"

[vivster7]

Problem

With the s3 backend, two agents running the same step in parallel can fail a cache restore with:

download failed: s3://<bucket>/<key> ... Contents of stored object "<key>" in bucket "<bucket>" did not match expected ETag.

Root cause is a race between a restore and a concurrent re-save of the same S3 key:

• Keys are content-addressed (lib/shared.bash hashes the manifest), so parallel jobs compute an identical key.
• Save overwrites non-atomically: exists (a list-objects-v2) then aws s3 cp. Two jobs can both see "not exists" and both upload — last write wins.
• Restore validates against a stale ETag: aws s3 cp (botocore s3transfer) HEADs the object, records its ETag, then issues ranged GETs with IfMatch=<etag>. An overwrite landing mid-download makes S3 return 412 PreconditionFailed, which the CLI surfaces as the error above. This only affects objects past multipart_threshold (8 MB), i.e. large dependency caches.

I reproduced this end-to-end against MinIO + AWS CLI v2.31.38 (deterministic stale-If-Match → 412, and a throttled cp overwritten mid-download producing the exact error string).

Fix

• save_cache: store single objects with a conditional create — aws s3api put-object --if-none-match '*'. The first writer wins; later writers get PreconditionFailed, treated as success. The object is never overwritten, so its ETag stays stable. Falls back to a normal copy when conditional writes aren't available (older CLI / S3-compatible endpoint) or the object exceeds the single-PUT limit. force still overwrites.
• restore_cache: retry the download on the transient did not match expected ETag failure, configurable via BUILDKITE_PLUGIN_S3_CACHE_DOWNLOAD_RETRIES (default 3). Content-addressed keys guarantee the retried download returns identical contents.

Tests

tests/cache_s3.bats (all 16 pass via buildkite/plugin-tester, shellcheck -x clean):

• conditional single-file save uses put-object --if-none-match
• force overwrites instead of conditional create
• concurrent writer (PreconditionFailed) is treated as success
• fallback to copy when conditional create is unsupported
• restore retries then succeeds; restore fails after exhausting retries

Backwards compatibility

No config changes required. Folder caches keep using sync; force behaviour is preserved; the only new (optional) knob is BUILDKITE_PLUGIN_S3_CACHE_DOWNLOAD_RETRIES.

Made with Cursor