fix(s3): bound concurrent multipart part uploads with a global semaphore

[jaredLunde]

Problem

object_store's WriteMultipart spawns one upload task per part with no concurrency limit, and glidefs acquired the max_s3_uploads semaphore per pack, not per part. So a large pack fanned out into ~20 simultaneous part PUTs, and across concurrent packs × 64 exports this opened 450+ connections — saturating the uplink into bufferbloat (~600 ms RTT), 12–15% packet loss, and TCP congestion collapse. Uploads then exceeded the 300 s timeout and failed in a loop. max_s3_uploads looked like "128" but the real connection count was 128 × ~20.

Fix

Replace WriteMultipart with BoundedMultipart: every part PUT (and every single-PUT small pack) acquires one permit from the shared upload semaphore and holds it until that request completes. max_s3_uploads now means exactly "max simultaneous outbound S3 write requests," globally — independent of how many exports/packs/parts exist. Bounded memory (≤ limit parts in flight) and natural backpressure (a slow part holds its permit) come for free.

This is the correct invariant: the limit is enforced at the granularity of the actual network operation (the part PUT), so the configured number maps directly to connections with no multiplicative surprise.

Tests

• stream_pack_bounds_global_inflight_part_uploads — 3 concurrent packs (18 parts of demand) sharing a 4-permit semaphore peak at exactly 4 concurrent part PUTs. Asserts == LIMIT: the bound is both never exceeded and actually reached (not over-throttling). Unbounded this would be 18.
• bounded_multipart_round_trips_pack_bytes — a multipart-streamed pack (256-byte parts, real InMemory upload) is byte-identical to the canonical assembled pack; index parses and data round-trips.
• 9 existing integration tests (compaction / cooldown / flush / crash-midway / abort-orphan) pass through the new upload path.

🤖 Generated with Claude Code