[Feature] Add cloud leaderboard benchmark cases and client support

[jamesgao-jpg]

Summary

This PR adds the CloudLeadboard v2 benchmark surface to VectorDBBench. The goal is to complement the existing raw-performance leaderboard with cloud-oriented cases that capture production behaviors managed vector database users care about: ingest readiness, payload-aware search, tenant-shaped traffic, cold latency, and cost-aware interpretation.

What is added

New cloud leaderboard cases

• CloudPayloadSearchCase: measures search performance with explicit response payload profiles: ids_only, scalar_label, and vector. It supports unfiltered search, integer-filter search, and scalar-label filter search.
• CloudInsertCase: measures insert throughput and separates client insert completion from downstream readiness signals such as fully searchable and fully indexed.
• CloudColdLatencyCase: measures cold and warm serial latency so first-query and cache-sensitive serving behavior are visible instead of hidden by warm concurrent loops.
• CloudMultiTenantSearchCase: models SaaS-style tenant-routed workloads with deterministic tenant assignment and tenant-aware query routing.

Runtime and metric plumbing

• Threads payload profile configuration through case config, runners, client search calls, metrics, and JSON result output.
• Adds payload byte estimation to result metrics so payload-heavy searches can be compared more explicitly.
• Adds a cold/warm search runner and task-runner integration for cold latency measurement.
• Adds concurrent insert readiness polling for cloud insert runs.
• Adds first-class result fields for cloud insert, cold latency, payload profile, and related cloud case metadata.

Client support

• Milvus/Zilliz Cloud: adds payload output handling, scalar-label support, multitenant partition-key validation, and related schema checks.
• Pinecone: adds payload-profile search behavior, metadata/vector return handling, insert readiness polling through write/index LSNs where available, and namespace-based multitenant routing.
• turbopuffer: adds payload-profile search behavior, scalar payload label configuration, multitenant namespace support, write backpressure control, and namespace pin/unpin CLI support for pinned benchmark runs.

CLI, frontend, and docs

• Adds CLI options for payload profile, cloud filters, cold query count, insert batch size/duration, tenant settings, and turbopuffer pinning.
• Adds frontend case config entries for the cloud payload search cases.
• Adds a May 2026 CloudLeadboard v2 release note under docs/release/2026-05-cloud-leadboard.md.
• Updates README.md to mention the new CloudLeadboard v2 benchmark cases and link to the release note.

Notes

• The raw cloud leaderboard result dump was intentionally removed from this PR. Result artifacts can be added later in a smaller, dedicated update or kept outside the source tree.
• Cloud cost and Pareto interpretation are described in the release note, but this PR focuses on benchmark case support and documentation rather than publishing a full result dataset.

Test Plan

• python -m pytest tests/test_cloud_payload_case.py tests/test_cloud_payload_search.py tests/test_cloud_insert_case.py tests/test_cloud_cold_latency_case.py tests/test_multitenant_case.py tests/test_turbopuffer_cli.py tests/test_pinecone_multitenant.py -q

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment