Skip to content

compute: split index peek literal constraints per worker#35026

Draft
antiguru wants to merge 1 commit intoMaterializeInc:mainfrom
antiguru:targeted_peek
Draft

compute: split index peek literal constraints per worker#35026
antiguru wants to merge 1 commit intoMaterializeInc:mainfrom
antiguru:targeted_peek

Conversation

@antiguru
Copy link
Member

Summary

  • Route index peek literal constraints to only the worker whose arrangement shard holds the key (row.hashed() % peers), instead of broadcasting all constraints to all workers
  • Workers with no matching constraints short-circuit immediately with an empty response, avoiding trace clones, compaction setup, and cursor seeks

Known limitation: errors in arrangements are not sharded by key, so workers that receive no constraints may miss errors. This PR is for performance exploration only.

Test plan

  • cargo check -p mz-compute
  • cargo clippy -p mz-compute -p mz-compute-client
  • cargo test -p mz-compute-client
  • bin/sqllogictest -- test/sqllogictest/transform/literal_constraints.slt
  • bin/sqllogictest -- test/sqllogictest/transform/notice/index_too_wide_for_literal_constraints.slt
  • bin/sqllogictest -- test/sqllogictest/indexes.slt

🤖 Generated with Claude Code

Arrangement data is sharded by `row.hashed() % peers`, so for index
peeks with literal constraints only one worker holds data for a given
key. Route each constraint to the owning worker in the command channel
and have workers with no constraints short-circuit with an empty
response, avoiding unnecessary trace clones, compaction, and cursor
seeks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link

Pre-merge checklist

  • The PR title is descriptive and will make sense in the git log.
  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant