Skip to content

Conversation

@fpacifici
Copy link
Collaborator

This is an alternative approach to #218.

The rust adapter chains map and filter steps together in one step in rust.
This means that, when a map or filter is processed, it is not added to the
Rust consumer immediately. This only happens when the chain is finalized.

that happens when we encounter a step that cannot be chained or a sink.

Problem: if there is a sequence of map operations and no sink, these were
never added to the rust consumer.

This PR requires aa Sink to the end of each branch.
This idea depends on the assumption that, if you terminate a branch with
anything that can produce output, is likely wrong.
If a step can produce an output and it is at the end of the branch,
what should we do with those messages ?

Requiring a sink implies asking this question to the author of the
pipeline and letting them decide.

@fpacifici fpacifici requested a review from a team as a code owner January 23, 2026 22:52
@github-actions
Copy link

github-actions bot commented Jan 23, 2026

Semver Impact of This PR

None (no version bump detected)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


  • Require sink at the end of a branch of the pipeline by fpacifici in #230
  • Move gcssink config override out of the adapter by fpacifici in #228

🤖 This preview updates automatically when you update the PR.

Copy link
Member

@evanh evanh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be my preferred way of handling this case. Once we have a sink primitive for tasks, I think that will cover all the cases.

@fpacifici fpacifici enabled auto-merge (squash) January 27, 2026 20:39
@fpacifici fpacifici merged commit 18cfd52 into main Jan 27, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants