kafka adapter: #637
Merged
kafka adapter: #637
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
robambalu
commented
Nov 28, 2025
robambalu
commented
- Major chagne is to allow multiple consumers / consumer threads to service the same topic
- Each consumer can now get assigned a subset of partitions for a topic. Consumers need to keep track of their own partitions
- Since we dont know which keys are on which partition, the subscriber map is moved from Consumer to AdapterManager which will have the full set of subscribers for a given topic
- Removed complexity of not subscribing to Rebalance events if not using start offset, always sub and wait for EOF even if not using start offset
- exposed kafka consumer and producer specific config options
robambalu
requested review from
AdamGlustein,
alexddobkin and
svatasoiu
as code owners
robambalu
force-pushed
the
rba/kafka_optim
branch
from
56c94d0 to
45caa41
Compare
Collaborator
|
Should we get #8 over the line so we can include the Kafka tests as part of CI? |
Collaborator
Author
Yes absolutely, I didnt even realize the kafka test was completely disabled. Its heavily relied upon and should be tested |
NeejWeej
reviewed
Nov 28, 2025
NeejWeej
reviewed
Nov 28, 2025
timkpaine
added
type: enhancement
NeejWeej
reviewed
Nov 28, 2025
robambalu
force-pushed
the
rba/kafka_optim
branch
3 times, most recently
from
3b949a9 to
e7fb0ce
Compare
NeejWeej
previously approved these changes
Dec 1, 2025
AdamGlustein
reviewed
Dec 3, 2025
robambalu
force-pushed
the
rba/kafka_optim
branch
from
e7fb0ce to
fab3df9
Compare
- Major change is to allow multiple consumers / consumer threads to service the same topic - Each consumer can now get assigned a subset of partitions for a topic. Consumers need to keep track of their own partitions - Since we dont know which keys are on which partition, the subscriber map is moved from Consumer to AdapterManager which will have the full set of subscribers for a given topic - Removed complexity of not subscribing to Rebalance events if not using start offset, always sub and wait for EOF even if not using start offset - exposed kafka consumer and producer specific config options Signed-off-by: Rob Ambalu <robert.ambalu@point72.com>
robambalu
force-pushed
the
rba/kafka_optim
branch
from
fab3df9 to
95dbfc8
Compare
AdamGlustein
previously approved these changes
Dec 3, 2025
robambalu
force-pushed
the
rba/kafka_optim
branch
from
5dd79f0 to
07576a6
Compare
AdamGlustein
approved these changes
Dec 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.