Skip to content

Conversation

@remi-or
Copy link
Collaborator

@remi-or remi-or commented Jan 28, 2026

This PR refactors all of the logic linked to the inputs and outputs of continuous batching into a new class ContinuousBatchingIOs itself in a new file. This will be quite useful when we introduce async scheduling, because there will be two sets of host-side IOs and one set of device-side IOs.
To avoid having this major refactoring AND the async logic in the same PR, I think it would be good if we can first merge this part, which does not change much, and then add the async logic.
This PR also makes the test file ty compatible and adds a bit more documentation.

@remi-or remi-or requested a review from ArthurZucker January 28, 2026 17:28
@remi-or remi-or self-assigned this Jan 28, 2026
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants