Skip to content

expose the queue limit when registering a backend in the JobManager. #851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VictorVerhaert wants to merge 1 commit into from
Closed

expose the queue limit when registering a backend in the JobManager. #851

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions openeo/extra/job_management/_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,7 @@ def add_backend(
name: str,
connection: Union[Connection, Callable[[], Connection]],
parallel_jobs: int = 2,
queueing_limit: int = 10,
):
"""
Register a backend with a name and a :py:class:`Connection` getter.
Expand All @@ -246,6 +247,8 @@ def add_backend(
Either a Connection to the backend, or a callable to create a backend connection.
:param parallel_jobs:
Maximum number of jobs to allow in parallel on a backend.
:param queueing_limit:
Maximum number of jobs to allow in queue on a backend. Setting this too high may lead to a backend rejecting jobs.
Copy link
Member

@soxofaan soxofaan Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Maximum number of jobs to allow in queue on a backend. Setting this too high may lead to a backend rejecting jobs.
Maximum number of queued jobs (on the backend) the job manager should stay below before starting more jobs. Setting this too high may lead to the backend rejecting jobs.

"""

# TODO: Code might become simpler if we turn _Backend into class move this logic there.
Expand All @@ -255,8 +258,9 @@ def add_backend(
c = connection
connection = lambda: c
assert callable(connection)
# TODO: expose queueing_limit?
self.backends[name] = _Backend(get_connection=connection, parallel_jobs=parallel_jobs, queueing_limit=10)
self.backends[name] = _Backend(
get_connection=connection, parallel_jobs=parallel_jobs, queueing_limit=queueing_limit
)

def _get_connection(self, backend_name: str, resilient: bool = True) -> Connection:
"""Get a connection for the backend and optionally make it resilient (adds retry behavior)
Expand Down