Bypass locks for release packages builders on bb-* branches #900
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TLDR; Autobake builds on release branches can ignore Locks. This patch is an experiment for the next release (March-April). DO NOT MERGE BEFORE JANUARY RELEASE IS DONE. Long-Story: Although the bb-*-release release branches have priority when being selected from the queue, they are often blocked in the acquiring locks status, a mechanism meant to prevent overloading a host. As a result, in most cases we need to manually stop other builds, thus freeing the hosts and allowing a release build to start. This is not guaranteed, however, because Buildbot may select a completely different builder to start on that host, often a non-release build, which causes the release build of interest to remain stuck in the same status. This patch aims to remove this limitation by allowing builds that produce packages (autobake) to start on configured hosts regardless of the Locks value. The risk of this patch is obvious: there is a possibility of overloading the host and causing the build to fail. We can, however, try this approach and rollback during the next release if we discover that the failure rate caused by overloading the host exceeds the benefifs. I am also relying on the fact that autobake builders are not memory-intensive but mostly CPU-bound, with parallel execution occurring only during the build phase, not when the packages are created.
RazvanLiviuVarzaru
force-pushed
the
release-locks
branch
from
f422971 to
5d4de7d
Compare
Collaborator
Author
|
@fauust and @grooverdan I was mostly motivated by this. This (patch) is the least I can think of for now, because the other obvious action is to scale up the infra. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TLDR;
Autobake builds on release branches can ignore Locks. This patch is an experiment for the next release (March-April). DO NOT MERGE BEFORE JANUARY RELEASE IS DONE.
Long-Story:
Although the
bb-*-releaserelease branches have priority when being selected from the queue, they are often blocked in the acquiring locks status, a mechanism meant to prevent overloading a host.As a result, in most cases we need to manually stop other builds, thus freeing the hosts and allowing a release build to start. This is not guaranteed, however, because Buildbot may select a completely different builder to start on that host, often a non-release build, which causes the release build of interest to remain stuck in the same status.
This patch aims to remove this limitation by allowing builds that produce packages (autobake) to start on configured hosts regardless of the Locks value.
The risk of this patch is obvious: there is a possibility of overloading the host and causing the build to fail.
We can, however, try this approach and rollback during the next release if we discover that the failure rate caused by overloading the host exceeds the benefifs.
I am also relying on the fact that autobake builders are not memory-intensive but mostly CPU-bound, with parallel execution occurring only during the build phase, not when the packages are created.