feat: add support for concurrent plugin installs#4491
Open
thepetk wants to merge 6 commits intoredhat-developer:mainfrom
Open
feat: add support for concurrent plugin installs#4491thepetk wants to merge 6 commits intoredhat-developer:mainfrom
thepetk wants to merge 6 commits intoredhat-developer:mainfrom
Conversation
Contributor
|
The container image build workflow finished with status: |
Author
FYI I've tried to run a test instance of the |
Member
|
/cc |
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Description
Tries to allow concurrency on the plugin installation phase. The dynamic plugin installer so far allowed only sequential installation.
According to my benchmarks it can reduce 2x or 3x the time needed to install the plugins for your RHDH instance.
Key changes:
ThreadPoolExecutorwhich defaults to 5 workers, but that's configurable viaMAX_CONCURRENT_INSTALLS.skopeo inspectin parallel and use cached values in the sequential loop._oci_path_cachedict avoids redundant inspects across plugins from the same image.threading.Lockguards all concurrent writes toplugin_path_by_hash.PluginStatusDisplayjust to help the user read more easy what is happening.Which issue(s) does this PR fix
PR acceptance criteria
Please make sure that the following steps are complete:
How to test changes / Special notes to the reviewer
I've tested this both locally but also on a custom image I've built. I'll need for sure some guidance if futhre testing is required from my side.
Note, that intentionally I didn't want to do a lot of structural changes, like removing the skopeo dependency cause I feel this set of changes is more straightforward and with less risk involved. Looking at the benchmarks I feel it worths the effort.
For sure according to my experiments, removing the skopeo dependency will make this process even faster, mostly because the skopeo copy needs to fetch everything while we're ok downloading only a part of each image.