Skip to content

feat: add support for concurrent plugin installs#4491

Open
thepetk wants to merge 6 commits intoredhat-developer:mainfrom
thepetk:support-concurrent-plugin-installs
Open

feat: add support for concurrent plugin installs#4491
thepetk wants to merge 6 commits intoredhat-developer:mainfrom
thepetk:support-concurrent-plugin-installs

Conversation

@thepetk
Copy link
Copy Markdown

@thepetk thepetk commented Mar 27, 2026

Description

Tries to allow concurrency on the plugin installation phase. The dynamic plugin installer so far allowed only sequential installation.

According to my benchmarks it can reduce 2x or 3x the time needed to install the plugins for your RHDH instance.

Key changes:

  • Plugin installation now runs in a ThreadPoolExecutor which defaults to 5 workers, but that's configurable via MAX_CONCURRENT_INSTALLS.
  • The OCI manifest inspection process is now pre-fetched and outside of the merge loop, as the merge loop has to be sequential. That allows us to run the skopeo inspect in parallel and use cached values in the sequential loop.
  • For the above we use a shared _oci_path_cache dict avoids redundant inspects across plugins from the same image.
  • A threading.Lock guards all concurrent writes to plugin_path_by_hash.
  • As the previous logging flow would not work with concurrent installs I've added PluginStatusDisplay just to help the user read more easy what is happening.

Which issue(s) does this PR fix

PR acceptance criteria

Please make sure that the following steps are complete:

  • GitHub Actions are completed and successful
  • Unit Tests are updated and passing
  • E2E Tests are updated and passing
  • Documentation is updated if necessary (requirement for new features)
  • Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

I've tested this both locally but also on a custom image I've built. I'll need for sure some guidance if futhre testing is required from my side.

Note, that intentionally I didn't want to do a lot of structural changes, like removing the skopeo dependency cause I feel this set of changes is more straightforward and with less risk involved. Looking at the benchmarks I feel it worths the effort.

For sure according to my experiments, removing the skopeo dependency will make this process even faster, mostly because the skopeo copy needs to fetch everything while we're ok downloading only a part of each image.

@thepetk thepetk changed the title Add support for concurrent plugin installs feat: Add support for concurrent plugin installs Mar 27, 2026
@thepetk thepetk changed the title feat: Add support for concurrent plugin installs feat: add support for concurrent plugin installs Mar 27, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions
Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions
Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions
Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@thepetk
Copy link
Copy Markdown
Author

thepetk commented Mar 30, 2026

Image was built and published successfully. It is available at:

FYI I've tried to run a test instance of the ai-rolling-demo-gitops using the quay.io/rhdh-community/rhdh:pr-4491 image. The comparison between this one and quay.io/rhdh/rhdh-hub-rhel9/rhdh:1.10-87 can be seen below. The suggested approach took 1.19 mins to end while the current one finished after 3.25 mins.

Init Containers:
  install-dynamic-plugins:
    Container ID: 
...
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 30 Mar 2026 11:10:30 +0100
      Finished:     Mon, 30 Mar 2026 11:11:49 +0100

Init Containers:
  install-dynamic-plugins:
    Container ID: 
...
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 30 Mar 2026 11:39:02 +0100
      Finished:     Mon, 30 Mar 2026 11:42:27 +0100

@rm3l
Copy link
Copy Markdown
Member

rm3l commented Mar 30, 2026

/cc

@openshift-ci openshift-ci bot requested a review from rm3l March 30, 2026 11:58
@github-actions
Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants