-
Notifications
You must be signed in to change notification settings - Fork 250
Switch CI & Documentation Build to Micromamba from Conda for Reduced Execution Time #2366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch CI & Documentation Build to Micromamba from Conda for Reduced Execution Time #2366
Conversation
|
The debug level logging in the call to provision-micromamba should also be removed once the rest of this looks operational. |
Codecov Report
@@ Coverage Diff @@
## main #2366 +/- ##
==========================================
- Coverage 48.25% 47.57% -0.68%
==========================================
Files 110 110
Lines 30616 30612 -4
Branches 7982 7982
==========================================
- Hits 14774 14564 -210
- Misses 14298 14532 +234
+ Partials 1544 1516 -28
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
I also propose that we run the CI on a schedule using GitHub actions scheduling. There have been a couple instances in the past couple weeks where the CI was failing on new PRs even when it was passing on main, only because it had not been run in months. We could do it with this simple addition to the script: which would run the CI every Monday-Friday at 8 am UTC (3 am EST) |
the micromamba configuration does not allow the version of python to be specified, so maybe we can set it up using a different action version. I suspect that incorrect python version is what caused the CI to fail to resolve the conda environment on the previous commit
108ef2a to
07c2185
Compare
the micromamba configuration does not allow the version of python to be specified, so maybe we can set it up using a different action version. I suspect that incorrect python version is what caused the CI to fail to resolve the conda environment on the previous commit
07c2185 to
5374c91
Compare
to find out about issues with greater frequency, this will force the CI to run on the main branch (where we are currently doing development) every weekday early in the morning
|
I have added the mamba build to the documentation action, as well as the runner schedule to the CI action. I did a comparison between the environment resolved by micromamba and conda (see it for yourself here and it resolves almost everything to be the same, with the exception of some operating system level packages that I assume are a function of the action being run on a different machine + VM at GitHub every time. All of our 'major' dependencies (those that we maintain or use heavily) are resolved to the correct major versions. Tests are still passing and this cuts ~30 minutes off CI and Documentation build. |
.github/workflows/CI.yml
Outdated
| channels: defaults,rmg,conda-forge | ||
| channel-priority: flexible | ||
| log-level: debug | ||
| log-level: warning |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we reset log level to warning here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you squash this with the commit that set it to debug?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debug writes more to stdout than is useful for any future test running (see this run with debug) -- I only turned it on to deal with the issue of channel-priority. The default log level is warning
.github/workflows/CI.yml
Outdated
| schedule: | ||
| # * is a special character in YAML so you have to quote this string | ||
| - cron: '0 0 8 ? * MON,TUE,WED,THU,FRI *' | ||
| - cron: '0 8 * * 1-5' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you squash this commit with the previous one?
hwpang
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR! Overall seems good with a few comments. Several of the commits can be squashed together. I check the fork, and it looks like it fails Trigger RMG-tests?
I left a quick comment about the logging level. Will squash after it is resolved. That action fails on forked repositories regardless of content. See here, where I changed nothing in the source code and it still fails at the same location. |
- all depdencies which are package by others should be used instead of those packaged by us - small formatting change in github workflow file
libgfortran-ng was included in one of the packages but is now missing, add the conda-forge package
remove packages that appear to be unused, change those which can be drawn from conda-forge or other official changes where possible
|
Moved to #2386 |
Motivation or Problem
Resolves #2365
Description of Changes
Changes the GitHub CI to use micromamba instead of conda, which resolves the RMG environment in ~2 mins instead of 30+
Testing
Successfully ran on this fork.
Reviewer Tips
The fact that it runs is perhaps sufficient evidence. Might want to do a line-by-line comparison of how the depdencies were resolved to ensure that versions are consistent with conda.
The changes, if good, will also need to be ported into the documentation builder.