Skip to content

Conversation

@Zeitsperre
Copy link
Member

@Zeitsperre Zeitsperre commented Sep 30, 2025

Pull Request Checklist:

What kind of change does this PR introduce?

  • Shortens the modelling example in order to reduce the processing time spent running the Perform_a_climate_change_impact_study_on_a_watershed.ipynb notebook
  • Fixes a broken test that was not adjusted in Use pooch #513

Does this PR introduce a breaking change?

Not really.

Other information:

The model example is not as accurate now, seeing as the training period has been shortened from 10 years to 4 years. Unfortunately, if we want to run this notebook in our CI pipelines, we can't have notebook cells that might run for 10+ minutes. @huard Can you double-check that the outputs are still relatively accurate with this adjustment?

@Zeitsperre Zeitsperre requested review from huard and tlvu September 30, 2025 18:28
@Zeitsperre Zeitsperre self-assigned this Sep 30, 2025
@Zeitsperre Zeitsperre added bug Something isn't working documentation Improvements or additions to documentation labels Sep 30, 2025
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@github-actions github-actions bot added the docs label Sep 30, 2025
@huard
Copy link
Collaborator

huard commented Sep 30, 2025

Just add in the intro a note about the fact that the periods are shorter than what appears in the paper to speed up the notebook's computations.

Copy link
Collaborator

@tlvu tlvu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rubber stamp for me. Did you test directly on CRIM pipeline because this notebook only fail on CRIM pipeline. The original version that was supposed to be too heavy was still passing on our Jenkins.

And this notebook did not change recently and was passing before on CRIM pipeline. Did you find out why suddenly it becomes "too heavy" since the newer RavenPy or RavenWPS refactoring with Pooch? Did we accidentally introduced a performance regression?

@tlvu
Copy link
Collaborator

tlvu commented Sep 30, 2025

Woo, all the unit test failed ! "tests/test_external_dataset_access.py::TestGet::test_get_CASPAR_dataset - ImportError: HTTPFileSystem requires "requests" and "aiohttp" to be installed" This must be existing before this change?

@Zeitsperre
Copy link
Member Author

Zeitsperre commented Sep 30, 2025

Did you find out why suddenly it becomes "too heavy" since the newer RavenPy or RavenWPS refactoring with Pooch? Did we accidentally introduced a performance regression?

There aren't any changes within RavenPy that would affect the running of this particular notebook. pooch isn't used for any of the running code there, and testing data isn't grabbed from GitHub.

Woo, all the unit test failed ! "tests/test_external_dataset_access.py::TestGet::test_get_CASPAR_dataset - ImportError: HTTPFileSystem requires "requests" and "aiohttp" to be installed" This must be existing before this change?

I think what may have happened was that either a new release of xarray may have removed these libraries or a new release of h5netcdf may have added DAP-like fetching to its API (which would require Python HTTP request-supporting libs). Not all unit tests are failing (Python3.10 on tox is passing).

@tlvu
Copy link
Collaborator

tlvu commented Sep 30, 2025

Did you find out why suddenly it becomes "too heavy" since the newer RavenPy or RavenWPS refactoring with Pooch? Did we accidentally introduced a performance regression?

There aren't any changes within RavenPy that would affect the running of this particular notebook. pooch isn't used for any of the running code there, and testing data isn't grabbed from GitHub.

Then either CRIM Jenkins got slower or this notebook hits our PAVICS prod for data and we are slower to respond over the network? It passes on our side because the data is "local", network round-trip is minimal.

What kind of data is this notebook accessing on our PAVICS prod? Maybe some .ncml we recently changed?

@Zeitsperre
Copy link
Member Author

Then either CRIM Jenkins got slower or this notebook hits our PAVICS prod for data and we are slower to respond over the network? It passes on our side because the data is "local", network round-trip is minimal.

My money is on the throughput speed may be slower; Running the original example on my home workstation had some cells hanging for five to 10 minutes. nbconvert is even slower for some reason (affecting the second pass when running on Jenkins; Francis already mentioned this in Ouranosinc/PAVICS-e2e-workflow-tests#154)

What kind of data is this notebook accessing on our PAVICS prod? Maybe some .ncml we recently changed?

I don't think so. It's pulling CASPAR data that hasn't been touched in years. Has the THREDDS version or twitcher version changed?

@tlvu
Copy link
Collaborator

tlvu commented Sep 30, 2025

nbconvert is even slower for some reason (affecting the second pass when running on Jenkins; Francis already mentioned this in Ouranosinc/PAVICS-e2e-workflow-tests#154)

That's weird, I do not understand why Francis even mentionned nbconvert because that step is skipped by default on his Pipeline ! nbconvert is enable only if we manually trigger a build, not for builds triggered by the pipeline.

Has the THREDDS version or twitcher version changed?

Not recently on our side. Oh but twitcher changed on the tip of master which the pipeline is testing !

@Zeitsperre
Copy link
Member Author

@fmigneault Can you verify that this branch is working better for your infra? I still have a handful of small documentation changes to make, but it should run without timing out now.

@Zeitsperre
Copy link
Member Author

@fmigneault I'll be merging this in an hour or so. LMK if you want more time to verify changes on CRIM infra.

@Zeitsperre Zeitsperre merged commit 2a2418e into main Oct 6, 2025
19 checks passed
@Zeitsperre Zeitsperre deleted the shorter-time-period branch October 6, 2025 18:23
@fmigneault
Copy link

@Zeitsperre Sorry. I did not have enough time to look into it. Ok to have it merged. We'll see how the CI reacts using them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working docs documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RavenPy paper notebook timeout on CRIM / birdhouse-deploy CI

5 participants