Skip to content

Add payu test suite for spatial configuration #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
abhaasgoyal merged 10 commits into from
Feb 29, 2024
Merged

Add payu test suite for spatial configuration #136

abhaasgoyal merged 10 commits into from
Feb 29, 2024

Conversation

@SeanBryan51
Copy link
Collaborator

@SeanBryan51 SeanBryan51 commented Sep 12, 2023 •
edited
Loading

Spatial tests use the payu framework. The payu framework was chosen so that we:

  • Encourage uptake of payu amongst users of CABLE
  • Have the foundations in place for running coupled models (atmosphere + land) with payu
  • Can easily test longer running simulations (payu makes it easy to run a model multiple times and have state persist in the model via restart files)

The design of the spatial tests assumes each payu experiment is tailored to running CABLE with a specific meteorological forcing. This has the benefit that all the required inputs are already defined in the payu configuration file. An alternative would be to build up the spatial namelist configurations from scratch. This would be problematic as it is unclear if CABLE requires 'forcing specific' namelist options to be enabled to run with a particular met forcing. That is, CABLE does not allow for easy plug and play with different met forcings via the namelist file.

The run directory structure is organised as follows:

runs/
├── spatial
│   └── tasks
│	├── <spatial-task-name> (a payu control / experiment directory)
│	└── ...
├── payu-laboratory
│   └── ...
└── fluxsite
    └── ...

Note we have a separate payu-laboratory directory. This is so we keep all CABLE outputs produced by benchcab under the bench_example work directory.

This change includes the following additional features:

  • Add the ability to build the CABLE executable with MPI at runtime so that we run the spatial configurations with MPI.
  • Add the --mpi flag to benchcab build command so that the user can run the MPI build step independently.
  • Add subcommands to run each step of the spatial workflow in isolation.
  • Add payu key in the benchcab config file so that users can easily configure payu experiments and add optional command line arguments to the payu run command.
  • Add met_forcings key to specify different met forcings and their respective payu experiment.

Fixes #5

@SeanBryan51 SeanBryan51 linked an issue Sep 12, 2023 that may be closed by this pull request
Add handling of spatial simulations #5
Closed
8 tasks
@codecov
Copy link

codecov bot commented Sep 12, 2023 •
edited
Loading

Codecov Report

Attention: Patch coverage is 85.42199% with 57 lines in your changes are missing coverage. Please review.

Project coverage is 63.50%. Comparing base (623c0a3) to head (ce3608e).
Report is 1 commits behind head on main.

Files Patch % Lines
benchcab/benchcab.py 20.37% 43 Missing ⚠️
benchcab/spatial.py 82.35% 12 Missing ⚠️
tests/test_spatial.py 98.03% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #136      +/-   ##
==========================================
+ Coverage   61.15%   63.50%   +2.35%     
==========================================
  Files          30       35       +5     
  Lines        2260     2521     +261     
==========================================
+ Hits         1382     1601     +219     
- Misses        878      920      +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@SeanBryan51
Copy link
Collaborator Author

SeanBryan51 commented Sep 12, 2023 •
edited
Loading

It looks like the build script will rename the serial executable from cable to cable.dd.mm.yy if we run the MPI build after the serial build. Benchcab then complains it cannot find the serial executable:

FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tm70/sb8430/bench_example/src/trunk/offline/cable'

Ideally we should not rename the file if we are building an MPI executable i.e. cable-mpi as this would not overwrite the existing serial executable.

It might be worth considering working directly with Makefile rather than the build script? This would resolve a lot of the headaches around caching serial and MPI builds, adding compiler flags for code coverage generation, switching between serial and MPI, building serial and MPI in parallel, etc.

Edit: this has been resolved in #139

@SeanBryan51 SeanBryan51 mentioned this pull request Sep 12, 2023
Closed
@SeanBryan51 SeanBryan51 force-pushed the 5-spatial-testing branch 2 times, most recently from 45ff91a to 72aad1e Compare September 20, 2023 06:29
@SeanBryan51 SeanBryan51 marked this pull request as ready for review September 20, 2023 06:33
@SeanBryan51 SeanBryan51 force-pushed the 5-spatial-testing branch 3 times, most recently from 6fd8a3b to 659c1d0 Compare September 28, 2023 05:48
@SeanBryan51 SeanBryan51 changed the title Spatial testing Add payu test suite for spatial configuration Sep 28, 2023
@SeanBryan51 SeanBryan51 force-pushed the 5-spatial-testing branch 3 times, most recently from 9e2df50 to 851cb65 Compare October 2, 2023 22:56
@SeanBryan51
Copy link
Collaborator Author

SeanBryan51 commented Oct 23, 2023

After discussing with @ccarouge, we should hold off on specifying the spatial met forcing paths in the namelist file using the standard structure for now, and instead use the namelist file in the payu experiment which already contains the met forcing paths. This is because it is still unclear if CABLE requires additional namelist options to be switched on to run with a particular met forcing, i.e. CABLE does not allow for easy plug and play with different met forcings.

@SeanBryan51
Copy link
Collaborator Author

SeanBryan51 commented Oct 24, 2023

Since we are going to be forcing CABLE with more than one spatial met forcing, I'm adding a met_forcings key which specifies the name of forcing and a URL pointing to its configured payu experiment so that the user can specify which spatial met forcings they want to run:

spatial:
  met_forcings:
    - name: crujra_access
      payu_experiment: https://github.com/CABLE-LSM/cable_example.git
    - ...

Benchcab will have a default value for the met_forcings key that is defined internally similar to science_configurations.

For each item in the list, the name and payu_experiment keys are required.

If the met_forcings key is specified, it cannot be empty.

@ccarouge
Copy link
Member

ccarouge commented Oct 25, 2023

@SeanBryan51 Why are we placing payu-laboratory/ outside of spatial/?

@SeanBryan51
Copy link
Collaborator Author

SeanBryan51 commented Oct 25, 2023

@ccarouge my thinking behind that was if we want to have another configuration that runs with payu, e.g. online cable, we can use the same custom payu laboratory.

@SeanBryan51
Copy link
Collaborator Author

SeanBryan51 commented Oct 25, 2023 •
edited
Loading

  • Update documentation for config parameters: spatial, payu, config, args, met_forcings, name, payu_experiment
  • Move the experiment key under the fluxsite key since experiment is specific to fluxsite tests. See Move experiment key under fluxsite #192

@ccarouge
Copy link
Member

ccarouge commented Nov 2, 2023

There is one major change I'd like to see so I'm posting it before finishing the review. I'm not a fan of:

spatial:
  met_forcings:
    - name: crujra_access
      payu_experiment: https://github.com/CABLE-LSM/cable_example.git
    - ...

I would rather see:

spatial:
  met_forcings:
    - crujra_access: https://github.com/CABLE-LSM/cable_example.git
    - ...

So met_forcings would be a dictionary with:

  • key: name of the forcing
  • value: payu repository

Otherwise, it feels like there is one unnecessary level of complexity.
I don't think we need to keep track of the met. forcings order and Python's dictionaries are now always ordered anyway.

ccarouge
ccarouge requested changes Nov 2, 2023
View reviewed changes
Copy link
Member

@ccarouge ccarouge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few other changes in here in addition to the change to config.yaml.

Also, this is very long. It might be nice next time if this could be done in steps. Some ideas of smaller PRs that may have been good here:

  • renaming Task to Fluxsite
  • setting the command design
  • setting up the work directory structure
  • moving the utility functions
  • the Spatial class

I realise this does not necessarily correspond to the way the development went, so it might have been separated differently. But this is definitely long. This means I may have missed something. For example, I assumed you haven't modified anything in the utility functions you moved or their test. So I didn't review their code.

@SeanBryan51 SeanBryan51 mentioned this pull request Nov 6, 2023
Closed
@SeanBryan51
Copy link
Collaborator Author

SeanBryan51 commented Nov 6, 2023 •
edited
Loading

@ccarouge thanks for the comments. Sorry about the size of the PR. I guess my problem is that I'm always tweaking things during development so nothing is really set in stone until the entire feature is implemented, which results in a large PR.

Also, this is very long. It might be nice next time if this could be done in steps. Some ideas of smaller PRs that may have been good here:

  • renaming Task to Fluxsite
  • setting the command design
  • setting up the work directory structure
  • moving the utility functions
  • the Spatial class

Do you see each of these "small" PRs being merged into main separately?

@ccarouge
Copy link
Member

ccarouge commented Nov 7, 2023 •
edited
Loading

@ccarouge thanks for the comments. Sorry about the size of the PR. I guess my problem is that I'm always tweaking things during development so nothing is really set in stone until the entire feature is implemented, which results in a large PR.

Also, this is very long. It might be nice next time if this could be done in steps. Some ideas of smaller PRs that may have been good here:

  • renaming Task to Fluxsite
  • setting the command design
  • setting up the work directory structure
  • moving the utility functions
  • the Spatial class

Do you see each of these "small" PRs being merged into main separately?

I know we discussed this in meeting but just to keep a trace of it.
I don't have any problem with any of these small PRs being merged into main. The most problematic one in my view is setting up the commands while they don't do anything. They could be setup to return a "not implemented yet", or even better if that was in the help message as well. When implementing them, the documentation could then be updated with a big warning admonition this isn't available yet for use.

By the way, this makes me think that the documentation of all this isn't in this PR. So the PR is big and not complete 😄 I'm not asking to add to it, just showing that next time we need to discuss more about how to split this up.

SeanBryan51 added a commit that referenced this pull request Nov 9, 2023
@SeanBryan51
Specify met_forcings key as a dictionary
719566c 
Changes requested by @ccarouge: #136 (comment)
ccarouge
ccarouge approved these changes Dec 11, 2023
View reviewed changes
Copy link
Member

@ccarouge ccarouge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small change to put in the documentation. The vocabulary used in this case is important.

SeanBryan51 and others added 5 commits January 2, 2024 12:07
@SeanBryan51
Add payu test suite for spatial configuration
cb5c5cc 
Spatial tests use the [payu framework][payu]. The payu framework was
chosen so that we:
- Encourage uptake of payu amongst users of CABLE
- Have the foundations in place for running coupled models (atmosphere +
  land) with payu
- Can easily test longer running simulations (payu makes it easy to run
  a model multiple times and have state persist in the model via restart
  files)

The design of the spatial tests assumes each payu experiment is tailored
to running CABLE with a specific meteorological forcing. This has the
benefit that all the required inputs are already defined in the payu
configuration file. An alternative would be to build up the spatial
namelist configurations from scratch. This would be problematic as it is
unclear if CABLE requires 'forcing specific' namelist options to be
enabled to run with a particular met forcing. That is, CABLE does not
allow for easy plug and play with different met forcings via the
namelist file.

The run directory structure is organised as follows:

runs/
├── spatial
│   └── tasks
│	├── <spatial-task-name> (a payu control / experiment directory)
│	└── ...
├── payu-laboratory
│   └── ...
└── fluxsite
    └── ...

Note we have a separate payu-laboratory directory. This is so we keep
all CABLE outputs produced by benchcab under the bench_example work
directory.

This change includes the following additional features:
- Add the ability to build the CABLE executable with MPI at runtime so
  that we run the spatial configurations with MPI.
- Add the --mpi flag to benchcab build command so that the user can run
  the MPI build step independently.
- Add subcommands to run each step of the spatial workflow in isolation.
- Add payu key in the benchcab config file so that users can easily
  configure payu experiments and add optional command line arguments to
  the payu run command.
- Add met_forcings key to specify different met forcings and their
  respective payu experiment.

Fixes #5

[payu]: https://github.com/payu-org/payu
[cable_example]: https://github.com/CABLE-LSM/cable_example
@SeanBryan51
Use one branch in integration test
2b9ee09 
This is to reduce the compilation work as we now compile the serial and
MPI executable.
@SeanBryan51
Remove repeated content
80ecad7
@SeanBryan51
Add section on supported configurations
58e08b6
@SeanBryan51 @ccarouge
Update README.md
2797746 
Co-authored-by: Claire Carouge <ccarouge@users.noreply.github.com>
abhaasgoyal
abhaasgoyal reviewed Jan 31, 2024
View reviewed changes
ccarouge
ccarouge approved these changes Jan 31, 2024
View reviewed changes
Copy link
Member

@ccarouge ccarouge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tested the integration test? Can you paste the std output and the PBS log in here to keep a trace that the integration test ran everything without an error?

@abhaasgoyal
Copy link

abhaasgoyal commented Feb 2, 2024 •
edited
Loading

Std output

(benchcab-dev-payu) [ag9761@gadi-login-08 benchcab]$ /bin/bash benchcab/data/test/integration.sh
+ TEST_DIR=/scratch/tm70/ag9761/benchcab/integration
+ EXAMPLE_REPO=git@github.com:CABLE-LSM/bench_example.git
+ rm -rf /scratch/tm70/ag9761/benchcab/integration
+ mkdir -p /scratch/tm70/ag9761/benchcab/integration
+ git clone git@github.com:CABLE-LSM/bench_example.git /scratch/tm70/ag9761/benchcab/integration
Cloning into '/scratch/tm70/ag9761/benchcab/integration'...
remote: Enumerating objects: 86, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 86 (delta 5), reused 9 (delta 3), pack-reused 58
Receiving objects: 100% (86/86), 24.67 KiB | 2.06 MiB/s, done.
Resolving deltas: 100% (32/32), done.
+ cd /scratch/tm70/ag9761/benchcab/integration
+ git reset --hard 6287539e96fc8ef36dc578201fbf9847314147fb
HEAD is now at 6287539 Merge pull request #7 from CABLE-LSM/dev
+ cat
+ benchcab run -v
Creating src directory
Checking out repositories...
git clone --branch main -- https://github.com/CABLE-LSM/CABLE.git src/main
Cloning into 'src/main'...
remote: Enumerating objects: 12637, done.
remote: Counting objects: 100% (11242/11242), done.
remote: Compressing objects: 100% (2633/2633), done.
remote: Total 12637 (delta 8743), reused 10985 (delta 8557), pack-reused 1395
Receiving objects: 100% (12637/12637), 120.61 MiB | 19.28 MiB/s, done.
Resolving deltas: 100% (9515/9515), done.
Successfully checked out main - commit 67a52dc5721f0da78ee7d61798c0e8a804dcaaeb
Writing revision number info to rev_number-1.log

Compiling CABLE serially for realisation main...
mkdir src/main/src/offline/.tmp
cp -p src/main/src/science/albedo/cbl_soilColour_albedo.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/albedo/cbl_albedo.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/albedo/cbl_snow_albedo.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/radiation/cbl_spitter.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/radiation/cbl_sinbet.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/radiation/cbl_init_radiation.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/radiation/cbl_radiation.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/radiation/cbl_rhoch.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_wetleaf.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_zetar.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_fwsoil.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_photosynthesis.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cable_canopy.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_LAI_canopy_height.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_friction_vel.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_latent_heat.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_SurfaceWetness.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_dryLeaf.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_qsat.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_within_canopy.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/canopy/cbl_pot_evap_snow.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_sumcflux.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_rplant.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_cnp.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_variable.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_inout.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/biogeochem_casa.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_phenology.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/bgcdriver.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_dimension.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_readbiome.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_feedback.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/casa-cnp/casa_param.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/gw_hydro/cable_psm.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/gw_hydro/cable_gw_hydro.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/misc/cable_climate.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/misc/cable_air.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/misc/cable_carbon.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/roughness/cable_roughness.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/roughness/roughnessHGT_effLAI_cbl.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_Oldconductivity.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_snowl_adjust.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_soilsnow_init_special.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_snowAccum.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_snowMelt.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_soilfreeze.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_snowDensity.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_thermal.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_smoisturev.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_conductivity.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_surfbv.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_snow_aging.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_stempv.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_hyd_redistrib.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_soilsnow_data.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_remove_trans.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_trimb.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_snowCheck.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_soilsnow_main.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/soilsnow/cbl_GW.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/landuse/landuse3.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/landuse/landuse_constant.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cbl_model_driver_offline.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_parameters.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_driver.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_site.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_mpimaster.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_initialise.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/landuse_inout.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_checks.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_output.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_define_types.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_metutils.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_read.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/casa_cable.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_mpicommon.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_input.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_mpiworker.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_namelist_input.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_LUC_EXPT.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/CASAONLY_LUC.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_mpidrv.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/spincasacnp.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_write.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_pft_params.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_abort.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/casa_ncdf.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/casa_offline_inout.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_weathergenerator.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_plume_mip.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_soil_params.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_iovars.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_phenology.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/cable_cru_TRENDY.F90 src/main/src/offline/.tmp
cp -p src/main/src/util/cable_climate_type_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/util/masks_cbl.F90 src/main/src/offline/.tmp
cp -p src/main/src/util/cable_runtime_opts_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/util/cable_common.F90 src/main/src/offline/.tmp
cp -p src/main/src/params/cable_params_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/params/cable_maths_constants_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/params/cable_other_constants_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/params/cable_photo_constants_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/params/grid_constants_cbl.F90 src/main/src/offline/.tmp
cp -p src/main/src/params/cable_phys_constants_mod.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/sli/cable_sli_solve.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/sli/cable_sli_roots.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/sli/cable_sli_numbers.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/sli/cable_sli_utils.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/sli/cable_sli_main.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/pop_types.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/pop_mpi.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/POP.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/pop_io.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/pop_constants.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/pop_def.F90 src/main/src/offline/.tmp
cp -p src/main/src/science/pop/POPLUC.F90 src/main/src/offline/.tmp
cp -p src/main/src/offline/Makefile src/main/src/offline/.tmp
Loading modules: intel-compiler/2021.1.1 netcdf/4.7.4 openmpi/4.1.0
make
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_define_types.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_def.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_constants.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_types.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c POP.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_dimension.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_param.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_variable.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c landuse_constant.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_phenology.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_iovars.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c grid_constants_cbl.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_pft_params.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_soil_params.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_runtime_opts_mod.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_common.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_cnp.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_rplant.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_ncdf.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_offline_inout.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_inout.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c biogeochem_casa.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_phenology.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c bgcdriver.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_abort.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_phys_constants_mod.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_air.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_friction_vel.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_qsat.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_pot_evap_snow.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_zetar.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_latent_heat.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_wetleaf.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_within_canopy.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_SurfaceWetness.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_other_constants_mod.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_photo_constants_mod.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_photosynthesis.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_fwsoil.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_dryLeaf.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_maths_constants_mod.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_spitter.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_rhoch.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_init_radiation.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_radiation.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c roughnessHGT_effLAI_cbl.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_roughness.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_numbers.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_utils.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_roots.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_solve.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_main.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilsnow_data.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowCheck.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowDensity.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowAccum.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowMelt.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowl_adjust.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_trimb.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_Oldconductivity.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_conductivity.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_GW.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_thermal.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_gw_hydro.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_psm.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_climate_type_mod.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_canopy.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_carbon.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_checks.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_climate.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_weathergenerator.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_cru_TRENDY.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_read.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_initialise.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_sinbet.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_LUC_EXPT.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_parameters.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c POPLUC.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_metutils.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_readbiome.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_input.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_namelist_input.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_write.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_output.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_plume_mip.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_site.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_cable.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_feedback.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c CASAONLY_LUC.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_sumcflux.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilColour_albedo.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snow_albedo.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_albedo.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_hyd_redistrib.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilfreeze.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_remove_trans.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_smoisturev.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_surfbv.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_stempv.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilsnow_main.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c masks_cbl.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snow_aging.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_model_driver_offline.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilsnow_init_special.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c landuse3.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c landuse_inout.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_io.F90
ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c spincasacnp.F90
ifort -warn nostderrors -diag-disable 10145 -O2 -fp-model precise -L/apps/netcdf/4.7.4/lib/Intel -O0 -lnetcdf -lnetcdff -o cable cable_driver.F90 bgcdriver.o biogeochem_casa.o cable_abort.o cable_air.o cable_canopy.o cable_carbon.o cable_checks.o cable_climate.o cable_climate_type_mod.o cable_common.o cable_cru_TRENDY.o cable_define_types.o cable_gw_hydro.o cable_initialise.o cable_input.o cable_iovars.o cable_LUC_EXPT.o cable_maths_constants_mod.o cable_metutils.o cable_namelist_input.o cable_other_constants_mod.o cable_output.o cable_parameters.o cable_pft_params.o cable_phenology.o cable_photo_constants_mod.o cable_phys_constants_mod.o cable_plume_mip.o cable_psm.o cable_read.o cable_roughness.o cable_runtime_opts_mod.o cable_site.o cable_sli_main.o cable_sli_numbers.o cable_sli_roots.o cable_sli_solve.o cable_sli_utils.o cable_soil_params.o cable_weathergenerator.o cable_write.o casa_cable.o casa_cnp.o casa_dimension.o casa_feedback.o casa_inout.o casa_ncdf.o casa_offline_inout.o CASAONLY_LUC.o casa_param.o casa_phenology.o casa_readbiome.o casa_rplant.o casa_sumcflux.o casa_variable.o cbl_albedo.o cbl_conductivity.o cbl_dryLeaf.o cbl_friction_vel.o cbl_fwsoil.o cbl_GW.o cbl_hyd_redistrib.o cbl_init_radiation.o cbl_latent_heat.o cbl_model_driver_offline.o cbl_Oldconductivity.o cbl_photosynthesis.o cbl_pot_evap_snow.o cbl_qsat.o cbl_radiation.o cbl_remove_trans.o cbl_rhoch.o cbl_sinbet.o cbl_smoisturev.o cbl_snowAccum.o cbl_snow_aging.o cbl_snow_albedo.o cbl_snowCheck.o cbl_snowDensity.o cbl_snowl_adjust.o cbl_snowMelt.o cbl_soilColour_albedo.o cbl_soilfreeze.o cbl_soilsnow_data.o cbl_soilsnow_init_special.o cbl_soilsnow_main.o cbl_spitter.o cbl_stempv.o cbl_SurfaceWetness.o cbl_surfbv.o cbl_thermal.o cbl_trimb.o cbl_wetleaf.o cbl_within_canopy.o cbl_zetar.o grid_constants_cbl.o landuse3.o landuse_constant.o landuse_inout.o masks_cbl.o pop_constants.o pop_def.o pop_io.o POPLUC.o POP.o pop_types.o roughnessHGT_effLAI_cbl.o spincasacnp.o -I/apps/netcdf/4.7.4/include/Intel
Unloading modules: intel-compiler/2021.1.1 netcdf/4.7.4 openmpi/4.1.0
mv src/main/src/offline/.tmp/cable src/main/src/offline/cable
Successfully compiled CABLE for realisation main

Compiling CABLE with MPI for realisation main...
mkdir src/main/src/offline/.mpitmp
cp -p src/main/src/science/albedo/cbl_soilColour_albedo.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/albedo/cbl_albedo.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/albedo/cbl_snow_albedo.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/radiation/cbl_spitter.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/radiation/cbl_sinbet.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/radiation/cbl_init_radiation.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/radiation/cbl_radiation.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/radiation/cbl_rhoch.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_wetleaf.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_zetar.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_fwsoil.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_photosynthesis.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cable_canopy.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_LAI_canopy_height.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_friction_vel.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_latent_heat.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_SurfaceWetness.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_dryLeaf.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_qsat.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_within_canopy.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/canopy/cbl_pot_evap_snow.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_sumcflux.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_rplant.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_cnp.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_variable.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_inout.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/biogeochem_casa.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_phenology.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/bgcdriver.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_dimension.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_readbiome.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_feedback.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/casa-cnp/casa_param.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/gw_hydro/cable_psm.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/gw_hydro/cable_gw_hydro.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/misc/cable_climate.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/misc/cable_air.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/misc/cable_carbon.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/roughness/cable_roughness.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/roughness/roughnessHGT_effLAI_cbl.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_Oldconductivity.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_snowl_adjust.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_soilsnow_init_special.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_snowAccum.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_snowMelt.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_soilfreeze.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_snowDensity.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_thermal.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_smoisturev.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_conductivity.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_surfbv.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_snow_aging.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_stempv.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_hyd_redistrib.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_soilsnow_data.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_remove_trans.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_trimb.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_snowCheck.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_soilsnow_main.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/soilsnow/cbl_GW.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/landuse/landuse3.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/landuse/landuse_constant.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cbl_model_driver_offline.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_parameters.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_driver.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_site.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_mpimaster.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_initialise.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/landuse_inout.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_checks.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_output.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_define_types.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_metutils.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_read.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/casa_cable.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_mpicommon.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_input.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_mpiworker.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_namelist_input.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_LUC_EXPT.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/CASAONLY_LUC.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_mpidrv.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/spincasacnp.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_write.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_pft_params.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_abort.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/casa_ncdf.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/casa_offline_inout.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_weathergenerator.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_plume_mip.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_soil_params.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_iovars.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_phenology.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/cable_cru_TRENDY.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/util/cable_climate_type_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/util/masks_cbl.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/util/cable_runtime_opts_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/util/cable_common.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/params/cable_params_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/params/cable_maths_constants_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/params/cable_other_constants_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/params/cable_photo_constants_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/params/grid_constants_cbl.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/params/cable_phys_constants_mod.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/sli/cable_sli_solve.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/sli/cable_sli_roots.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/sli/cable_sli_numbers.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/sli/cable_sli_utils.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/sli/cable_sli_main.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/pop_types.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/pop_mpi.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/POP.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/pop_io.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/pop_constants.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/pop_def.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/science/pop/POPLUC.F90 src/main/src/offline/.mpitmp
cp -p src/main/src/offline/Makefile src/main/src/offline/.mpitmp
Loading modules: intel-compiler/2021.1.1 netcdf/4.7.4 openmpi/4.1.0
make mpi
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_define_types.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_def.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_constants.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_types.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c POP.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_dimension.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_param.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_variable.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c landuse_constant.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_phenology.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_iovars.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c grid_constants_cbl.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_pft_params.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_soil_params.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_runtime_opts_mod.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_common.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_cnp.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_rplant.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_ncdf.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_offline_inout.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_inout.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c biogeochem_casa.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_phenology.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c bgcdriver.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_abort.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_phys_constants_mod.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_air.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_friction_vel.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_qsat.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_pot_evap_snow.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_zetar.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_latent_heat.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_wetleaf.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_within_canopy.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_SurfaceWetness.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_other_constants_mod.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_photo_constants_mod.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_photosynthesis.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_fwsoil.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_dryLeaf.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_maths_constants_mod.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_spitter.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_rhoch.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_init_radiation.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_radiation.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c roughnessHGT_effLAI_cbl.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_roughness.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_numbers.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_utils.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_roots.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_solve.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_sli_main.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilsnow_data.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowCheck.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowDensity.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowAccum.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowMelt.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snowl_adjust.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_trimb.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_Oldconductivity.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_conductivity.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_GW.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_thermal.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_gw_hydro.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_psm.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_climate_type_mod.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_canopy.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_carbon.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_checks.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_climate.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_weathergenerator.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_cru_TRENDY.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_read.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_initialise.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_sinbet.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_LUC_EXPT.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_parameters.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c POPLUC.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_metutils.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_readbiome.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_input.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_namelist_input.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_write.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_output.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_plume_mip.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_site.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_cable.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_feedback.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c CASAONLY_LUC.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_sumcflux.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilColour_albedo.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snow_albedo.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_albedo.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_hyd_redistrib.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilfreeze.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_remove_trans.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_smoisturev.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_surfbv.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_stempv.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilsnow_main.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c masks_cbl.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_snow_aging.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_model_driver_offline.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cbl_soilsnow_init_special.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c landuse3.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c landuse_inout.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_io.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c spincasacnp.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_mpicommon.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c pop_mpi.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_mpiworker.F90
mpif90 -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_mpimaster.F90
mpif90 -warn nostderrors -diag-disable 10145 -O2 -fp-model precise -L/apps/netcdf/4.7.4/lib/Intel -O0 -o cable-mpi cable_mpidrv.F90 cable_mpicommon.o cable_mpimaster.o cable_mpiworker.o pop_mpi.o bgcdriver.o biogeochem_casa.o cable_abort.o cable_air.o cable_canopy.o cable_carbon.o cable_checks.o cable_climate.o cable_climate_type_mod.o cable_common.o cable_cru_TRENDY.o cable_define_types.o cable_gw_hydro.o cable_initialise.o cable_input.o cable_iovars.o cable_LUC_EXPT.o cable_maths_constants_mod.o cable_metutils.o cable_namelist_input.o cable_other_constants_mod.o cable_output.o cable_parameters.o cable_pft_params.o cable_phenology.o cable_photo_constants_mod.o cable_phys_constants_mod.o cable_plume_mip.o cable_psm.o cable_read.o cable_roughness.o cable_runtime_opts_mod.o cable_site.o cable_sli_main.o cable_sli_numbers.o cable_sli_roots.o cable_sli_solve.o cable_sli_utils.o cable_soil_params.o cable_weathergenerator.o cable_write.o casa_cable.o casa_cnp.o casa_dimension.o casa_feedback.o casa_inout.o casa_ncdf.o casa_offline_inout.o CASAONLY_LUC.o casa_param.o casa_phenology.o casa_readbiome.o casa_rplant.o casa_sumcflux.o casa_variable.o cbl_albedo.o cbl_conductivity.o cbl_dryLeaf.o cbl_friction_vel.o cbl_fwsoil.o cbl_GW.o cbl_hyd_redistrib.o cbl_init_radiation.o cbl_latent_heat.o cbl_model_driver_offline.o cbl_Oldconductivity.o cbl_photosynthesis.o cbl_pot_evap_snow.o cbl_qsat.o cbl_radiation.o cbl_remove_trans.o cbl_rhoch.o cbl_sinbet.o cbl_smoisturev.o cbl_snowAccum.o cbl_snow_aging.o cbl_snow_albedo.o cbl_snowCheck.o cbl_snowDensity.o cbl_snowl_adjust.o cbl_snowMelt.o cbl_soilColour_albedo.o cbl_soilfreeze.o cbl_soilsnow_data.o cbl_soilsnow_init_special.o cbl_soilsnow_main.o cbl_spitter.o cbl_stempv.o cbl_SurfaceWetness.o cbl_surfbv.o cbl_thermal.o cbl_trimb.o cbl_wetleaf.o cbl_within_canopy.o cbl_zetar.o grid_constants_cbl.o landuse3.o landuse_constant.o landuse_inout.o masks_cbl.o pop_constants.o pop_def.o pop_io.o POPLUC.o POP.o pop_types.o roughnessHGT_effLAI_cbl.o spincasacnp.o -I/apps/netcdf/4.7.4/include/Intel -lnetcdf -lnetcdff
Unloading modules: intel-compiler/2021.1.1 netcdf/4.7.4 openmpi/4.1.0
mv src/main/src/offline/.mpitmp/cable-mpi src/main/src/offline/cable-mpi
Successfully compiled CABLE for realisation main

Setting up run directory tree for fluxsite tests...
Creating runs/fluxsite directory
Creating runs/fluxsite/logs directory
Creating runs/fluxsite/outputs directory
Creating runs/fluxsite/tasks directory
Creating runs/fluxsite/analysis directory
Creating runs/fluxsite/analysis/bitwise-comparisons directory
Setting up tasks...
Setting up task: AU-Tum_2002-2017_OzFlux_Met_R0_S0
Creating runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0 directory
  Cleaning task
  Copying namelist files from namelists to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0
  Copying CABLE executable from src/main/src/offline/cable to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/cable
  Adding base configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/cable.nml
  Adding science configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/cable.nml
Setting up task: AU-Tum_2002-2017_OzFlux_Met_R0_S1
Creating runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1 directory
  Cleaning task
  Copying namelist files from namelists to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1
  Copying CABLE executable from src/main/src/offline/cable to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/cable
  Adding base configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/cable.nml
  Adding science configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/cable.nml
Setting up task: AU-Tum_2002-2017_OzFlux_Met_R0_S2
Creating runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2 directory
  Cleaning task
  Copying namelist files from namelists to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2
  Copying CABLE executable from src/main/src/offline/cable to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/cable
  Adding base configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/cable.nml
  Adding science configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/cable.nml
Setting up task: AU-Tum_2002-2017_OzFlux_Met_R0_S3
Creating runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3 directory
  Cleaning task
  Copying namelist files from namelists to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3
  Copying CABLE executable from src/main/src/offline/cable to runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/cable
  Adding base configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/cable.nml
  Adding science configurations to CABLE namelist file runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/cable.nml
Successfully setup fluxsite tasks

Setting up run directory tree for spatial tests...
Setting up tasks...
Setting up task: crujra_access_R0_S0
git clone https://github.com/CABLE-LSM/cable_example.git runs/spatial/tasks/crujra_access_R0_S0
  Updating experiment config parameters in runs/spatial/tasks/crujra_access_R0_S0/config.yaml
  Adding science configurations to CABLE namelist file runs/spatial/tasks/crujra_access_R0_S0/cable.nml
Setting up task: crujra_access_R0_S1
git clone https://github.com/CABLE-LSM/cable_example.git runs/spatial/tasks/crujra_access_R0_S1
  Updating experiment config parameters in runs/spatial/tasks/crujra_access_R0_S1/config.yaml
  Adding science configurations to CABLE namelist file runs/spatial/tasks/crujra_access_R0_S1/cable.nml
Setting up task: crujra_access_R0_S2
git clone https://github.com/CABLE-LSM/cable_example.git runs/spatial/tasks/crujra_access_R0_S2
  Updating experiment config parameters in runs/spatial/tasks/crujra_access_R0_S2/config.yaml
  Adding science configurations to CABLE namelist file runs/spatial/tasks/crujra_access_R0_S2/cable.nml
Setting up task: crujra_access_R0_S3
git clone https://github.com/CABLE-LSM/cable_example.git runs/spatial/tasks/crujra_access_R0_S3
  Updating experiment config parameters in runs/spatial/tasks/crujra_access_R0_S3/config.yaml
  Adding science configurations to CABLE namelist file runs/spatial/tasks/crujra_access_R0_S3/cable.nml
Successfully setup spatial tasks

Creating PBS job script to run fluxsite tasks on compute nodes: benchmark_cable_qsub.sh
qsub benchmark_cable_qsub.sh
PBS job submitted: 107364962.gadi-pbs
The CABLE log file for each task is written to runs/fluxsite/logs/<task_name>_log.txt
The CABLE standard output for each task is written to runs/fluxsite/tasks/<task_name>/out.txt
The NetCDF output for each task is written to runs/fluxsite/outputs/<task_name>_out.nc

Running spatial tasks...
payu run
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
payu: Found modules in /opt/Modules/v4.3.0
qsub -q normal -P tm70 -l walltime=3600 -l ncpus=16 -l mem=64GB -N crujra_accessN9 -l wd -j n -v PAYU_PATH=/home/659/ag9761/.conda/envs/benchcab-dev-payu/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -l storage=gdata/rp23+scratch/tm70 -- /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/python3.9 /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/payu-run
107364963.gadi-pbs
payu run
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
payu: Found modules in /opt/Modules/v4.3.0
qsub -q normal -P tm70 -l walltime=3600 -l ncpus=16 -l mem=64GB -N crujra_accessN9 -l wd -j n -v PAYU_PATH=/home/659/ag9761/.conda/envs/benchcab-dev-payu/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -l storage=gdata/rp23+scratch/tm70 -- /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/python3.9 /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/payu-run
107364964.gadi-pbs
payu run
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
payu: Found modules in /opt/Modules/v4.3.0
qsub -q normal -P tm70 -l walltime=3600 -l ncpus=16 -l mem=64GB -N crujra_accessN9 -l wd -j n -v PAYU_PATH=/home/659/ag9761/.conda/envs/benchcab-dev-payu/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -l storage=gdata/rp23+scratch/tm70 -- /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/python3.9 /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/payu-run
107364965.gadi-pbs
payu run
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
payu: Found modules in /opt/Modules/v4.3.0
qsub -q normal -P tm70 -l walltime=3600 -l ncpus=16 -l mem=64GB -N crujra_accessN9 -l wd -j n -v PAYU_PATH=/home/659/ag9761/.conda/envs/benchcab-dev-payu/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/hh5/public/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -l storage=gdata/rp23+scratch/tm70 -- /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/python3.9 /home/659/ag9761/.conda/envs/benchcab-dev-payu/bin/payu-run
107364966.gadi-pbs
Successfully dispatched payu jobs

PBS Tasks

(benchcab-dev-payu) [ag9761@gadi-login-08 integration]$ cat benchmark_cable_qsub.sh.o107364962 
/home/659/ag9761/.local/bin/benchcab fluxsite-run-tasks --config=config.yaml -v
Running fluxsite tasks...
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S3... CABLE standard output saved in runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/out.txt
./cable cable.nml
Adding attributes to output file: runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S2... CABLE standard output saved in runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/out.txt
./cable cable.nml
Adding attributes to output file: runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S1... CABLE standard output saved in runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/out.txt
./cable cable.nml
Adding attributes to output file: runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S0... CABLE standard output saved in runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/out.txt
./cable cable.nml
Adding attributes to output file: runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc
Successfully ran fluxsite tasks


/home/659/ag9761/.local/bin/benchcab fluxsite-bitwise-cmp --config=config.yaml -v
Running comparison tasks...
Successfully ran comparison tasks


======================================================================================
                  Resource Usage on 2024-02-02 13:45:56:
   Job Id:             107364962.gadi-pbs
   Project:            tm70
   Exit Status:        0
   Service Units:      0.86
   NCPUs Requested:    18                     NCPUs Used: 18              
                                           CPU Time Used: 00:05:23        
   Memory Requested:   30.0GB                Memory Used: 469.42MB        
   Walltime requested: 06:00:00            Walltime Used: 00:01:26        
   JobFS requested:    100.0MB                JobFS used: 0B              
======================================================================================

@abhaasgoyal
Copy link

abhaasgoyal commented Feb 15, 2024 •
edited
Loading

Hi, I did resolve the conflicts from main and used the logger for any standard output prints in the new parts of the code for the spatial simulations.

I wanted to mention some points to look at and potentially improve before merging:

  • render_job_script in pbs.py has verbose as a parameter, which would currently always be false. I think it's better to set it within the function, similar to subprocess.py
  • subprocess.py uses print, which could use logger <- the corresponding tests could use caplog instead of capturing from capfd (they only check for file descriptors 1 and 2
  • Tests using set_log_level could be persistent across all tests if get_logger() isn't called from scratch, potentially leading to bugs in future code <- a similar solution of using caplog
  • spatial and run functions in benchcab module call many of the same functions. Is there a way to modularize it better according to the requirements?
  • spatial function does not use skip parameter, should we remove it?

A usage example of caplog is in test_spatial.py

@ccarouge
Copy link
Member

ccarouge commented Feb 16, 2024

@bschroeter see @abhaasgoyal comment above if you have anything to add about the logger points (or other points if you like...)

@bschroeter
Copy link
Collaborator

bschroeter commented Feb 29, 2024

OK, some points following a discussion with @abhaasgoyal.

I agree, there should not be a verbose flag for subprocess.py - it was leftover while we figured out how to capture the different streams of verbosity in this part of the code.

There are a number of things I think we should do to resolve this:

  1. Remove the verbose flag from subprocess run_cmd() and pbs render_job_script() and instead rely on a utility function to get the verbosity.

  2. Write a utility function to assess verbosity in utils/init.py and move the debug level constant as well, which will allow us to get the verbosity (from the effective log level) anywhere in the code with minimal effort.

DEBUG_LEVEL = 10

def is_verbose():
    return get_logger().getEffectiveLevel() == DEBUG_LEVEL

(documented of course)

We can then use utils.is_verbose() anywhere we need to change logic based on verbosity, such as in subprocess.py and pbs.py (render_job_script).

  1. Update the subprocess.py code as per my comments in DRAFT: Redesign test_subprocess logic #253 to have verbosity engage in both the logging AND the captured standard out from a subprocess command.

  2. Update tests to reflect all of this, including writing a test for is_verbose().

Regarding the tests where things happen in different logging levels (verbosities), I think it is OK to just ensure that the logger is set to the correct level immediately before running the test. Assume that each test is sandboxed and it is the responsibility of the test to prepare the logger for the expected output prior to whatever assertion is being made. You could always parameterise this with pytest if needed.

Outside of testing, changing the log level never done, as we only set the logger level once upon the first call of get_logger() and singleton is shipped around for all logging and extraction of verbosity.

Hope this all makes sense. Happy to discuss if you have any queries, especially if you can think of a better way to do this.

@abhaasgoyal
Copy link

abhaasgoyal commented Feb 29, 2024 •
edited
Loading

Ok, so most of the outcomes and suggestions seem good. I just had one more query:
Regarding ensuring that the logger is set to the correct level before running tests, in case the developer forgets, it could lead to hard to detect bugs, and also the behaviour of logging being persistent is not being shown clearly here. Maybe we could set the logger level with caplog fixture - caplog.set_level() before running each test, and in case it is not mentioned, logging will be set to default - logging.INFO.

@bschroeter could you provide your views on the same? (Happy to have further discussion as well)

@ccarouge
Copy link
Member

ccarouge commented Feb 29, 2024

@bschroeter @abhaasgoyal from reading the discussion about the logger, can I argue none of this is related to the spatial simulations? Should/could we take this to a separate issue?
And more importantly, I don't see why this should delay getting this PR in the code. Comments?

@abhaasgoyal
Copy link

abhaasgoyal commented Feb 29, 2024

@ccarouge I agree this is not related to this specific PR and only the logger from a closed issue/PR. Considering that this contains the required work for spatial simulations, we can merge this branch (I have fixed the conflict). I'm also happy to create a new issue referencing discussions here if it looks good.

@bschroeter
Copy link
Collaborator

bschroeter commented Feb 29, 2024

@bschroeter @abhaasgoyal from reading the discussion about the logger, can I argue none of this is related to the spatial simulations? Should/could we take this to a separate issue? And more importantly, I don't see why this should delay getting this PR in the code. Comments?

I have no problem with merging the spatial simulations work to keep the scope of this issue tight and move the rest of the discussion to a new issue - I think we should just make sure that we test what we think we are testing regarding subprocess verbosity. As it stands, I'm not sure it does.

@bschroeter
Copy link
Collaborator

bschroeter commented Feb 29, 2024

Ok, so most of the outcomes and suggestions seem good. I just had one more query: Regarding ensuring that the logger is set to the correct level before running tests, in case the developer forgets, it could lead to hard to detect bugs, and also the behaviour of logging being persistent is not being shown clearly here. Maybe we could set the logger level with caplog fixture - caplog.set_level() before running each test, and in case it is not mentioned, logging will be set to default - logging.INFO.

@bschroeter could you provide your views on the same? (Happy to have further discussion as well)

The logger is defaulted to DEBUG (see the method signature for get_logger()) - so if the developer fails to set it to something else, then they are going to cause their tests to fail and wonder why - which is kind of the point...

The only reason the current tests surrounding the logger override the level in various circumstances is because we are using a fixture for the logger object itself. It is arguably a commensurate amount of code to simply instantiate a new logger with the desired level in each test method:

logger = get_logger(level='INFO')
# test that something to do with INFO works..

But I would argue that using the fixture and the override is explicit enough for our purposes and allows reuse of the fixture.

I'm yet to be convinced that caplog is going to give us any additional benefit over what we already have, and tend not to add additional dependencies without strong reasons.

But then again, happy to be proven otherwise!

I think we can probably merge this and hash out logging in another downstream issue.

@abhaasgoyal
Copy link

abhaasgoyal commented Feb 29, 2024

Ok, for now caplog doesn't seem to provide strong benefits to what we have. I'll merge this and create a new issue for the remaining points w.r.t. logger

@abhaasgoyal abhaasgoyal merged commit 9482458 into main Feb 29, 2024
@abhaasgoyal abhaasgoyal deleted the 5-spatial-testing branch February 29, 2024 04:25
@abhaasgoyal abhaasgoyal mentioned this pull request Feb 29, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abhaasgoyal abhaasgoyal abhaasgoyal left review comments

@bschroeter bschroeter Awaiting requested review from bschroeter

@ccarouge ccarouge Awaiting requested review from ccarouge

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add handling of spatial simulations

5 participants

@SeanBryan51 @ccarouge @abhaasgoyal @bschroeter