Enhance the documentation with additional explanations on creating chained environments

[natalie-perlin]

Provide clearer guidance and explanations about building chained environments to spack-stack NOAA RDHPCS Tier 1 systems users. UFS developers have frequent needs to test higher versions of packages or add another package(s) to the software environment. By elaborating the details and user-oriented knowledge required to build chained environments on top of existing upstream ones, tasks of intermediate spack-stack updates work could be largely passed to developers.

[AlexanderRichert-NOAA]

Can you please update or elaborate on what should be updated in https://spack-stack.readthedocs.io/en/latest/AddingTestPackages.html?

[climbfuji]

@natalie-perlin Can you please respond to the question from @AlexanderRichert-NOAA (#1749 (comment)) on what needs to be improved in the documentation for chained environments?

[natalie-perlin]

@AlexanderRichert-NOAA @climbfuji - My suggestion to add some more transparency to Chapter 5 as shown below. Please feel free to edit terminology as more appropriate ("chaining environments" vs "creating a chained environment", or "chain environment"? - it would be very helpful to unify the term to make search efforts more productive...)

5. Chaining environments (adding test packages)

Releases of spack-stack are deployed quarterly on supported platforms. Between releases, additional packages may be installed through the following steps, which make use of Spack’s environment chaining capabilities. This allows parts of the stack to be replaced while leaving the release environment untouched.

What “Chained Environment” Means

A chained environment is a Spack environment ("downstream") that builds on top of one or more existing environments (“upstreams”). Instead of creating a standalone installation, users point the new environment to the install tree(s) of previously deployed environments. The upstream environment provides an existing, fully configured collection of packages and modulefiles, which the downstream environment may then reuse.

In a chained environment, users may override specific packages (e.g., to test a new version) or add new packages on top of what already exists. Because the downstream environment references the upstream one without modifying it, a chain of environments is formed: the new environment extends or customizes the existing stack while preserving the stability of the base installation.

Setting Up an Environment for Chaining

To install in an additional environment within an official spack-stack space, simply cd into the appropriate spack-stack root directory and run . setup.sh before continuing to the following steps. To create a chained environment in a personal space outside of an official installation, it is recommended that you use the same spack-stack release as the one from which you wish to use an upstream environment.

For example, if you are targeting an environment installed under spack-stack-1.4.1/, clone and check out the matching branch:

git clone --recurse-submodules https://github.com/jcsda/spack-stack -b release/1.4.1

Then, in the newly created spack-stack directory:

. setup.sh

and proceed with environment creation.

The following example of creating a "netcdf-test" chained environment on Hera off the existing gcc-based environment, in which a hypothetical new version of NetCDF-C is to be built with intel compiler, and which also rebuilds all dependencies needed for the UFS Weather Model. Note that it is recommended to use one or more meta-modules (e.g., ufs-weather-model-env) rather than individual packages, because meta-modules ensure that all required modulefiles from the upstream environment are made available in the downstream environment when spack module lmod refresh --upstream-modules is invoked. Make sure to use the same compiler as the upstream environment.

spack stack create env --name netcdf-test --template empty --site hera --compiler gcc \
    --upstream /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.4.1/envs/ue-gcc/install \
    [--upstream /path/to/second/install] [--modify-pkg netcdf-c]
cd envs/netcdf-test
spack env activate .

Upstream Environment Declaration in spack.yaml

When chained environment is created, it will include an upstreams: section at the end of its spack.yaml. This section records the name of each upstream environment and the location of its install tree. For example:

upstreams:
  my-base-env:
    install_tree: /full/path/to/spack-stack/envs/my-base-env/install

This configuration tells Spack to reuse packages already installed in the upstream environment, unless overridden in the downstream one. Multiple upstream environments may be listed, and Spack will search them in order during concretization.

Adding Packages After Creating the Chained Environment

Once the chained environment has been created and activated, additional packages can be added directly from the command line (i) or being added to spack.yaml specs (ii).

(i) An example below adds the UFS Weather Model meta-environment and requests a new version of NetCDF-C:

spack add ufs-weather-model-env%intel ^netcdf-c@5.0.0

(ii) Adding Packages by Editing spack.yaml

Users may alternatively edit spack.yaml (and configuration files under ./site/ and ./common/ if needed) in the same way they are prepared for a base environment installation. This approach is useful when adding several packages or adjusting specifications in bulk. In the below example, in addition to a meta modules ufs-weather-model-env with netcdf-c@5.0.0, standalone packages of netcdf-fortran@4.1.6 and nco@5.0.6 built with intel compiler are added to the environment. Additional packages can be listed under specs:

specs:
  - ufs-weather-model-env%intel ^netcdf-c@5.0.0
  - netcdf-fortran@4.6.1 %intel
  - nco@5.0.6 %intel

After addition of new package(s) in step (i) or (ii), the proceed with usual concretization and installation steps, adding the flag "--upstream-modules" when refreshing/rebuilding lmod modules:

spack concretize 2>&1 | tee log.concretize
spack install 2>&1 | tee log.install
spack module lmod refresh --upstream-modules
spack stack setup-meta-modules

Using the Chained Environment

To use the chained environment, access it in the same way as a regular spack-stack installation: add the directory provided by spack stack setup-meta-modules (ending with /modulefiles/Core) to your MODULEPATH. This provides access to the correct combination of upstream and downstream modulefiles while avoiding conflicts. Do not add the upstream environment’s module directory directly to MODULEPATH.

Notes

The --upstream option in spack stack create env may be passed multiple times to include several upstream environments. A warning is issued (but the command does not fail) if an invalid path is provided.

The --modify-pkg option should be used by spack-stack maintainers when patches, variants, or new versions must be introduced between official releases. It creates a custom package repository under $SPACK_ENV/envrepo/ and updates the environment configuration so that only the modified environment is affected.

Additional details and cautions regarding Spack environment chaining can be found in the Spack documentation, particularly regarding avoiding the deletion of upstream modulefiles or dependencies.

[natalie-perlin]

If this looks OK, I'd be glad to make a PR with these documentation changes. My concerns are the following:

• The example mentioned older version of spack-stack 1.4.1, and also mentions Hera platform that is being decommissioned.
• It mentions a mixed-compiler environment: gcc-based upstream environment and packages build with intel. It is likely intentional, but on the other hand it says "Make sure to use the same compiler as the upstream environment". The latter may refer to use of the same ./site/compilers.yaml configuration file (it may be obvious for a spack-stack developer, but could be confusing for a user)

[climbfuji]

Closed via #1855