Introduction for using Spack environments

Overview

The following tutorial assumes you have a functioning spack-stack environment, either installed local to your system, or on a fully supported site.

There are three steps in setting up a usable development environment. The first step is to load the spack-stack environment. The second (optional) step is to create a python virtual environment that is based on the Python executable included within the spack-stack installation. The reason for the Python virtual environment is to ensure that Python-based applications are utilizing the spack-stack Python modules in a consistent manner. The third step is to configure your build system to use the Python virtual environment created in the second step, or otherwise the Python environment provided in the spack-stack environment.

When using a spack-stack environment, please utilize the spack-stack installed Python modules as much as possible to help maintain the consistency mentioned above. Note that after loading the spack-stack environment, all of the spack-stack installed Python modules have been added to PYTHONPATH and are immediately accessible.

Load the spack-stack environment

Spack environments are used by loading the modulefiles generated at the end of the installation process. These modules control the unix environment and allow CMake, ecbuild, and other build toolchains to resolve the version of software intended for the compilation task. The spack command itself is not needed in this setup, hence the instructions for creating new environments (source setup.sh etc.) can be ignored. The following is sufficient for loading the modules, allowing them to be used while compiling and running user code.

Warning. Customizations of the user environment in .bashrc, .bash_profile, ..., that load modules automatically may interfere with environment setup or updates. It is highly advised to avoid "polluting" the standard environment. If you frequently reuse the same module set, you should put your setup procedure into a shell script that can be sourced as needed.

Load the spack meta-modules directory into the module path using the appropriate path for the preconfigured sites or your local spack-stack environments. Note that the meta-module does not update your environment and only informs your module tool of the location of a new set of modules.

module use /path/to/spack-stack-env//modules/Core

If you run module available now, you will see only one option; the compiler (stack-<compiler>/<version>). Loading the compiler meta-module will give access to the MPI meta-module and access to other packages that only depend on the compiler. Loading the MPI meta-module will then add the MPI-dependent packages to the module path, and so on.

module load stack-compiler-name/compiler-version
module load stack-mpi-name/mpi-version

Now list all available modules via module available. You may be required to load additional packages depending on your build target's requirements, but now you have loaded a basic spack environment and you are ready to build.

(Optional) Create and activate a Python virtual environment

If additional Python packages are required that are not provided by the spack-stack environment, a virtual environment must be created. It is important that the creation of the Python virtual environment is based on the Python executable from the spack-stack installation: If the python/<version> module provided by the spack-stack environment isn't loaded already, make sure to load it prior to creating the virtual environment. This ensures consistency for Python applications between the Python executable and the spack-stack installed Python packages (eg., numpy). Without this consistency, it is easy for the wrong underlying library versions to get dynamically loaded and cause problems with applications crashing.

Check that the correct Python interpreter is found after loading the spack-stack Python module

which python3

The path should match the environment variable python_ROOT. Create the virtual environment:

python3 -m venv <path-to-python-virtual-env>

Once the virtual environment is set, it must be activated:

source <path-to-python-virtual-env>/bin/activate

The implication of this is that you should activate the Python virtual environment as the last step in setting up your environment, i.e. after loading all spack-stack modules, to ensure that the path to the virtual environment python remains first in your PATH.

Here is an example of the whole process:

# Start from clean slate
module purge

# Load the base packages from the spack-stack environment
module use /path/to/my/example/spack-stack-env/modules/Core
module load stack-gcc/13.3.1
module load stack-openmpi/5.0.8

# Load the additional environments required for your target application
module load jedi-fv3-env
module load ewok-env
module load soca-env

# Create and activate the spack-stack based Python virtual environment
# Note that you only need to create the virtual environment the first time.
# Once created you only need to activate the virtual environment.
cd $HOME/projects/jedi
python3 -m venv jedi_py_venv # first time only
source jedi_py_venv/bin/activate

Missing Python packages can now be installed as follows:

python3 -m pip install <package-name>

Warning. If new packages require a different version of a package than available in the spack-stack environment, pip will attempt to uninstall the existing version. This will break the spack-stack environment. This is only an issue when the user has write permissions for the existing spack-stack environment. In this case, a possible way to prevent this is to make the spack-stack installation read-only prior to installing packages with pip.

Configure build system to utilize the correct Python environment

There are a variety of build systems in use. CMake will be used as an example for this step. The CMake variable Python3_FIND_STRATEGY can be used in conjunction with the python virtual environment to direct CMake to find and use the desired python virtual environment.

By default, CMake chooses the latest Python installation regardless of which comes first in your $PATH environment variable. By setting Python3_FIND_STRATEGY=LOCATION, CMake will instead find and use the first python installation found in $PATH. This is the reason for making the spack-stack based Python (virtual) environment first in $PATH in the step above.

Python3_FIND_STRATEGY can be set in two ways, either in the project's top-level CMakeLists.txt file or as an argument on the cmake (or ecbuild) command line.

# In CMakeLists.txt
set(Python3_FIND_STRATEGY LOCATION)

# On the command line
cmake -DPython3_FIND_STRATEGY=LOCATION ...