-
Notifications
You must be signed in to change notification settings - Fork 55
Spack Mirrors
Spack support different types of mirrors, often also referred to as "caches": source mirrors, binary mirrors, bootstrap mirrors. In addition, spack-stack provides utilities to create cargo and go mirrors. The purpose of these mirrors is to speed up the provisoning of new spack-stack environments (binary mirrors) and to support installing on air-gapped systems (systems without access to the internet; all types of mirrors).
Spack source mirrors allow downloading the source code required to build environments once to a local directory and then use this directory for subsequent installations. If a package cannot be found in the mirror (e.g. because a newer version is required), it will automatically be pulled from the web. It won’t be added to the source cache automatically; this is a step that needs to be done manually.
Spack mirrors also make it possible to download the source code for an air-gapped machine on another system, then transferring the entire mirror to the system without internet access and using it during the installation.
Since all spack-stack installations are based on environments, we only cover Spack mirrors for environments here. For a more general discussion, users are referred to the Spack Documentation.
Create an environment as usual, activate it and run the concretization step (spack concretize), but do not start the installation yet.
Next, create the Spack source mirror in /path/to/spack-mirror.
spack mirror create -a -d /path/to/spack-source
If the spack mirror already exists, then existing packages will be ignored and only new packages will be added to the mirror.
If not already included in the environment (i.e., via the spack-stack site config), add the mirror:
spack mirror list
spack mirror add local-source file:///path/to/spack-source-mirror
The newly created local mirror should be listed at the top, which means that spack will search this directory first.
Proceed with the installation as usual. The benefit of maintaining a local source mirror, even on a machine with internet access, is that it avoids repeated downloads of the same packages every time a new spack environment is created.
To use Spack on an air-gapped system, the first step is to create a mirror that allows Spack to bootstrap from. The logic is similar to that for creating source caches for Spack installations. On a system with access to the internet, a series of commands is run to create a bootstrap mirror. The bootstrap mirror is then copied over to the air-gapped system and registered so that Spack can use it.
On the system with access to the internet, check out the same version of spack-stack (and its spack submodule) as on the air-gapped target system. After running source setup.sh in the spack-stack top-level directory, create a bootstrap mirror:
spack bootstrap mirror --binary-packages ./bootstrap-mirror
Transfer ./bootstrap-mirror to the air-gapped system, preferably to the same location inside the spack-stack top-level directory (i.e. ${SPACK_STACK_DIR}/bootstrap-mirror/).
On the air-gapped system, run the following series of commands after running source setup.sh to register the bootstrap mirror:
spack bootstrap add --trust local-sources /absolute/path/to/spack-stack/bootstrap-mirror/metadata/sources
spack bootstrap add --trust local-binaries /absolute/path/to/spack-stack/bootstrap-mirror/metadata/sources
Update the boostrap binary cache index (silent response to this command means success)
spack buildcache update-index /absolute/path/to/spack-stack/bootstrap-mirror/bootstrap_cache
Bootstrap Spack explicitly before creating environments and check the status. There may be error messages along the way, but as long as the command completes and spack bootstrap status shows PASS for both "Core Functionalities" and "Binary Packages", it was successful.
spack bootstrap now
spack bootstrap status
The procedure is similar to using spack mirrors for local reuse, but a few additional steps are needed in between.
On the air-gapped system: Create an environment as usual, activate it and run the concretization step (spack concretize), but do not start the installation yet. The concretization step may require configuring a bootstrap mirror first, see above.
After successful concretization, copy the file spack.lock (in envs/env-name/) to the machine with full internet access using scp, for example.
On the machine with full internet access: Load the basic external modules, if using a machine that is preconfigured for spack-stack, and make sure that git-lfs support is available (if necessary, load the external modules that spack-stack also uses). Check out the same version of spack-stack, run setup.sh, and then the following sequence of commands. The source mirror based on spack.lock will be placed in the directory specified by the -d argument passed to spack mirror create.
spack env create air_gapped_mirror_env spack.lock
spack env activate air_gapped_mirror_env
spack mirror create -a -d ./mirror/
On the air-gapped system: Copy the directory from the system with internet access to the local destination for the spack mirror. It is recommended to use rsync to avoid deleting existing packages, if updating an existing mirror on the air-gapped system.
On the air-gapped system: Add the mirror to the spack environment’s mirror list, unless already included in the site config.
spack mirror add locals-source file:///path/to/spack-source-mirror
spack mirror list
The newly created local mirror should be listed at the top, which means that spack will search this directory first.
Proceed with the installation as usual on the air-gapped system.
Note Depending on the environment to be installed, additional mirrors for cargo and go may be required. See next sections for more information.
spack-stack provides a set of utilities util/fetch_cargo_deps.py/install_rust.sh to download Rust/Cargo package dependencies and store them in a local directory for later use during spack install. This is required for installing on systems that do not have access to the internet during the spack install phase and complements other mirrors such as the Spack source mirror.
Run util/fetch_cargo_deps.py in an active, concretized Spack environment to fetch Rust dependencies and store them in ${CARGO_HOME}. You must either run the script with spack-python or have spack-python in your ${PATH}. Ensure ${CARGO_HOME} has the same value when spack install is run, or the correct value for the air-gapped system after transferring the Cargo mirror from the system with access to the internet.
For each spec that is a CargoPackage or a PythonPackage with a rust dependency, the script will attempt to fetch all of its Cargo dependencies using cargo if available in the user’s environment, but will fall back to installing cargo and rustup from the internet using install_rust.sh.
The util/fetch_go_deps.py utility prefetches Go dependencies and stores them in a local directory for later use during spack install. This is required for installing on systems that do not have access to the internet during the spack install phase and complements other mirrors such as the Spack source mirror.
The $GOMODCACHE variable must be set, and the utility must be run in an active, concretized environment. It will scan Spack packages of type GoPackage, and fetch all dependencies based on the dependency listings found in those packages. The utilities will attempt to use each package’s go dependency, in which case these utilities must be run after go is installed. It will revert to using a system-installed go if available.
The go mirror in $GOMODCACHE can then be transferred to an air-gapped system and used during spack install by setting $GOMODCACHE accordingly.
Spack provides the capability to compress and archive installed packages in binary cache and reinstall from the cache through a process called "binary relocation" instead of building from source if and only if the package hash matches. That is, all specs for a given package must be identical: compiler, build options, dependencies, etc. The ability to reinstall from a binary cache instead of building from source is a powerful feature that can be used to install on air-gapped systems, to mirror spack-stack installations on identical twins, and in general to greatly speed up the deployment of new spack-stack environments.
Because the library headers need to provide sufficient space (number of characters in a string) to hold the final path to the library in the installed environment, Spack provides an option to pad the length of the install tree when building packages for populating or updating a binary cache. In principle, these installations can be used as is. However, some downstream applications and some systems do not work well with exceedingly long paths in LD_LIBRARY_PATH etc.
The recommendation is therefore to use binary caches in a two-step process:
In a first step, create the environment and build all packages from source (or an existing binary cache) with an added config option padded_length in the Spack config section. This option can be added via
spack config add config:install_tree:padded_length:200
prior to concretizing and installing the environment. Following a successful build, the Spack binary cache must be added to the site config, if not already present - check for local-binary in the output of spack mirror list:
spack mirror list
spack mirror add local-binary file:///path/to/spack-build-cache
Next, push any new packages to the binary cache and update the binary cache index with:
spack buildcache push -u
spack buildcache update-index local-binary
Note that the -u option forces Spack to push unsigned packages to the binary cache, which is acceptable for deployments on local systems.
In the second step, a new environment is created that is identical to the one used for creating the binary cache, except that the padded_length config option is omitted. After creating the environment and verifying that the binary cache is configured as a mirror, concretize and install the environment as follows:
spack concretize --force --fresh 2>&1 | tee log.concretize
spack install [--verbose] --no-check-signature 2>&1 | tee log.install
Note the --no-check-signature to install unsigned packages, and the --force --fresh for the concretization step. The latter ensures that the concretization of the final Spack environment is identical to the one that was used to populate the binary cache.
Users wishing to adopt Spack binary caches and integrate them in their workflows are encouraged to inspect the GitHub actions workflows as well as the NRL batch install script in the spack-stack repository.