7. HPC-Stack Additional Notes

7.1. Setting Compiler Flags and Other Options

Often it is necessary to specify compiler flags (e.g., gfortran-10 requires -fallow-argument-mismatch``to be used with ``FFLAGS. There are 2 ways this can be achieved:

  1. For all packages: One can define variable e.g., STACK_FFLAGS=-fallow-argument-mismatch in the config file config_custom.sh. This will append STACK_FFLAGS to FFLAGS in every build script under ./libs/ directory.

  2. Package-specific flags: To compile only the specific package under libs with the above compiler flag, one can define variable FFLAGS=-fallow-argument-mismatch in the <package> section of the YAML file stack_custom.yaml. This will append STACK_<package>_FFLAGS to FFLAGS in the build script for that package only.

7.2. Adding a New Library or Package

If you want to add a new library to the stack you need to follow these steps:

  1. Write a new build script in libs, using existing scripts as a template.

  2. Define a new section in the yaml file for that library/package in config directory.

  3. If the package is a python virtual environment, add a requirements.txt or environment.yml file listing the python packages required to install the package. These files should be named and placed in pyvenv/package_name.txt and pyvenv/package_name.yml. VENVTYPE=pyvenv will use the pyvenv/package_name.txt and VENVTYPE=condaenv will use pyvenv/package_name.yml.

  4. Add a call to the new build script in build_stack.sh.

  5. Create a new module template at the appropriate place in the modulefiles directory, using exising files as a template.

  6. Update the HPC Components file to include the name of the new library or package.

7.3. Configuring for a new HPC

If you want to port this to a new HPC, you need to follow these steps:

  1. Write a new config file config/config_<hpc>.sh, using existing config files as a template. Also create a new yaml file config/stack_<hpc>.yaml, using existing yaml files as a template.

  2. Add/remove basic modules for that HPC.

  3. Choose the appropriate Compiler/MPI combination.

  4. If a template modulefile does not exist for that Compiler/MPI combinattion, create module templates at the appropriate place in the modulefiles directory, using existing files as a template (e.g., hpc-ips or hpc-smpi).

  5. If the new HPC system provides some basic modules for e.g., Git, CMake, etc., they can be loaded in config/config_<hpc>.sh.

7.4. Using the DOWNLOAD_ONLY Option

If an HPC (e.g., NOAA RDHPCS Hera) does not allow access to online software via wget or git clone, you will have to download all the packages using the DOWNLOAD_ONLY option in the config_custom.sh. Execute build_stack.sh as you would on a machine that does allow access to online software with DOWNLOAD_ONLY=YES and all the packages will be downloaded in the pkg directory. Transfer the contents of the pkg directory to the machine where you wish to install the HPC-Stack, and execute build_stack.sh. The build_stack.sh script will detect the already-downloaded packages and use them rather than fetching them.

7.5. Using the HPC-Stack

  • If Lmod is used to manage the software stack, you will need to activate the HPC-Stack in order to use it. This is done by loading the hpc module under $PREFIX/modulefiles/stack as follows:

    module use $PREFIX/modulefiles/stack
    module load hpc/1.0.0

This will put the hpc-<compilerName> module in your MODULEPATH, which can be loaded as:

module load hpc-<compilerName>/<compilerVersion>
  • If the HPC-Stack is not managed via modules, you need to add $PREFIX to the PATH as follows:

    export PATH="$PREFIX/bin:$PATH"

7.6. Known Workaround for Certain Installations of Lmod

  • On some machines (e.g., WCOSS_DELL_P3), LMod is built to disable loading of default modulefiles and requires the user to load the module with an explicit version of the module (e.g., module load netcdf/4.7.4 instead of module load netcdf). The latter looks for the default module which is either the latest version or a version that is marked as default. To circumvent this, it is necessary to place the following lines in modulefiles/stack/hpc/hpc.lua prior to executing setup_modules.sh or in $PREFIX/modulefiles/stack/hpc/1.0.0.lua after executing setup_modules.sh.

    setenv("LMOD_EXACT_MATCH", "no")
    setenv("LMOD_EXTENDED_DEFAULT", "yes")

    See more on the Lmod website.

7.7. Known Issues

  • NetCDF-C++ does not build with LLVM Clang. It can be disabled by setting disable_cxx: YES in the stack file under the NetCDF section.

  • Json-schema-validator does not build with LLVM Clang. It can be disabled in the stack file in the json-schema-validator-section.