Modules

Introduction

A structured software stack with libraries (optimized for the respective target architectures), compilers, parallel run-time environments and open-source applications is supplied via software modules to all HPC cluster users. Although the available software modules cover a wide range of applications, they may not be sufficient to satisfy all individual user needs. We provide a preconfigured package manager (Spack) that allows users to install and build their own packages and modules in their home directory, but this may still be too limited for some users. In some reasonable cases, we may also make additional software manually available on the cluster nodes upon request, but please be aware that in general, we do not install individual software for every user. If you want to run your application in your individual software environment, have a look into Software Containers!

The setting up of the modules package is performed automatically at login. This includes

• Making the module command (as described below) available
• By loading a default set of module files. The exact set depends on the platform used.

terrabyte modules

We provide additional software via our own terrabyte modules. Generally, these are the latest versions of some software modules, which are not available through the default software stack. However, there may be also software that will be made available for terrabyte users exclusively. The additional software modules can be added by typing:

module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/

To always make the terrabyte modules available it is advisable to add this line to the .bashrc File in your Home

In order to make a software available in the current shell type:

module load <package>

Here <package> specifies the name and obtionally the version of the software you wish to use. If no version is provided the newest Version is the default. This works for the default modules offered by LRZ as well as for the additional terrabyte modules if you added them using the hint above

In order to unload a previously loaded software module, type:

module unload <package>

Module command options

This section gives an overview of the most important command options.

module list: This lists all the modules which are currently loaded into the users' environment, in the order in which they were loaded. With an additional -t option, only a single column is printed.

module avail: This option lists all the modules which are available to be loaded. Notice that most have version numbers associated with them. One of the versions is marked out as default; this is the version that is referred to when the version number is omitted in a module command.  Adding the -d switch will only list the default versions of each module. Furthermore, limited pattern matching will also work, so

module avail gs

will print the list of available modules that start with the letters "gs". The -t switch will display the modules as a single column.

module help: This option requires a module name (with or without a version number) as an argument. It will display additional information for the specified module; at minimum, a pointer to web documentation for the package should be delivered.

module show: This option requires a module name (with or without a version number) as an argument. It will display the changes to the environment that would be done if the module were loaded.

module load: This option requires one or more (space separated) module names (with or without a version number) as an argument. For example,

module load gaussian

will make the default version of the package "gaussian" available for your use. If you need a specific supplied version that is non-default, you need to specify the version number explicitly, for example:

module load gaussian/09.E.01

Please also note that

• Some modules may automatically load others, if a package depends on some other packages to be available. Unloading dependent modules may give you trouble!
• Certain default modules should also not be unloaded, since your environment will not function properly without the settings performed there. Therefore they are set to sticky to make sure they are not unloaded by accident
• If you specify multiple modules, they will be loaded in the supplied order
• Instead of "load", you can also say "add".

module unload: This option requires one or more (space separated) module names (with or without a version number) as an argument. It will unload the specified modules, removing all previously supplied environment settings. Please note that

• For multiple modules to be removed, it is recommended to do this in the reverse order of loading, especially if there exist dependencies between two such modules
• It is not necessary to specify a version number - the loaded version will be removed even if not explicitly specified
• Instead of "unload", you can also say "rm".

module search: This option requires a text pattern as an argument. Each module contains a module-whatis entry with a keyword list that will be searched for the provided pattern. All modules that contain a matching entry will then be listed. In particular, the module classes (with a colon appended) are always used as a keyword, so this facility can be used to search the module system by application classes:

module search libraries:

You can also say "apropos" or "keyword" instead of "search".

module switch/module swap: This option requires exactly two (space separated) module names (with or without a version number) as an argument. The command performs a transition from the module specified as the first argument to that specified by the second one with a single command. Note that this is not exactly the same as an unload followed by a load, since all modules which were loaded after the first specified will also be reloaded, to assure that dependencies are correctly resolved. If this causes difficulties, you may need to revert to explicitly performing an unload/load sequence.

module purge: This command removes all modules from your environment. On the HPC systems, this causes the environment to become rather unusable. Therefore, after a purge, you should issue the command

unset LRZ_LOAD_DEFAULTS # or unsetenv for csh-like shells
module load lrz

before loading or unloading any further modules.

module use: This option requires an existing directory name (or multiple directory names) as an argument. It causes its argument to be added to the modules path, with the result that all module files stored within that directory become visible for the module command. The entries are all stored in the $MODULESPATH environment variable. The -a option can be used to append the directory to the path instead of prepending it. It is also possible to remove a path again by specifying it as an argument for the module unuse command.

module path: This option requires one module name (with or without a version number) as an argument. The complete pathname of the module-file will be printed as a result**.**

Spack Generated Modules

On our clusters, many software modules are autmatically provided via Spack, which is a package manager for supercomputers that automatizes and manages the builds of software packages. Collections of packages built with Spack will be released on our clusters periodically and are available through the environment module system. The latest release of packages will be listed at the bottom when using the 'module avail' command. Users can use Spack to install and build own packages in their user space (see further below).

Naming conventions

Environment modules generated by Spack use the naming scheme

<package name>/<full version number>[-<suffix1>][-<suffix2>][…]

It is highly recommended to use the complete module name when using the spack-generated modules. If you need a specific version, you will have to specify the version number explicitly.

# For example, asking for how many FFTW modules are available, do the following:
module avail fftw

# You will get a list which looks
# (can differ depending on which system and software stack a user is using)
# like the following:
fftw/3.3.10
fftw/3.3.10-gcc11-impi-openmp
fftw/3.3.10-intel21-impi-openmp
fftw/mkl-2018-fftw
fftw/mkl-2018-fftw-gcc11
fftw/mkl-2019-fftw
fftw/mkl-2019-fftw-gcc11
fftw/mkl-2020-fftw
fftw/mkl-2020-fftw-gcc11

# To load a specific module of FFTW, as recommended,
# a user must use a complete module name, like so:
module load fftw/3.3.10-intel21-impi-openmp

-intel<version> E.g. intel19, intel21	compiled with the Intel 21 (major version) compiler suite	gsl/2.5-intel21
-gcc<version> E.g. gcc11, gcc9	compiled with the gcc 11 (major version) compiler suite	gsl/2.5-gcc11
-impi	with MPI support (Intel MPI)	fftw/3.3.8-intel21-impi
-ompi	with MPI support (OpenMPI)	petsc/3.6.4-intel21-ompi-shared
-openmp	with threading support by OpenMP	fftw/3.3.8-intel21-openmp
-i32 / -i64 -r32 / -r64	32/64 bit integer/float support	metis/5.1.0-gcc11-i64-r64
-X11	with X11 support	r/3.4.4-gcc8-X11
-python	with python support	none available at the moment

Currently, most Spack generated packages are not supplied with a default version, so a 'module load <package name>' will just give you the highest lexical version.

Spack Branches and Releases

Spack Branches

release: stable version for long term support staging: addon to current release or preview of a future release master: development version of LRZ - accessible for test installations, may be subject to a full rebuild

LRZ will periodically release a new version of the software stack built with Spack and change the default version to the new release. Older versions will remain available by loading specific modules, e.g.

module switch spack/<version>

Sometimes we will provide new software prior to a new Spack software-stack release in a staging branch. To access such versions, load the staging module

module switch spack/staging[/<version>]

On the HPDA terrabyte clusters, at the moment only Spack version 22.2.1 is available.

Release number (click for details)	Release date	Platform	Compiler Version
			GCC	Intel	Intel MPI
22.2.1	04/2022	SLES15	11.2.0	21.4.0	2019.12.320

Install and build own packages in user space using Spack

We provide a preconfigured spack installation that allows users to install and build their own packages and modules in their home directory. It will build upon the packages installed by LRZ via the spack chaining mechanism. A 'module load user_spack' activates this environment. Please check here for a detailed usage description.

Package Variables

When loading a module, package-specific environment variables follow the conventions given in this section. In the following table, the placeholder PAC is used for the package name. These variables can be used in scripts and Makefiles e.g.

mpif90 … $MKL_LIB

PAC_BASE	base directory for the package
PAC_LIBDIR	library directory for the package, usually $PAC_BASE/lib
PAC_LIB	library setting for (static) linking, typically $PAC_BASE/lib/lib<library_name.a>
PAC_SHLIB	library setting for dynamic linking, typically -L$PAC_BASE/lib -l<library_name>
PAC_INC	include the setting for compilation, usually -I$PAC_BASE/include
PAC_DOC	the directory containing documentation (in other than man or info format)
PAC_WWW	URL referring to the location of package-specific documentation on the LRZ WWW server.

Note that occasionally a module may contain sub packages e.g., SUB, in which case additional variables of the form PAC_SUB_LIB, etc. may be defined.

Specifics on the setup for LRZ HPC systems

Module classes

In order to impose some structure on a large number of provided modules, a number of module classes have been defined, one of which each module falls into:

• spack: modules compiled with the spack building system
• applications: third party application programs will be entered here
• compilers: programs for converting computer language source code to (executable) machine code
• environment: modules for providing LRZ specific environment variables
• graphics: modules for providing graphics and visualization related functionality
• io_tools: modules for I/O related libraries and tools
• libraries: performance and functionality libraries
• parallel: access to parallelization software, especially MPI
• tools: development, tuning, and debugging aids

There is a further, special extfiles category that will be used e.g., for applications with a rapidly changing profile or for testing. However, this class is not made available by default. You will need to activate it by issuing

module use -a /lrz/sys/share/modules/extfiles

Please note that modules in this area are not guaranteed any support; also they may be modified or removed without any prior warning, on short order.

Shells and scripting languages

Supported Shells

The following shells are supported by the module system:

• bash
• csh
• ksh
• sh
• tcsh

Using modules from Perl or Python

It is possible to use the module command also from non-shell scripting languages. In the case of Perl, a LRZ specific package is available. Here is an example of how to use this:

#! /usr/bin/perl use lib "/lrz/sys/share/modules/init";
use Environment_Modules;
module("load","intel mkl");
module("list");
system("make myprog");
system("./dosomething");

Note the following details:

• Use standard perl function call syntax with brackets, especially when specifying more than one argument
• In contrast to the shell setup, only the LRZ administrative modules are loaded. Please load all other required modules yourself.
• Within Perl, the environment variables are accessible via the %ENV hash.
• All environment variables will be propagated to processes called e.g., via system. Hence, you can use these variables in Makefiles and scripts which are executed/called from Perl.

import os
exec(open(‘/lrz/sys/tools/modules/5.1.1/init/python.py’).read(), globals())
module(‘load’, ‘geos/3.9.1-gcc11’) system("make myprog")
module('list')

Generating new shells

While the environment generated within a shell should be inherited to newly generated ones (either the same shell or other shells), the module command itself may not be. We recommend inserting the following lines into your shell init script (e.g. .bashrc for bash, or $ENV for ksh):

typeset -f module > /dev/null
if [ $? != 0 -a -r /etc/profile.d/modules.sh ] ; then
    source /etc/profile.d/modules.sh
fi

Note that this will preserve the environment set up in the calling shell.

Using modules in batch jobs

The modules package is loaded by default at login. However, the typical batch systems are not able to reliably load the modules package. Hence, if you want to use modules within batch scripts, please do not forget to insert the following command as the first line of your script:

source /etc/profile.d/modules.sh

or in case your script is csh derived:

source /etc/profile.d/modules.csh

Additional utilities

Some additional utilities which are provided specifically with the LRZ installation are described in this section.

1. ismoduleloaded: if issued with an argument, will tell whether that module is loaded. If the argument is a versioned module name, the exact version will be checked, otherwise only whether any version of the given module is loaded. The output is a "YES" or "NO" to stdout as well as an error exit of 0 or 1, respectively. If the -s command-line switch is provided, output to stdout is suppressed. If no module argument is given, "YES" will be written to stdout if any module is loaded, "NO" otherwise.

2. defaultversion: if issued with a module name as an argument (with or without a version number), this will return the default version of the module on STDOUT. Error 1 is returned if the module does not exist.

3. loadedversion: if issued with a module name as an argument (with or without a version number), this will return the presently loaded version of the module on STDOUT. Error 1 is returned if a module of the given name is not loaded. The version number on the argument need not be the correct or even an existing one.

4. removemodules.[c]sh: this can be used to completely remove all module settings from your environment by sourcing the appropriate version into your running shell; all modules are unloaded, and the module alias, as well as all module-related variables, are unset. Beware that this will rather cripple your environment since many important settings are performed via LRZ administrative modules.

Writing your own module files

A user can perform her/his own module initializations at login time using a shell-specific script in the $HOME/.modules subdirectory. This script must have the same name as your login shell, i.e., $HOME/.modules/$SHELL, and will typically contain

• Extension of the MODULEPATH environment variable (e.g., via the module use command) to contain some subdirectory where a user's own module files can reside
• Loading of additional LRZ or self-written modules at login as needed

Full example of creating a module:

Install the software in the home directory. As an example, we install the software SPAdes.

cd $HOME

mkdir -p modules/software/spades # this is where the software will be installed
mkdir -p modules/files/spades # this is where the module file will be located

cd modules/software/spades

wget https://cab.spbu.ru/files/release3.15.4/SPAdes-3.15.4-Linux.tar.gz
tar -xzf SPAdes-3.15.4-Linux.tar.gz
rm SPAdes-3.15.4-Linux.tar.gz

This results in a directory called SPAdes-3.15.4-Linux, containing the directories

bin # contains the executables
share # often contains documentation

Now we create the module file

cd ~/modules/files/spades
# Create and open a new module file haveing the version muber as name
vi 3.15.4

# Add the following lines to the file (copy + paste, :wq for writing and exiting the file):

#%Module
prepend-path PATH {/usr/local/usrapps/bioinfo/spades/SPAdes-3.15.4-Linux/bin};

In case there are or will be other versions of SPAdes installed, set 3.15.4 as default by creating a file /modules/files/spades/.version containing

#%Module
set ModulesVersion 3.15.4

The module for SPAdes may be called from the command line or from a batch script:

module load ~/modules/files/spades/3.15.4

or

module use ~/modules/files

module avail spades # lists the available versions of spades, which now should list spades/3.15.4

module load spades # loads the default version
module unload spades # unloads spades
module load spades/3.15.4 # loads the specific version

Versioning policies

Please note that software packages as well as the module system undergo a maintenance and update process. As a consequence, the following may happen at LRZ HPC staff's discretion and - unless the user impact is considered very large - without special announcement:

• The default version of a software package may change; the method of loading the default module will however normally stay the same.
• The availability and the module name of non-default versions of a software package may change
• A specific module version may become invisible. This means that it will not be listed anymore by the module avail command because the name or the version of the module begins with a dot. To see all versions of a module including the hidden ones, please use e.g., the bash command

ls -a $(dirname $(module path <package>))

We apologize for the potential impact on the processing of user jobs but given a large number of packages, we make available we presently see no alternative. For a given software package, please check out its documentation page on the LRZ web server for updates and/or use the module avail <package_name> command to see the presently available releases.

Support and sources of information on modules

Further information on modules can be found by reading the man pages on the clusters (module, modulefile). The command "module help" will also provide some information.

The sources can be found on the Sourceforge git repository. For the use of modules, a Tcl (http://www.tcl.tk/) installation is needed as a prerequisite.

Documentation on all commands can be found at https://modules.readthedocs.io.

News on modules can be found at https://twitter.com/EnvModules.