Introduction
To simplify setting up environments for compilers, MPI-versions and applications, Colossus has the Environment Modules package installed. This provides a means for dynamic modification of a user's environment via modulefiles.
Each modulefile contains the information needed to configure the shell for an environment. Typically, a modulefile modifies environment variables like PATH, MANPATH, LD_LIBRARY_PATH, CPATH and PKG_CONFIG_PATH to give access to an application or library. To invoke a module file you run the module command with arguments.
In TSD, the module command can be used from the command line on submit hosts and as part of a job script when submitting a job to Colossus. You can reach the submit host assigned to your project by connecting to pXX-hpc-01.tsd.usit.no using PUTTY from your Windows VM, or by using ssh from your linux VM.
Listing available modules
The available modulefiles can be listed by module avail. Example output:
$ module avail ----------------- /cluster/software/MODULEFILES/custom ----------------------- atlas-lapack/3.10.3 matlab/R2018b bcftools/1.8 netcdf-fortran.gnu/4.4.4 bedtools/2.17.0 netcdf.gnu/4.6.1 bismark/0.22.1 openmpi.gnu/1.8 bowtie/1.2.2 openmpi.gnu/3.1.3 [...] ----------------- /cluster/software/MODULEFILES/easybuild/all ---------------- AFNI/18.3.00-foss-2018b-Python-3.6.6 libreadline/7.0-GCCcore-6.4.0 Anaconda3/5.3.0 libreadline/7.0-GCCcore-7.3.0 arpack-ng/3.7.0-foss-2019a libreadline/8.0-GCCcore-8.2.0 [...]
You'll see custom and Easybuild modules. The Easybuild modules are usually the most recent and should be preferred. When loading a custom module a notification will be displayed to clarify this.
You can also specify a name:
$ module avail MATLAB ----------------- /cluster/software/MODULEFILES/easybuild/all ---------------- MATLAB/2017b MATLAB/2018a MATLAB/2018b MATLAB/2019a MATLAB/2019b MATLAB/2020a MATLAB/2020b
Loading modules
Modulefiles can be used from most shells, as well as Perl and Python. Modulefiles can be loaded an unloaded on the command line or within your scripts by typing:
module load software/version module unload software/version
If a version is not specified, no software will be loaded. For the purpose of reproducibility, we enforce users to load specific versions of modulefiles.
You will get a list of currently loaded modulefiles by:
$ module list Currently Loaded Modulefiles: 1) MATLAB/2020b
To view what exactly a modulefile is doing to your environment, type:
module show modulefile/version
Switch to a different version of a module
Switching to another version is similar to loading a specific version. As an example, if you want to switch from the current loaded R-bundle-Bioconductor to an more recent one:
module switch R-bundle-Bioconductor/3.9-foss-2019a-R-3.6.0 R-bundle-Bioconductor/3.11-foss-2020a-R-4.0.0
This, more compact syntax will fortunately also work:
module switch R-bundle-Bioconductor R-bundle-Bioconductor/3.11-foss-2020a-R-4.0.0
Loading multiple modules simultaneously
A module loads all necessary dependencies along with the parent module. This can lead to conflicts when different modules have conflicting dependencies, e.g. they depend on conflicting versions of the tool chains (foss, GCC, GCCcore). We recommend loading the modules sequentially and resolve any conflicts. When modules get reloaded with a version change this indicates a conflict and will break the dependencies of the previously loaded module(s), e.g.:
module load R-bundle-Bioconductor/3.11-foss-2020a-R-4.0.0 module load libGLU/9.0.0-foss-2018b The following have been reloaded with a version change: 1) FFTW/3.3.8-gompi-2020a => FFTW/3.3.8-gompi-2018b 2) GCC/9.3.0 => GCC/7.3.0-2.30 3) GCCcore/9.3.0 => GCCcore/7.3.0 4) GMP/6.2.0-GCCcore-9.3.0 => GMP/6.1.2-GCCcore-7.3.0
Purge modules
We recommend using
module purge
prior to any module load commands in job scripts, to prevent inheriting environment variables set by module commands in the shell used when submitting the job.
Help
A help text exists for most of the modulefiles. To view it, type:
module help modulefile/version
For further documentation, please refer to the Environment Modules homepage.
Personalize default modulefiles
You may add loading of modulefiles to your .bash_login file to make sure your favorite modulefile is always loaded when logging in. For instance, to always have the MATLAB/2020b modulefile load on login, add:
module load MATLAB/2020b
to the .bash_login file.