HPC High Performance Computing: 7. Working with Conda and Slurm

Depending on your needs, running your simulation using a virtual environment is more suitable for you. In this section, we will focus on how we can create a Conda virtual environment in the HPC cluster.

First of all, we will submit an interactive job. Before the virtual environment creation, it will be necessary to load the Anaconda3 or Miniconda3 module in order to use Conda in the HPC cluster.

[nsurname@ohpc ~]$ salloc
salloc: Granted job allocation 411436
salloc: Waiting for resource configuration
[nsurname@node007 ~]$ module load Miniconda3/4.9.2

Then, we will have to add the following line in order to set up shell functions for Conda:

[nsurname@node007 ~]$ eval "$(conda shell.bash hook)"
(base) [nsurname@node007 ~]$

After that, we will proceed with the environment creation (in this example, with a Python 3.9 as the base):

(base) [nsurname@node007 ~]$ conda create -n environment python==3.9

We will activate the environment. If we have not installed all required software in the previous step, we can use "pip install":

(base) [nsurname@node007 ~]$ conda activate environment
(environment) [nsurname@node007 ~]$ pip install numpy matplotlib

Finally, with the environment enabled, we will run the Python script:

(environment) [nsurname@node007 ~]$ python python-script.py

In case that we want to submit a batch job, we will configure the script with the following way:

#!/bin/bash
#SBATCH -J prova_uname10
#SBATCH -p short
#SBATCH -N 1
#SBATCH -n 2 
#SBATCH --chdir=/homedtic/nsurname/slurm_jobs
#SBATCH --time=2:00
#SBATCH -o %N.%J.out # STDOUT
#SBATCH -e %N.%j.err # STDERR

#Load Miniconda module 
module load Miniconda3/4.9.2

#Enable the bash shell
eval "$(conda shell.bash hook)"

#Enable the conda environment
conda activate environment

#Run your script
python python-script.py

For more information, please, take a look to the official Conda webpage: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html 

If you already have an existing environment and you want to export it to your HPC home, you will have to do the following steps:

Firstly, enable your environment and export that in .yml file format:

conda activate environment
conda env export > environment.yml

Secondly, transfer this file by SFTP to your HPC home.

Thirdly, edit the environment.yml file replacing the path prefix for your home directory. For example: prefix: /home/nsurname/.conda/envs/environment

Fourthly, submit an interactive job and load the Miniconda3 or the Anaconda3 module:

[nsurname@ohpc ~]$ salloc
salloc: Granted job allocation 411436
salloc: Waiting for resource configuration
[nsurname@node007 ~]$ module load Miniconda3/4.9.2

Then, add the following line in order to establish the bash session for Conda:

[nsurname@node007 ~]$ eval "$(conda shell.bash hook)"
(base) [nsurname@node007 ~]$

Finally, create the virtual environment from your environment.yml file:

(base) [nsurname@node007 ~]$ conda env create -f environment.yml