Guies BibTIC: HPC High Performance Computing: 10.3. Running Singularity containers on Slurm

Tools

Introduction

Next section shows the process to run a container from an existing image, the software used to do this task is a wrapper of docker that fit the container inside the resources allocated by the current slurm job. By default Slurm creates cgroups for the jobs and deletes them automatically.

Important information before hands-on:

Singularity containers are able to run in batch or interactive sessions.
X-Sessions aren't possible.
You can't write outside of HOME or /datasets.
Containers and images are destroyed after every execution.

1. Use of Singularity command line

Singularity software is installed locally in compute nodes:

Singularity has many options to interacte with containers, most important are:

 build       Build a Singularity image
 exec        Run a command within a container
 pull        Pull an image from a URI
 push        Upload image to the provided URI
 run         Run the user-defined default command within a container
 shell       Run a shell within a container

Modes

In the following examples, we will run an Abaqus container and we will check the software license state.

1. Batch mode

$ singularity run <singularity-image> "<command>"
$ singularity run /soft/singularity/abaqus_2020-gfortran.sif "/simulia/abaqus licensing lmstat"

2. Interactive mode

$ singularity run <singularity-image>
$ singularity run /soft/singularity/abaqus_2020-gfortran.sif

IMPORTANT: Only images created by the IT department will be located in the /soft/singularity directory. Custom images created by users have to be located in their home directory.

2. [Example] Batch task

Write a Slurm file accordance with the needs of the job to deploy an Abaqus container and run bone simulation:

ohpc $ vi slurm_singularity_abaqus.sh 

#!/bin/bash
#SBATCH --job-name=bone_sim
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=1G
#SBATCH --partition=high
#SBATCH -o %x-%j.out # File to which STDOUT will be written
#SBATCH -e %x-%j.err # File to which STDERR will be written

singularity run /soft/singularity/abaqus_2020-gfortran.sif "/homedtic/user/./bone.sh"

You can create a script file with the commands to run in the singularity container:

ohpc $ vi bone.sh
#!/bin/bash

# Run your abaqus simulation
cd /homedtic/user/bone
/simulia/abaqus job=${SLURM_JOB_NAME} input=0008799900.inp user=sub_bone.f cpus=${SLURM_CPUS_PER_TASK} scratch=${TMPDIR} interactive

Assign execution permissions:

ohpc $ chmod +x bone.sh

As usual submit the task:

ohpc $ sbatch slurm_singularity_abaqus.sh

Read the log file with the results:

ohpc $ cat bone_sim-9132.out

:: initializing oneAPI environment ...
   BASH version = 4.2.46(2)-release
:: clck -- latest
:: compiler -- latest
:: dev-utilities -- latest
:: inspector -- latest
:: itac -- latest
:: mpi -- latest
:: tbb -- latest
:: oneAPI environment initialized ::
Analysis initiated from SIMULIA established products
Abaqus JOB bone_sim
Abaqus 2020
Begin Compiling Abaqus/Standard User Subroutines
Tue Feb 22 16:08:57 2022
End Compiling Abaqus/Standard User Subroutines
Begin Linking Abaqus/Standard User Subroutines
End Linking Abaqus/Standard User Subroutines
Tue Feb 22 16:08:57 2022
Begin Analysis Input File Processor
Tue Feb 22 16:08:57 2022
Run pre
Tue Feb 22 16:09:11 2022
End Analysis Input File Processor
Begin Abaqus/Standard Analysis
Tue Feb 22 16:09:11 2022
Run standard

3. [Example] Interactive task

Request an interactive session to log into a compute node:

ohpc $ salloc -c 4

Two options:

1) Batch container in an interactive slurm job:

node0XX $ singularity run /soft/singularity/abaqus_2020-gfortran.sif "abaqus input=0008799900.inp job=bone user=sub_bone.f cpus=4 inter"

:: initializing oneAPI environment ...
   BASH version = 4.2.46(2)-release
:: clck -- latest
:: compiler -- latest
:: dev-utilities -- latest
:: inspector -- latest
:: itac -- latest
:: mpi -- latest
:: tbb -- latest
:: oneAPI environment initialized ::
Analysis initiated from SIMULIA established products
Abaqus JOB bone
Abaqus 2020

2) Interactive session in an interactive slurm job:

node0XX $ singularity run /soft/singularity/abaqus_2020-gfortran.sif bash

:: initializing oneAPI environment ...
   BASH version = 4.2.46(2)-release
:: clck -- latest
:: compiler -- latest
:: dev-utilities -- latest
:: inspector -- latest
:: itac -- latest
:: mpi -- latest
:: tbb -- latest
:: oneAPI environment initialized ::
Singularity> abaqus input=0008799900.inp job=bone user=sub_bone.f cpus=4 inter

Analysis initiated from SIMULIA established products
Abaqus JOB bone
Abaqus 2020

We can also use the shell option:

nodeXXX $ singularity shell /soft/singularity/abaqus_2020-gfortran.sif
Singularity>