SLURM partitions and nodes

The IGBMC HPC cluster is organized into several SLURM partitions. Each partition gathers a set of compute nodes that have similar usage.

The default partition used by SLURM (igbmc) is suitable for most jobs.

To view all partitions available on the cluster run :

sinfo

Please note that you may not have the rights to use all partitions

To view all available nodes run :

sinfo -Nl

Submitting a job to the cluster

Submitting a batch job

A batch job is a job running in an automated way on the cluster and that doesn't require user interaction.

Usage

The job starts when resources are available. The command only returns the job id. The outputs are sent to file(s). This works ONLY with shell scripts. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input.

Batch scripts rules

The script can contain srun commands. Each srun is a job step. The script must start with shebang (#!) followed by the path of the interpreter

#!/bin/bash

#!/usr/bin/env python

The execution parameters can be set:

At runtime in the command sbatch

sbatch --mem=40GB bowtie2.sbatch

Or within the shell itself

#!/bin/bash
#
#SBATCH --mem 40GB
srun bowtie2 -x hg19 -1 sample_R1.fq.gz -2 sample_R2.fq.gz -S sample_hg19.sam

sbatch bowtie2.sbatch

The scripts can contain slurm options just after the shebang but before the script commands → #SBATCH

Execution parameters

These parameters are common to the commands srun and sbatch.

Parameters for log

#!/bin/bash
#
#SBATCH -o slurm.%N.%j.out  # STDOUT file with the Node name and the Job ID
#SBATCH -e slurm.%N.%j.err  # STDERR file with the Node name and the Job ID

Parameters to control the job

• -partition=<partition_names>, p

Request a specific partition for the resource allocation. Each partition (queue in SGE) have their own limits: time, memory, nodes ...

Cf: sinfo to know which partitions are available.

• -mem=<size[units]>

Specify the real memory required per node. The default units is MB (Default: 2GB)

The job is killed if it exceeds the limit

Note that you can use the variable $SLURM_MEM_PER_NODE in the command line to synchronize the software settings and the resource allocated.

• -time=<time>, t

Set a limit on the total run time of the job allocation.

Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

Parameters for multithreading

• -cpus-per-task=<ncpus>, -cpus, c

Request a number of CPUs (default 1)

Note that you can use the variable $SLURM_CPUS_PER_TASK in the command line to avoid mistake between the resource allocated and the job.

#!/bin/bash
#
#SBATCH --cpus-per-task=8

srun bowtie2 --threads $SLURM_CPUS_PER_TASK -x hg19 -1 sample_R1.fq.gz -2 sample_R2.fq.gz -S sample_hg19.sam

To learn more about the sbatch command, see the official documentation

How to size a job, i.e. how much CPU, memory, other resources should be requested

We do not know about tools that include sizing routines that estimate how much resources are required to process a data set. Therefore these parameters must be selected by a mix of documentation reading, experience, trial-and-error and common sense.

CPUs

If your tool is capable of multithreading (calculations in parallel) you can use more than one CPU. Beware that the program might require that you inform it of how many cpus it should use and therefore you will need to specify that information twice, once for your program and once for Slurm.

There are usually limits to performance gains of parallelization because of various factors such as data input/output rates. If your processing is not a bunch of completely independent programs that can run on different compute nodes, you should investigate how additional cpus lower the processing time and when it becomes useless to add more cpus (rule of thumb : around 4-8 cpus and up a careful examination should be applied).

Memory

The memory required by a program can sometimes be estimated from the dataset that is being processed. If all the data is to be loaded at once you can request the size of the data times some fixed factor.

Otherwise you can gradually increase the requested memory of 50-100% each time your program crashes due to an out of memory error. When it has run successfully you can check how much memory was consumed with the sacct or seff command and fix your request accordingly.

Sizing wisely

The cluster is a shared resource so while it is acceptable to reserve large resources for short tests, you should eventually query them as efficiently as possible because :

• reserved but unused resources negatively impact your “fair use” factor which can lower your priority when the cluster is crowded.
• large jobs may need to wait longer for resources’ availability before they can be scheduled

Full example of the sbatch command

Random

1. Open a script file with any text editor (but not Word)

For beginners, we suggest to use nano, which has restricted functionalities but is quite intuitive.

Copy

nano slurm_random.sh

2. Copy/Paste the following script which is writing 10 000 random numbers in a file and then sort them :
Copy

#!/bin/bash
#
#SBATCH -p fast                      # partition
#SBATCH -N 1                         # nombre de nœuds
#SBATCH -n 1                         # nombre de cœurs
#SBATCH --mem 100                    # mémoire vive pour l'ensemble des cœurs
#SBATCH -t 0-2:00                    # durée maximum du travail (D-HH:MM)
#SBATCH -o slurm.%N.%j.out           # STDOUT
#SBATCH -e slurm.%N.%j.err           # STDERR

for i in {1..10000}; do
  echo $RANDOM >> SomeRandomNumbers.txt
done

sort -n SomeRandomNumbers.txt > SomeRandomNumbers_sorted.txt

Press Ctrl-x to exit nano, then Y when nano asks you whether the modified buffer should be saved, then press the Enter key to confirm the file name.

3. Check the content of the script
Copy

cat slurm_random.sh

4. Submit the job
Copy

sbatch slurm_random.sh

5. Check the result

Since this script is running a very basic task, the results should promptly be available.

Check the output files with ls and head.

☝
These commands can be run on the login node since they are consuming very little computing resources.
Copy

# List the result files
ls -l SomeRandomNumbers*.txt

# Print the 20 first lines of the original random numbers
head -n 20 SomeRandomNumbers.txt

# Print the 20 first lines of the sorted random numbers
head -n 20 SomeRandomNumbers_sorted.txt

# Print the 20 last lines of the sorted random numbers
tail -n 20 SomeRandomNumbers_sorted.txt

Salmon

1. Open a script file with any text editor (but not Word)
Copy

nano slurm_salmon.sh

2. Set the slurm parameters, the [conda] environment and the command itself
Copy

#!/bin/bash
#
#SBATCH -o slurm.%N.%j.out
#SBATCH -e slurm.%N.%j.err
#SBATCH --mail-type END
#SBATCH --mail-user foo.bar@france-bioinformatique.fr
#
#SBATCH --partition fast
#SBATCH --cpus-per-task 6
#SBATCH --mem 5GB

module load salmon

salmon quant --threads $SLURM_CPUS_PER_TASK -i transcripts_index -l A -1 reads1.fq -2 reads2.fq -o transcripts_quant

3. Submit the job
Copy

sbatch slurm_salmon.sh

Interactive job

To submit an interactive job, you will need two SLURM command :

• salloc to request a ressources allocation
• srun to run interactive job steps on an allocation

The salloc command lets you request a ressources allocation and start an interactive session :

salloc takes the same parameters than the sbatch command.

salloc --mem-per-cpu=2G --cpus-per-task=10 

The salloc command returns as soon as the requested ressources have been allocated.

You can then run srun command to start interactive job steps over this allocation :

Example:

srun hostname

The outputs are returned to the terminal.

If you need to interact with the command using your keyboard, add the --pty option to srun :

Example:

module load alphafold/2.0.1
srun --pty python

To relinquish the allocation, use the exit command

exit

To learn more about the salloc command, see the official documentation

Job information

List a user's current jobs:

squeue -u <username>

List a user's running jobs:

squeue -u <username> -t RUNNING

List a user's pending jobs:

squeue -u <username> -t PENDING

View accounting information for all user's job for the current day :

sacct --format=JobID,JobName,User,Submit,ReqCPUS,ReqMem,Start,NodeList,State,CPUTime,MaxVMSize%15 -u <username>

View accounting information for all user's job for the 2 last days (it worth an alias) :

sacct -a -S $(date --date='2 days ago' +%Y-%m-%dT%H:%M) --format=JobID,JobName,User%15,Partition,ReqCPUS,ReqMem,State,Start,End,CPUTime,MaxVMSize -u <username>

List detailed job information:

scontrol show -dd jobid=<jobid>

Manage jobs

To cancel/stop a job:

scancel <jobid>

To cancel all jobs for a user:

scancel -u <username>

To cancel all pending jobs for a user:

scancel -t PENDING -u <username>

• SLURM partitions and nodes
• Submitting a job to the cluster
• Submitting a batch job
• Usage
• Batch scripts rules
• Execution parameters
• How to size a job, i.e. how much CPU, memory, other resources should be requested
• Full example of the sbatch command
• Interactive job
• Job information
• Manage jobs