SLURM ARCHITECTURE
Draco uses a job scheduling system called SLURM to manage the assignment of resources within the cluster. To summarize quickly, users log into login nodes, where no heavy computations are allowed. Login nodes are a shared resource where users can submit jobs to compute nodes using a few commands SLURM provides. This is a simplified overview of the SLURM architecture:
To log in, you can use ssh:
# replace <fsuid> with your actual id like ab12cde
ssh <fsuid>@login1.draco.uni-jena.de
Overview
We have to get appropriate resources allocated for our jobs. Each job needs its OWN script with different resources. These scripts are a great way to keep track of what resources you needed for the program and also what parameters your program required.
salloc
To get resources for an interactive session on the short node for 1 hour with 10 cpus and 1G memory - maybe for testing a program or script
salloc --partition=short --time=01:00:00 --cpus-per-task=10 --mem=1G
You can run the sbatch scripts introduced below in this interactive session with bash your_script.sbatch, which will ignore the parameters set in them for slurm.
sbatch script
To submit a job, you have to make a script my_slurm_script.sh with follow headers. (Change the parameters)
A template sbatch script
#!/bin/bash
#SBATCH --tasks=1
#SBATCH --cpus-per-task=<threads>
#SBATCH --partition=<partitions,listed,here>
#SBATCH --mem=<>
#SBATCH --time=<hh:mm:ss>
#SBATCH --job-name=<job_name_id>
#SBATCH --output=<outdir>/<tool>.slurm.out.%j
#SBATCH --error=<outdir>/<tool>.slurm.err.%j
# activate the conda environment
source /path/to/miniconda3/etc/profile.d/conda.sh && conda activate <tool_env>
echo date
my commands here
Resources
These options tell slurm:
- how many tasks (a slurm term) you want to run
- how many threads you want per task (commonly called “threads” parameters for bioinformatic programs - this has to match)
- how much memory you want allocated. The maximum memory allowed here is 250GB - but this is a lot and none of the programs we use require this much for our application.
- how long we expect our job to run - an educated guess.
#SBATCH --tasks=1
#SBATCH --cpus-per-task=<threads>
#SBATCH --mem=<>
#SBATCH --time=<hh:mm:ss>
Other options
These tell slurm:
- your job name
- which partitions you want to run the task on. We will primarily be using the “short” and “standard” partitions
- standard output from the program
- standard error from the program - usually has more helpful info if they program crashes
#SBATCH --job-name=<job_name_id>
#SBATCH --partition=<partitions,listed,here>
#SBATCH --output=<outdir>/<tool>.slurm.out.%j
#SBATCH --error=<outdir>/<tool>.slurm.err.%j
Example sbatch script
#!/bin/bash
#SBATCH --tasks=1
#SBATCH --cpus-per-task=10
#SBATCH --partition=short,standard
#SBATCH --mem=1G
#SBATCH --time=2:00:00
#SBATCH --job-name=sequencing_QC
#SBATCH --output=10_nanoplot/nanoplot.slurm.out.%j
#SBATCH --error=10_nanoplot/nanoplot.slurm.err.%j
# First, we check the quality of the reads using nanoplot
# activate the conda environment containing nanoplot
source /vast/groups/VEO/tools/anaconda3/etc/profile.d/conda.sh && conda activate nanoplot_v1.41.3
Submitting the script
sbatch my_slurm_script
Other useful slurm commands
# submit a job
sbatch
# check on your jobs
squeue -u <fsuid>
squeue --me
# how many resources are being consumed
sstat <job id>
# kill a job
scancel <job id>
# information about nodes
sinfo --long
scontrol show node