previous episode

Viromics2026

next episode

Draco Architecture and submitting Jobs with sbatch

Overview

Teaching: 20 min
Exercises: 20 min
Objectives

  • Describe the difference between a login node and a compute node

  • Interpret an sbatch script’s header and commands before running it

  • Submit a minimal sbatch script

  • Monitor a running job with squeue

  • Locate and read .out and .err log files

Where have you actually been working?

So far today, almost everything you’ve typed (navigating folders, copying your sample data) has run on what’s called the login node: the machine you land on when you connect to Draco. Think of it as the reception desk of the cluster: great for finding your way around, moving files, and light tasks, but it’s a single machine, shared by everyone using Draco at that moment.

Any real computational analysis, the kind we’ll do during the rest of the course, needs much more CPU and memory and often for longer periods. If everyone who uses Draco ran that directly on the login node simultaneously it would slow to a crawl (or crash) for everyone.

Draco solves this with compute nodes: many other machines dedicated to actually running the work, that you can request access to.

The job scheduler: Slurm

You don’t get to pick a compute node yourself. Instead, you describe what you need (how many CPUs, how much memory, roughly how long it’ll take) to a job scheduler called Slurm, and it finds you an available compute node to run on. This keeps things fair: everyone’s jobs queue up and run as resources free up, rather than everyone fighting over the same login node.

There are two ways to get access to a compute node:

We’ll use sbatch for almost everything in this course using provided scripts, so that you don’t have to program much yourself.

The figure below provides a simplified overview of the SLURM/DRACO architecture: slurm_architecture_simple

Anatomy of an sbatch script

Each job needs its own script in which you specify what computational resources you need for the analysis you want to run. These and other parameters are defined in the header of the script (lines starting with #):

#!/bin/bash
#SBATCH --tasks=1
#SBATCH --cpus-per-task=<threads>
#SBATCH --partition=<partitions,listed,here>
#SBATCH --mem=<memory>
#SBATCH --time=<hh:mm:ss>
#SBATCH --job-name=<job_name_id>
#SBATCH --output=<outdir>/<tool>.slurm.out.%j
#SBATCH --error=<outdir>/<tool>.slurm.err.%j

The actual commands you want to execute on the compute node

1. Your first job

Exercise 1 — reading an SBATCH script

Inspect the 1.0_SLURM/1.0_10_test.sbatch script found in the Viromics2026_workspace folder (don’t submit it yet).

Look at the header, then answer:

  • How many CPUs and how much memory is this job requesting?
  • What do you think the other parameters mean? Do you spot anything that might be particularly interesting?

Walk through the three commands after the #SBATCH lines, in order.

  • What do you think will happen when this script runs?
  • Predict: when this job runs, what do you expect the hostname command to print? Will it match the login node’s hostname you’ve been seeing all day, or something different? Discuss with a neighbour. If you are unsure, have a look at the SLURM figure above.

Solution

  • CPU and 500 MB of memory.
  • --time:  maximum runtime; job is killed if it exceeds this.
--partition:  which shared partitions to run on (we’ll mostly use short and standard).
--job-name: a name for your job. You might want to change it to something meaningful. 
--output: standard output: normal messages that would otherwise be printed to the terminal. 
--error: standard error: warnings, errors, crash messages that would otherwise be printed to the terminal. Check this first when something goes wrong.

    The output file and error file are the main ways to keep track of what your script has been doing.

  • It prints a greeting that includes what hostname returns, waits 20 seconds doing nothing, then prints a second message.
  • You should expect a different hostname than the login node’s. Slurm will assign this job to whichever compute node is available, not the login node you’re typing commands into.

Now let’s find out if your prediction was right.

Excercise 2 - Submitting your first job

  1. Navigate to the 1.0_SLURM/ folder in the terminal and type: 
    sbatch 1.0_10_test.sbatch
  2. Immediately after submitting the script, run squeue --me a few times in a row. What do you observe about the job’s state over time?

Solution

  1. You should see something like Submitted batch job 123456. That number is your job ID. You can you it to check on or cancel the job.
  2. You should see the job listed with a state like PD (pending) and then R (running), and then it disappears from the list once finished. the whole job takes about 20 seconds, so watch closely!

2. Reading the logs

Once test.sbatch has finished (it disappears from squeue --me) answer the following questions:

Exercise 3

  1. Check the contents of 1.0_SLURM/. You should see two new files (you might have to press refresh in VS Code).
  2. Inspect the .out file. Does its content match what you expect from the echo commands?
  3. Inspect the .err file. Is it empty? What does an empty .err file usually mean?

Solution

  1. hello.slurm.out.<jobid> should contain hello from <compute node hostname> and done sleeping — notice the hostname is different from the login node’s, exactly as you hopefully predicted earlier.
    1. hello.slurm.err.<jobid> should be empty, because nothing went wrong. An empty error log is a good sign, not a bug.

Exercise 4 (optional) — diagnose a broken job

Debugging failed jobs is one of the most useful skills for the rest of this course. If you have time practice it now, otherwise feel free to skip ahead to the checkpoint below.

Submit the job 1.0_20_broken.sbatch script.

This job will fail. Without asking a TA first, use the .out and .err files to figure out why.

Solution

The .err file should show something like CondaEnvironmentNotFoundError — the environment nanoplot_v9.99.9 doesn’t exist (the real one is nanoplot_v1.41.3). This is the single most common failure mode you’ll see this week: a typo in a conda environment name, a wrong file path, or a missing flag. Always check the .err file first.

(Optional reading) More information on Draco

Some useful commands:

squeue --me          # see your own jobs
squeue -u <fsuid>    # see jobs for a specific user
sstat <job_id>       # resource usage of a running job
scancel <job_id>     # cancel a job

Key Points

  • Draco has a login node for light tasks and many compute nodes for resource-heavy hobs

  • Because Draco is shared, the Slurm job scheduler decides which compute node your work runs on and when

  • An sbatch script describes both the resources you need and the commands to run

  • .out holds normal output

  • .err holds error messages — check .err first when a job fails

previous episode

next episode