Skip to content

Slurm Job Submission and Execution

Important

Slurm scheduler is currently in use on Complementary System only. Starting July 17., Slurm will be in use on the Barbora cluster.

Slurm Introduction

Slurm is a workload manager which facilitates access to cluster resources, and manages scheduling, starting, and executing jobs, and job monitoring. We highly encourage you have a look at the interactive Slurm tutorial.

A man page exists for all Slurm commands, as well as --help command option, which provides a brief summary of options. Slurm documentation and man pages are also available online.

To view all of the Slurm's environment variables, use the command:

set | grep ^SLURM

Quick Overview of Common Commands

Command Explanation
sacct Display accounting data for all jobs and job steps in the job accounting log or Slurm database.
salloc Obtain a job allocation (a set of nodes), execute a command, and then release the allocation when the command is finished.
sattach Attach to a job step.
sbatch Submit a batch script to Slurm.
sbcast Transmit a file to the nodes allocated to a job.
scancel Used to signal jobs or job steps that are under the control of Slurm.
sinfo View information about nodes and partitions.
squeue View information about jobs located in the scheduling queue.
srun Run parallel jobs.

Job Submission Options

Slurm provides three different commands for job submission: salloc, srun, and sbatch. Each of those serves a slightly different purpose; salloc is used to obtain an allocation, srun runs parallel jobs (and obtains appropriate allocation first if necessary), and serves as Slurm's native way of executing MPI-enabled applications, and sbatch reads a batch script, asks for required allocation, and then executes it. While all of these may be used interchangeably, their specifics make them more useful in different scenarios:

Command Preferred use case
sbatch Production jobs
salloc Interactive jobs, debugging
srun Quick interactive jobs, short analysis of output data, running MPI applications

Batch Mode

sbatch command can be used to submit a script for later execution. A batch script usually consists of three parts:

  • interpreter used for code execution, denoted by shebang sign (#!) followed by an absolute path to the interpreter
  • job options, denoted by a hash sign (#), a keyword for scheduler to pick upon, and submission options
  • job instructions; these contain everything an interpreter should do once the job has been initialized, including scheduler-independent environment setup

Your most commonly used interpreter will be bash, or some other shell command language interpreter, such as dash, zsh, etc. You would then define your interpreter on the first line of your batch script like this:

#!/usr/bin/bash

To define Slurm job options within the batch script, use SBATCH keyword followed by your job option. Multiple SBATCH statements can be specified:

#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4

Here we asked for 4 tasks in total to be executed on partition p03-amd using OPEN-00-00's project resources.

Job instructions should contain everything you'd like your job to do; that is, every single command the job is supposed to execute:

ml OpenMPI/4.1.4-GCC-11.3.0

srun hostname | uniq -c && \
        exit 0 || \
        echo "Unable to execute MPI job!"

exit 1

Combined together, the previous examples make up a following script:

#!/usr/bin/bash
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4

ml OpenMPI/4.1.4-GCC-11.3.0

srun hostname | uniq -c && \
        exit 0 || \
        echo "Unable to execute MPI job!"

exit 1

And by submitting it from the login node of Complementary System

$ sbatch ./batchscript.sh
Submitted batch job 1511

we get an output file with the following contents:

$ cat slurm-1511.out
      1 p03-amd01.cs.it4i.cz
      3 p03-amd02.cs.it4i.cz

Notice that Slurm spread our job across 2 different nodes; by default, Slurm selects the number of nodes to minimize wait time before job execution. However, sometimes you may want to restrict your job to only a certain minimum or maximum number of nodes (or both). You may also require more time for your calculation to finish than the default allocated time. For an overview of such job options, see table below.

Quick Overview of Common Batch Job Options

Job Option Specification
Job Name -J, --job-name=<name>
Account -A, --account=<account>
Queue / Partition -p, --partition=<partition_names>
Node Count -N, --nodes=<minnodes>[-maxnodes]
Total Task Count -n, --ntasks=<number>
Tasks Per Node* --ntasks-per-node=<ntasks>
CPUs Per Task* -c, --cpus-per-task=<ncpus>
Wall Clock Limit -t, -t <time>
Copy Environment --export={[ALL,]<environment_variables>|ALL|NONE}
Job Dependency -d, --dependency=<dependency_list>
Job Arrays -a, --array=<indexes>
  • When the --ntasks-per-node option is used in conjunction with --ntasks, it is treated as a maximum count of tasks per node instead. For more information, see the man page.

  • Beginning with 22.05, srun will not inherit the --cpus-per-task value requested by salloc or sbatch. It must be requested again with the call to srun or set with the SRUN_CPUS_PER_TASK environment variable if desired for the task(s).

Common Job Workflow

Note

Production jobs must use the /scratch directory for I/O

The recommended way to run production jobs is to change to the /scratch directory early in the jobscript, copy all inputs to /scratch, execute your calculations, and copy outputs back to the /home directory. While it is possible to prepare and submit your jobs from the /scratch itself, note that it is meant only for temporary data storage during job execution; files older than 90 days may be automatically deleted.

#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4

cd $SLURM_SUBMIT_DIR

SCRDIR="/scratch/project/open-00-00/${USER}/myjob"
mkdir -p "${SCRDIR}"

# change to scratch directory, exit on failure
cd "${SCRDIR}" || exit 1

# copy input files to scratch
cp "${SLURM_SUBMIT_DIR}/input" .
cp "${SLURM_SUBMIT_DIR}/mympiprog.x" .

# load required module(s)
# (Always specify the module's name and version in your script;
#  see https://docs.it4i.cz/software/modules/lmod/#loading-modules.)
ml OpenMPI/4.1.4-GCC-11.3.0

# execute the calculation
srun ./mympiprog.x

# copy output file to home
cp output "${SLURM_SUBMIT_DIR}/"

exit

In this example, a directory in /home holds the input file input and the mympiprog.x executable. We create the myjob directory on the /scratch filesystem, copy input and executable files from the /home directory where the sbatch was invoked ($SLURM_SUBMIT_DIR) to /scratch, execute the MPI program mympiprog.x, and copy the output file back to the /home directory. mympiprog.x is run with 4 tasks, allocated according to Slurm configuration.

Selecting the Number of Nodes

Some jobs may run significantly slower when the tasks are spread too thin; in that case, it may be beneficial to specify the number of nodes:

#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -n 4
#SBATCH -N 2

srun hostname | sort | uniq -c

exit

The -N, or --nodes option takes two arguments: minnodes, minimum number of nodes to allocate, and maxnodes, maximum number of nodes to allocate. When only one number is specified, it means to allocate exectly the specified number of nodes. Output of the above batchscript may then look something like this:

$ cat slurm-1642.out
      2 p01-arm01.cs.it4i.cz
      2 p01-arm02.cs.it4i.cz

On the other hand, you may also want to allow Slurm some flexibility in scheduling your job, which may result in earlier job execution. Taking the last jobscript as an example, Slurm may not be able to satisfy a request for 4 tasks on 2 nodes. In that case, the following jobscript

#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -n 4
#SBATCH -N 2-4

srun hostname | sort | uniq -c

exit

may run earlier than jobs asking strictly for 4 tasks on 2 nodes, because the number of required slots may be easier to satisfy by simply spreading them out across more nodes:

$ cat slurm-1643.out
      1 p01-arm01.cs.it4i.cz
      1 p01-arm02.cs.it4i.cz
      1 p01-arm03.cs.it4i.cz
      1 p01-arm04.cs.it4i.cz

Interactive Mode

Sometimes you may want to run your job interactively, for example for debugging, running your commands one by one from the command line. Slurm provides two diferrent commands for these cases, each with a slightly different purpose: salloc and srun. Both of these take options similar to sbatch.

Resource Allocation

salloc requests an allocation, and once obtained, starts a shell on one of the nodes (master node). For example, you can allocate 2 nodes for 10 minutes with the following command:

user@login$ salloc -N 2 -A OPEN-00-00 -p p01-arm -t 10
salloc: Granted job allocation 1539
salloc: Waiting for resource configuration
salloc: Nodes p01-arm[01-02] are ready for job

user@p01-arm01$

Here we can see that Slurm started a shell session on node p01-arm01. To start a parallel execution on allocated nodes, you can run:

$ srun hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz

To finish the job, you can either use the exit keyword, or Ctrl+D (^D) control sequence:

$ exit
salloc: Relinquishing job allocation 1539

If your job exceeds either the specified time limit or the maximum possible time limit for the selected partition, you will instead be booted out by the scheduler with the following message:

$ salloc: Job 1544 has exceeded its time limit and its allocation has been revoked.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 1544.interactive ON p01-arm01 CANCELLED AT 2023-02-07T15:18:07 DUE TO TIME LIMIT ***
exit
$

Job Submission

Note

srun functions as Slurm's native way of executing MPI-aware applications, with several benefits over mpirun/mpiexec; it knows the exact machine configuration, job allocation, and Slurm's job environment variables. While our OpenMPI modules are compiled with Slurm support, and Intel MPI has its own support built-in, some applications use their own MPI implementation which may require you to specify additional parameters (such as number of tasks) for a job to run in the desired way. For this reason, we recommend you always try to substitute the mpirun/mpiexec commands in your job scripts with srun.

srun is used to submit a job for real time execution, obtains an allocation first if necessary, and also serves as Slurm's native way of executing MPI jobs. For example, to launch a job on 2 nodes from a login node, you can run:

$ srun -N 2 -p p01-arm -A OPEN-00-00 hostname
p01-arm02.cs.it4i.cz
p01-arm01.cs.it4i.cz

Similarly, you can use srun from within the allocation obtained via salloc, both utilizing either whole or part of the whole allocation:

$ salloc -N 4 -p p01-arm -A OPEN-00-00
salloc: Granted job allocation 1551
salloc: Waiting for resource configuration
salloc: Nodes p01-arm[01-04] are ready for job
$ srun hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
p01-arm03.cs.it4i.cz
p01-arm04.cs.it4i.cz
$ srun -N 2 hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz

Job Dependency Submission

To submit dependent jobs in sequence, use the -d, or --dependency flag. sbatch also provides --parsable flag, which conveniently outputs only the job ID number, and the cluster name if present, and can thus be used to submit a large number of jobs dependent on each other in an automated way. For example, to submit 3 jobs, where the next one always depends on the previous one, we can use:

$ first=$(sbatch --parsable job1.sh)
$ second=$(sbatch --parsable --dependency=afterok:${first} job2.sh)
$ sbatch --parsable --dependency=afterok:${second} job3.sh
1581
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1579   p01-arm     job1  opr0019 PD       0:00      8 (Priority)
              1580   p01-arm     job2  opr0019 PD       0:00      8 (Dependency)
              1581   p01-arm     job3  opr0019 PD       0:00      8 (Dependency)
$ scontrol show job 1580 | grep JobState
   JobState=PENDING Reason=Dependency Dependency=afterok:1579(unfulfilled)
$ scontrol show job 1581 | grep JobState
   JobState=PENDING Reason=Dependency Dependency=afterok:1580(unfulfilled)

Multiple job dependencies can be specified by using the colon (:) separator for job IDs, and comma (,) separator for different job exit statuses:

$ sbatch --dependency=afterok:1585:1587,afternotok:1581
Submitted batch job 1596
$ scontrol show job 1596 | grep JobState
   JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled),afterok:1587(unfulfilled),afternotok:1581(unfulfilled)

Job dependencies can also be used for submitting long running jobs for which the maximum allowed wall time of partition is insufficient. The only prerequisite is the program has a way of safely saving its progress before it actually reaches end (for example, by creating a STOPCAR file when using VASP). If the job is capable of this, we can submit a job multiple times in a loop, restarting it each time until it finishes:

$ previous=$(sbatch --parsable jobstart.sh)
$ for id in $(seq 2 6)
> do
> submit=$(sbatch --parsable --dependency=afterok:${previous} restart.sh)
> previous=$submit
> done
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1582   p01-arm jobstart  opr0019 PD       0:00      8 (Priority)
              1583   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1584   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1585   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1586   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1587   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
$ for id in $(seq 1582 1587)
> do
> scontrol show job $id | grep JobState
> done
   JobState=PENDING Reason=Priority Dependency=(null)
   JobState=PENDING Reason=Dependency Dependency=afterok:1582(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1583(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1584(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1586(unfulfilled)

Here we have two different submission scripts; one for the first iteration, and one for restarts, since it is often necessary to use a slightly altered input file in order for the job to restart from saved state. As we can see, only one of the jobs will be eligible to run at a time, since the rest of them depend on the correct exit status (zero) of the previous one.

Overview of Job Dependency Options

Job Status Format Explanation
after:job_id[[+time][:jobid[+time]...]] After the specified jobs start or are cancelled and 'time' in minutes from job start or cancellation happens, this job can begin execution. If no 'time' is given then there is no delay after start or cancellation.
afterany:job_id[:jobid...] This job can begin execution after the specified jobs have terminated.
afterburstbuffer:job_id[:jobid...] This job can begin execution after the specified jobs have terminated and any associated burst buffer stage out operations have completed.
aftercorr:job_id[:jobid...] A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully (ran to completion with an exit code of zero).
afternotok:job_id[:jobid...] This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc.).
afterok:job_id[:jobid...] This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero).
singleton This job can begin execution after any previously launched jobs sharing the same job name and user have terminated. In other words, only one job by that name and owned by that user can be running or suspended at any point in time. In a federation, a singleton dependency must be fulfilled on all clusters unless DependencyParameters=disable_remote_singleton is used in slurm.conf.

Job Management

Apart from job submission and execution, Slurm also provides a number of commands dedicated to various job management purposes, such as viewing current state of the job queue, printing information about existing partitions and their configurations, showing job priorities and how they were arrived at, displaying accounting data and many other.

Job Partition Information

To view information about available job partitions, use the sinfo command:

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
p00-arm      up 1-00:00:00      1   idle p00-arm01
p01-arm*     up 1-00:00:00      8   idle p01-arm[01-08]
p02-intel    up 1-00:00:00      2   idle p02-intel[01-02]
p03-amd      up 1-00:00:00      2   idle p03-amd[01-02]
p04-edge     up 1-00:00:00      1   idle p04-edge01
p05-synt     up 1-00:00:00      1   idle p05-synt01

Here we can see output of the sinfo command ran on the Complementary System. By default, it shows basic node and partition configurations.

To view partition summary information, use sinfo -s, or sinfo --summarize:

$ sinfo -s
PARTITION AVAIL  TIMELIMIT   NODES(A/I/O/T) NODELIST
p00-arm      up 1-00:00:00          0/1/0/1 p00-arm01
p01-arm*     up 1-00:00:00          0/8/0/8 p01-arm[01-08]
p02-intel    up 1-00:00:00          0/2/0/2 p02-intel[01-02]
p03-amd      up 1-00:00:00          0/2/0/2 p03-amd[01-02]
p04-edge     up 1-00:00:00          0/1/0/1 p04-edge01
p05-synt     up 1-00:00:00          0/1/0/1 p05-synt01

This lists only a partition state summary with no dedicated column for partition state. Instead, it is summarized in the NODES(A/I/O/T) column, where the A/I/O/T stands for allocated/idle/other/total.

sinfo can also report more granular information, such detailed exact node-oriented information:

$ sinfo -Nel
Mon Feb 13 10:06:54 2023
NODELIST     NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
p00-arm01        1   p00-arm        idle 64     64:1:1 260988        0      1 aarch64, none
p01-arm01        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm02        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm03        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm04        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm05        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm06        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm07        1  p01-arm*        idle 48     48:1:1  32132        0      1 aarch64, none
p01-arm08        1  p01-arm*        idle 48     48:1:1  32132        0      1 aarch64, none
p02-intel01      1 p02-intel        idle 64     64:1:1 257252        0      1 x86_64,i none
p02-intel02      1 p02-intel        idle 64     64:1:1 257252        0      1 x86_64,i none
p03-amd01        1   p03-amd        idle 64     64:1:1 257270        0      1 x86_64,a none
p03-amd02        1   p03-amd        idle 64     64:1:1 257270        0      1 x86_64,a none
p04-edge01       1  p04-edge        idle 16     16:1:1 128433        0      1 x86_64,i none
p05-synt01       1  p05-synt        idle 8       8:1:1 128303        0      1 x86_64,a none

For more information about the sinfo command, its flags, and formatting options, see the manual, either by using the man sinfo command or online.

Job Queue Information

To view information about queued jobs, use the squeue command:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1556   p01-arm interact  opr0019  R       1:07      8 p01-arm[01-08]
              1558 p02-intel interact easybuil CD       0:21      2 p02-intel[01-02]
              1557   p03-amd interact  opr0019  R       0:57      1 p03-amd01

By default, this shows the job ID, partition, name of the job, job owner's username, job state, how long has the job been already running, number of allocated nodes, and a list of allocated nodes.

To view jobs only belonging to a particular user, you can either use --user=<username>, or --me, which serves as an alias for --user=$USER, to shows only your jobs:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1559   p01-arm interact  opr0019  R       3:37      8 p01-arm[01-08]
              1560 p02-intel interact easybuil  R       0:05      2 p02-intel[01-02]
              1557   p03-amd interact  opr0019  R      10:22      1 p03-amd01
$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1559   p01-arm interact  opr0019  R       4:04      8 p01-arm[01-08]
              1557   p03-amd interact  opr0019  R      10:49      1 p03-amd01

squeue also allows for printing information about specific jobs using the --jobs flag:

$ squeue --jobs 1557,1560
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1560 p02-intel interact easybuil  R       2:09      2 p02-intel[01-02]
              1557   p03-amd interact  opr0019  R      12:26      1 p03-amd01

For more information about the squeue command, its flags, and formatting options, see the manual, either by using the man squeue command or online.

Job State Codes Overview

Code Job State Explanation
BF BOOT_FAIL Job terminated due to launch failure, typically due to a hardware failure (e.g. unable to boot the node or block and the job can not be requeued).
CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
CD COMPLETED Job has terminated all processes on all nodes with an exit code of zero.
CF CONFIGURING Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
CG COMPLETING Job is in the process of completing. Some processes on some nodes may still be active.
DL DEADLINE Job terminated on deadline.
F FAILED Job terminated with non-zero exit code or other failure condition.
NF NODE_FAIL Job terminated due to failure of one or more allocated nodes.
OOM OUT_OF_MEMORY Job experienced out of memory error.
PD PENDING Job is awaiting resource allocation.
PR PREEMPTED Job terminated due to preemption.
R RUNNING Job currently has an allocation.
RD RESV_DEL_HOLD Job is being held after requested reservation was deleted.
RF REQUEUE_FED Job is being requeued by a federation.
RH REQUEUE_HOLD Held job is being requeued.
RQ REQUEUED Completing job is being requeued.
RS RESIZING Job is about to change size.
RV REVOKED Sibling was removed from cluster due to other cluster starting the job.
SI SIGNALING Job is being signaled.
SE SPECIAL_EXIT The job was requeued in a special state. This state can be set by users, typically in EpilogSlurmctld, if the job has terminated with a particular exit value.
SO STAGE_OUT Job is staging out files.
ST STOPPED Job has an allocation, but execution has been stopped with SIGSTOP signal. CPUS have been retained by this job.
S SUSPENDED Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.
TO TIMEOUT Job terminated upon reaching its time limit.

Job Cancelation

scancel is used to send signals or cancel jobs. For example, to cancel a specific job, you would run:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1563   p01-arm interact  opr0019 PD       0:00      3 (Resources)
              1562   p01-arm interact  opr0019  R       0:20      8 p01-arm[01-08]
              1561   p03-amd interact  opr0019  R       0:25      1 p03-amd01
$ scancel 1562
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1562   p01-arm interact  opr0019 CA       0:57      8 p01-arm[01-08]
              1563   p01-arm interact  opr0019  R       0:03      3 p01-arm[01-03]
              1561   p03-amd interact  opr0019  R       1:06      1 p03-amd01

To cancel multiple jobs, simply list all of their job IDs:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1562   p01-arm interact  opr0019 CA       0:57      8 p01-arm[01-08]
              1564   p01-arm interact  opr0019 PD       0:00      8 (Resources)
              1563   p01-arm interact  opr0019  R       3:31      3 p01-arm[01-03]
              1561   p03-amd interact  opr0019  R       4:34      1 p03-amd01
              1565   p03-amd interact  opr0019  R       0:07      1 p03-amd01
$ scancel 1562 1563 1561 1565 1564
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1562   p01-arm interact  opr0019 CA       0:57      8 p01-arm[01-08]
              1563   p01-arm interact  opr0019 CA       4:01      3 p01-arm[01-03]
              1564   p01-arm interact  opr0019 CA       0:00      8
              1561   p03-amd interact  opr0019 CA       5:04      1 p03-amd01
              1565   p03-amd interact  opr0019 CA       0:37      1 p03-amd01

Slurm also allows for canceling only jobs which fulfill certain criteria, for example, the partition to which they have been submitted, or their job state (or a mixture of both):

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1569   p01-arm interact  opr0019 PD       0:00      3 (Resources)
              1571   p01-arm interact  opr0019 PD       0:00      3 (Priority)
              1568   p01-arm interact  opr0019  R       1:52      8 p01-arm[01-08]
              1566   p03-amd interact  opr0019  R       1:55      1 p03-amd01
              1567   p03-amd interact  opr0019  R       1:53      1 p03-amd01
              1570   p03-amd interact  opr0019  R       0:26      1 p03-amd01
$ scancel --partition p03-amd
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1569   p01-arm interact  opr0019 PD       0:00      3 (Resources)
              1571   p01-arm interact  opr0019 PD       0:00      3 (Priority)
              1568   p01-arm interact  opr0019  R       3:08      8 p01-arm[01-08]
              1566   p03-amd interact  opr0019 CA       2:42      1 p03-amd01
              1567   p03-amd interact  opr0019 CA       2:40      1 p03-amd01
              1570   p03-amd interact  opr0019 CA       1:13      1 p03-amd01
$ scancel --partition p01-arm --state R
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1568   p01-arm interact  opr0019 CA       4:29      8 p01-arm[01-08]
              1569   p01-arm interact  opr0019  R       0:07      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R       0:07      3 p01-arm[04-06]
              1566   p03-amd interact  opr0019 CA       2:42      1 p03-amd01
              1567   p03-amd interact  opr0019 CA       2:40      1 p03-amd01
              1570   p03-amd interact  opr0019 CA       1:13      1 p03-amd01

For more information about the scancel command, its flags, and formatting options, see the manual, either by using the man scancel command or online.

Accessing Detailed Job Information and Advanced Job Options

Note

Please note that most of the scontrol functionality is reserved for system administrator's use only.

scontrol is a program used for Slurm configuration, and as such can be used for accessing detailed job information and advanced functions, such as holding, releasing, requeueing, and suspending jobs.

To view detailed job information, you can use scontrol show job <job_id>, for example:

$ scontrol show job 1571
   UserId=opr0019(5856) GroupId=opr0019(6432) MCS_label=N/A
   Priority=4294901692 Nice=0 Account=easybuild QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
   DerivedExitCode=0:0
   RunTime=00:59:02 TimeLimit=02:00:00 TimeMin=N/A
   SubmitTime=2023-02-13T12:47:01 EligibleTime=2023-02-13T12:47:01
   AccrueTime=2023-02-13T12:47:01
   StartTime=2023-02-13T12:49:51 EndTime=2023-02-13T14:49:51 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2023-02-13T12:49:51 Scheduler=Main
   Partition=p01-arm AllocNode:Sid=login:1122068
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=p01-arm[04-06]
   BatchHost=p01-arm04
   NumNodes=3 NumCPUs=144 NumTasks=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=144,node=3,billing=144
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   JOB_GRES=(null)
     Nodes=p01-arm[04-06] CPU_IDs=0-47 Mem=0 GRES=
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=(null)
   WorkDir=/home/opr0019
   Power=

Note

sbatch, salloc, and srun both support the -H, or --hold flag to submit a job directly in a held state. However, they can only be released via scontrol release <job_id>.

Sometimes you may want to temporarily prevent your job from running. For this reason, you can tell Slurm to hold your job; this will temporarily change its job priority to zero. Once you are sure you want your jobs to execute again, you can tell Slurm to release them back to the queue:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1572   p01-arm interact  opr0019 PD       0:00      8 (Resources)
              1576   p01-arm interact  opr0019 PD       0:00      8 (Priority)
              1569   p01-arm interact  opr0019  R    1:14:11      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R    1:14:11      3 p01-arm[04-06]
$ scontrol hold 1572 1576
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1576   p01-arm interact  opr0019 PD       0:00      8 (JobHeldUser)
              1572   p01-arm interact  opr0019 PD       0:00      8 (JobHeldUser)
              1569   p01-arm interact  opr0019  R    1:14:32      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R    1:14:32      3 p01-arm[04-06]
$ scontrol release 1572 1576
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1572   p01-arm interact  opr0019 PD       0:00      8 (Resources)
              1576   p01-arm interact  opr0019 PD       0:00      8 (Priority)
              1569   p01-arm interact  opr0019  R    1:14:39      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R    1:14:39      3 p01-arm[04-06]

scontrol also offers an interactive mode, where you can run commands in quick succession:

$ scontrol
scontrol: hold 1572
scontrol: release 1572
scontrol: exit
$

For more information about the scontrol command, its flags and formatting options, see manual, either by using the man scontrol command or online.

Capacity Computing

Job arrays offer a mechanism for submitting and managing collections of similar jobs quicky and easily by specifying an additional -a, or --array=<indexes> parameter. Jobs have the same initial options (such as size, time limit, etc.), but also have several additional environment variables set:

Variable Value
SLURM_ARRAY_JOB_ID Equals to the first job ID of the array.
SLURM_ARRAY_TASK_ID Equals to the job array index value.
SLURM_ARRAY_TASK_COUNT Equals to the number of tasks in the job array.
SLURM_ARRAY_TASK_MAX Equals to the highest job array index value.
SLURM_ARRAY_TASK_MIN Equals to the lowest job array index value.
SLURM_ARRAY_TASK_STEP Equals to the step size of task IDs.

Consider following batch script, batchscript.sh:

#!/bin/sh
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -N 1

set | grep -e "^SLURM_JOB_ID" -e "^SLURM_ARRAY"

exit

We can use it to submit 3 jobs in an array

sbatch --array=1-3 batchscript.sh

which would result in 3 output files, by default called slurm-${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}.out

$ ls -l slurm*.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_1.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_2.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_3.out

with the following contents:

$ cat slurm-1632_1.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1633

$ cat slurm-1632_2.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1634

$ cat slurm-1632_3.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1632

For more information about job arrays, see Job Array Support section of the official Slurm documentation.