Slurm Job Submission and Execution¶
Important
Slurm scheduler is currently in use on Complementary System only. Starting July 17., Slurm will be in use on the Barbora cluster.
Slurm Introduction¶
Slurm is a workload manager which facilitates access to cluster resources, and manages scheduling, starting, and executing jobs, and job monitoring. We highly encourage you have a look at the interactive Slurm tutorial.
A man page exists for all Slurm commands, as well as --help command option, which provides a brief summary of options. Slurm documentation and man pages are also available online.
To view all of the Slurm's environment variables, use the command:
set | grep ^SLURM
Quick Overview of Common Commands¶
| Command | Explanation |
|---|---|
| sacct | Display accounting data for all jobs and job steps in the job accounting log or Slurm database. |
| salloc | Obtain a job allocation (a set of nodes), execute a command, and then release the allocation when the command is finished. |
| sattach | Attach to a job step. |
| sbatch | Submit a batch script to Slurm. |
| sbcast | Transmit a file to the nodes allocated to a job. |
| scancel | Used to signal jobs or job steps that are under the control of Slurm. |
| sinfo | View information about nodes and partitions. |
| squeue | View information about jobs located in the scheduling queue. |
| srun | Run parallel jobs. |
Job Submission Options¶
Slurm provides three different commands for job submission: salloc, srun, and sbatch. Each of those serves a slightly different purpose; salloc is used to obtain an allocation, srun runs parallel jobs (and obtains appropriate allocation first if necessary), and serves as Slurm's native way of executing MPI-enabled applications, and sbatch reads a batch script, asks for required allocation, and then executes it. While all of these may be used interchangeably, their specifics make them more useful in different scenarios:
| Command | Preferred use case |
|---|---|
| sbatch | Production jobs |
| salloc | Interactive jobs, debugging |
| srun | Quick interactive jobs, short analysis of output data, running MPI applications |
Batch Mode¶
sbatch command can be used to submit a script for later execution. A batch script usually consists of three parts:
- interpreter used for code execution, denoted by shebang sign (
#!) followed by an absolute path to the interpreter - job options, denoted by a hash sign (
#), a keyword for scheduler to pick upon, and submission options - job instructions; these contain everything an interpreter should do once the job has been initialized, including scheduler-independent environment setup
Your most commonly used interpreter will be bash, or some other shell command language interpreter, such as dash, zsh, etc. You would then define your interpreter on the first line of your batch script like this:
#!/usr/bin/bash
To define Slurm job options within the batch script, use SBATCH keyword followed by your job option. Multiple SBATCH statements can be specified:
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4
Here we asked for 4 tasks in total to be executed on partition p03-amd using OPEN-00-00's project resources.
Job instructions should contain everything you'd like your job to do; that is, every single command the job is supposed to execute:
ml OpenMPI/4.1.4-GCC-11.3.0
srun hostname | uniq -c && \
exit 0 || \
echo "Unable to execute MPI job!"
exit 1
Combined together, the previous examples make up a following script:
#!/usr/bin/bash
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4
ml OpenMPI/4.1.4-GCC-11.3.0
srun hostname | uniq -c && \
exit 0 || \
echo "Unable to execute MPI job!"
exit 1
And by submitting it from the login node of Complementary System
$ sbatch ./batchscript.sh
Submitted batch job 1511
we get an output file with the following contents:
$ cat slurm-1511.out
1 p03-amd01.cs.it4i.cz
3 p03-amd02.cs.it4i.cz
Notice that Slurm spread our job across 2 different nodes; by default, Slurm selects the number of nodes to minimize wait time before job execution. However, sometimes you may want to restrict your job to only a certain minimum or maximum number of nodes (or both). You may also require more time for your calculation to finish than the default allocated time. For an overview of such job options, see table below.
Quick Overview of Common Batch Job Options¶
| Job Option | Specification |
|---|---|
| Job Name | -J, --job-name=<name> |
| Account | -A, --account=<account> |
| Queue / Partition | -p, --partition=<partition_names> |
| Node Count | -N, --nodes=<minnodes>[-maxnodes] |
| Total Task Count | -n, --ntasks=<number> |
| Tasks Per Node* | --ntasks-per-node=<ntasks> |
| CPUs Per Task* | -c, --cpus-per-task=<ncpus> |
| Wall Clock Limit | -t, -t <time> |
| Copy Environment | --export={[ALL,]<environment_variables>|ALL|NONE} |
| Job Dependency | -d, --dependency=<dependency_list> |
| Job Arrays | -a, --array=<indexes> |
-
When the
--ntasks-per-nodeoption is used in conjunction with--ntasks, it is treated as a maximum count of tasks per node instead. For more information, see themanpage. -
Beginning with 22.05,
srunwill not inherit the--cpus-per-taskvalue requested bysallocorsbatch. It must be requested again with the call tosrunor set with theSRUN_CPUS_PER_TASKenvironment variable if desired for the task(s).
Common Job Workflow¶
Note
Production jobs must use the /scratch directory for I/O
The recommended way to run production jobs is to change to the /scratch directory early in the jobscript, copy all inputs to /scratch, execute your calculations, and copy outputs back to the /home directory. While it is possible to prepare and submit your jobs from the /scratch itself, note that it is meant only for temporary data storage during job execution; files older than 90 days may be automatically deleted.
#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4
cd $SLURM_SUBMIT_DIR
SCRDIR="/scratch/project/open-00-00/${USER}/myjob"
mkdir -p "${SCRDIR}"
# change to scratch directory, exit on failure
cd "${SCRDIR}" || exit 1
# copy input files to scratch
cp "${SLURM_SUBMIT_DIR}/input" .
cp "${SLURM_SUBMIT_DIR}/mympiprog.x" .
# load required module(s)
# (Always specify the module's name and version in your script;
# see https://docs.it4i.cz/software/modules/lmod/#loading-modules.)
ml OpenMPI/4.1.4-GCC-11.3.0
# execute the calculation
srun ./mympiprog.x
# copy output file to home
cp output "${SLURM_SUBMIT_DIR}/"
exit
In this example, a directory in /home holds the input file input and the mympiprog.x executable. We create the myjob directory on the /scratch filesystem, copy input and executable files from the /home directory where the sbatch was invoked ($SLURM_SUBMIT_DIR) to /scratch, execute the MPI program mympiprog.x, and copy the output file back to the /home directory. mympiprog.x is run with 4 tasks, allocated according to Slurm configuration.
Selecting the Number of Nodes¶
Some jobs may run significantly slower when the tasks are spread too thin; in that case, it may be beneficial to specify the number of nodes:
#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -n 4
#SBATCH -N 2
srun hostname | sort | uniq -c
exit
The -N, or --nodes option takes two arguments: minnodes, minimum number of nodes to allocate, and maxnodes, maximum number of nodes to allocate. When only one number is specified, it means to allocate exectly the specified number of nodes. Output of the above batchscript may then look something like this:
$ cat slurm-1642.out
2 p01-arm01.cs.it4i.cz
2 p01-arm02.cs.it4i.cz
On the other hand, you may also want to allow Slurm some flexibility in scheduling your job, which may result in earlier job execution. Taking the last jobscript as an example, Slurm may not be able to satisfy a request for 4 tasks on 2 nodes. In that case, the following jobscript
#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -n 4
#SBATCH -N 2-4
srun hostname | sort | uniq -c
exit
may run earlier than jobs asking strictly for 4 tasks on 2 nodes, because the number of required slots may be easier to satisfy by simply spreading them out across more nodes:
$ cat slurm-1643.out
1 p01-arm01.cs.it4i.cz
1 p01-arm02.cs.it4i.cz
1 p01-arm03.cs.it4i.cz
1 p01-arm04.cs.it4i.cz
Interactive Mode¶
Sometimes you may want to run your job interactively, for example for debugging, running your commands one by one from the command line. Slurm provides two diferrent commands for these cases, each with a slightly different purpose: salloc and srun. Both of these take options similar to sbatch.
Resource Allocation¶
salloc requests an allocation, and once obtained, starts a shell on one of the nodes (master node). For example, you can allocate 2 nodes for 10 minutes with the following command:
user@login$ salloc -N 2 -A OPEN-00-00 -p p01-arm -t 10
salloc: Granted job allocation 1539
salloc: Waiting for resource configuration
salloc: Nodes p01-arm[01-02] are ready for job
user@p01-arm01$
Here we can see that Slurm started a shell session on node p01-arm01. To start a parallel execution on allocated nodes, you can run:
$ srun hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
To finish the job, you can either use the exit keyword, or Ctrl+D (^D) control sequence:
$ exit
salloc: Relinquishing job allocation 1539
If your job exceeds either the specified time limit or the maximum possible time limit for the selected partition, you will instead be booted out by the scheduler with the following message:
$ salloc: Job 1544 has exceeded its time limit and its allocation has been revoked.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 1544.interactive ON p01-arm01 CANCELLED AT 2023-02-07T15:18:07 DUE TO TIME LIMIT ***
exit
$
Job Submission¶
Note
srun functions as Slurm's native way of executing MPI-aware applications, with several benefits over mpirun/mpiexec; it knows the exact machine configuration, job allocation, and Slurm's job environment variables. While our OpenMPI modules are compiled with Slurm support, and Intel MPI has its own support built-in, some applications use their own MPI implementation which may require you to specify additional parameters (such as number of tasks) for a job to run in the desired way. For this reason, we recommend you always try to substitute the mpirun/mpiexec commands in your job scripts with srun.
srun is used to submit a job for real time execution, obtains an allocation first if necessary, and also serves as Slurm's native way of executing MPI jobs. For example, to launch a job on 2 nodes from a login node, you can run:
$ srun -N 2 -p p01-arm -A OPEN-00-00 hostname
p01-arm02.cs.it4i.cz
p01-arm01.cs.it4i.cz
Similarly, you can use srun from within the allocation obtained via salloc, both utilizing either whole or part of the whole allocation:
$ salloc -N 4 -p p01-arm -A OPEN-00-00
salloc: Granted job allocation 1551
salloc: Waiting for resource configuration
salloc: Nodes p01-arm[01-04] are ready for job
$ srun hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
p01-arm03.cs.it4i.cz
p01-arm04.cs.it4i.cz
$ srun -N 2 hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
Job Dependency Submission¶
To submit dependent jobs in sequence, use the -d, or --dependency flag. sbatch also provides --parsable flag, which conveniently outputs only the job ID number, and the cluster name if present, and can thus be used to submit a large number of jobs dependent on each other in an automated way. For example, to submit 3 jobs, where the next one always depends on the previous one, we can use:
$ first=$(sbatch --parsable job1.sh)
$ second=$(sbatch --parsable --dependency=afterok:${first} job2.sh)
$ sbatch --parsable --dependency=afterok:${second} job3.sh
1581
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1579 p01-arm job1 opr0019 PD 0:00 8 (Priority)
1580 p01-arm job2 opr0019 PD 0:00 8 (Dependency)
1581 p01-arm job3 opr0019 PD 0:00 8 (Dependency)
$ scontrol show job 1580 | grep JobState
JobState=PENDING Reason=Dependency Dependency=afterok:1579(unfulfilled)
$ scontrol show job 1581 | grep JobState
JobState=PENDING Reason=Dependency Dependency=afterok:1580(unfulfilled)
Multiple job dependencies can be specified by using the colon (:) separator for job IDs, and comma (,) separator for different job exit statuses:
$ sbatch --dependency=afterok:1585:1587,afternotok:1581
Submitted batch job 1596
$ scontrol show job 1596 | grep JobState
JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled),afterok:1587(unfulfilled),afternotok:1581(unfulfilled)
Job dependencies can also be used for submitting long running jobs for which the maximum allowed wall time of partition is insufficient. The only prerequisite is the program has a way of safely saving its progress before it actually reaches end (for example, by creating a STOPCAR file when using VASP). If the job is capable of this, we can submit a job multiple times in a loop, restarting it each time until it finishes:
$ previous=$(sbatch --parsable jobstart.sh)
$ for id in $(seq 2 6)
> do
> submit=$(sbatch --parsable --dependency=afterok:${previous} restart.sh)
> previous=$submit
> done
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1582 p01-arm jobstart opr0019 PD 0:00 8 (Priority)
1583 p01-arm restart opr0019 PD 0:00 8 (Dependency)
1584 p01-arm restart opr0019 PD 0:00 8 (Dependency)
1585 p01-arm restart opr0019 PD 0:00 8 (Dependency)
1586 p01-arm restart opr0019 PD 0:00 8 (Dependency)
1587 p01-arm restart opr0019 PD 0:00 8 (Dependency)
$ for id in $(seq 1582 1587)
> do
> scontrol show job $id | grep JobState
> done
JobState=PENDING Reason=Priority Dependency=(null)
JobState=PENDING Reason=Dependency Dependency=afterok:1582(unfulfilled)
JobState=PENDING Reason=Dependency Dependency=afterok:1583(unfulfilled)
JobState=PENDING Reason=Dependency Dependency=afterok:1584(unfulfilled)
JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled)
JobState=PENDING Reason=Dependency Dependency=afterok:1586(unfulfilled)
Here we have two different submission scripts; one for the first iteration, and one for restarts, since it is often necessary to use a slightly altered input file in order for the job to restart from saved state. As we can see, only one of the jobs will be eligible to run at a time, since the rest of them depend on the correct exit status (zero) of the previous one.
Overview of Job Dependency Options¶
| Job Status Format | Explanation |
|---|---|
| after:job_id[[+time][:jobid[+time]...]] | After the specified jobs start or are cancelled and 'time' in minutes from job start or cancellation happens, this job can begin execution. If no 'time' is given then there is no delay after start or cancellation. |
| afterany:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated. |
| afterburstbuffer:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated and any associated burst buffer stage out operations have completed. |
| aftercorr:job_id[:jobid...] | A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully (ran to completion with an exit code of zero). |
| afternotok:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc.). |
| afterok:job_id[:jobid...] | This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero). |
| singleton | This job can begin execution after any previously launched jobs sharing the same job name and user have terminated. In other words, only one job by that name and owned by that user can be running or suspended at any point in time. In a federation, a singleton dependency must be fulfilled on all clusters unless DependencyParameters=disable_remote_singleton is used in slurm.conf. |
Job Management¶
Apart from job submission and execution, Slurm also provides a number of commands dedicated to various job management purposes, such as viewing current state of the job queue, printing information about existing partitions and their configurations, showing job priorities and how they were arrived at, displaying accounting data and many other.
Job Partition Information¶
To view information about available job partitions, use the sinfo command:
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
p00-arm up 1-00:00:00 1 idle p00-arm01
p01-arm* up 1-00:00:00 8 idle p01-arm[01-08]
p02-intel up 1-00:00:00 2 idle p02-intel[01-02]
p03-amd up 1-00:00:00 2 idle p03-amd[01-02]
p04-edge up 1-00:00:00 1 idle p04-edge01
p05-synt up 1-00:00:00 1 idle p05-synt01
Here we can see output of the sinfo command ran on the Complementary System. By default, it shows basic node and partition configurations.
To view partition summary information, use sinfo -s, or sinfo --summarize:
$ sinfo -s
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
p00-arm up 1-00:00:00 0/1/0/1 p00-arm01
p01-arm* up 1-00:00:00 0/8/0/8 p01-arm[01-08]
p02-intel up 1-00:00:00 0/2/0/2 p02-intel[01-02]
p03-amd up 1-00:00:00 0/2/0/2 p03-amd[01-02]
p04-edge up 1-00:00:00 0/1/0/1 p04-edge01
p05-synt up 1-00:00:00 0/1/0/1 p05-synt01
This lists only a partition state summary with no dedicated column for partition state. Instead, it is summarized in the NODES(A/I/O/T) column, where the A/I/O/T stands for allocated/idle/other/total.
sinfo can also report more granular information, such detailed exact node-oriented information:
$ sinfo -Nel
Mon Feb 13 10:06:54 2023
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
p00-arm01 1 p00-arm idle 64 64:1:1 260988 0 1 aarch64, none
p01-arm01 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
p01-arm02 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
p01-arm03 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
p01-arm04 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
p01-arm05 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
p01-arm06 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
p01-arm07 1 p01-arm* idle 48 48:1:1 32132 0 1 aarch64, none
p01-arm08 1 p01-arm* idle 48 48:1:1 32132 0 1 aarch64, none
p02-intel01 1 p02-intel idle 64 64:1:1 257252 0 1 x86_64,i none
p02-intel02 1 p02-intel idle 64 64:1:1 257252 0 1 x86_64,i none
p03-amd01 1 p03-amd idle 64 64:1:1 257270 0 1 x86_64,a none
p03-amd02 1 p03-amd idle 64 64:1:1 257270 0 1 x86_64,a none
p04-edge01 1 p04-edge idle 16 16:1:1 128433 0 1 x86_64,i none
p05-synt01 1 p05-synt idle 8 8:1:1 128303 0 1 x86_64,a none
For more information about the sinfo command, its flags, and formatting options, see the manual, either by using the man sinfo command or online.
Job Queue Information¶
To view information about queued jobs, use the squeue command:
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1556 p01-arm interact opr0019 R 1:07 8 p01-arm[01-08]
1558 p02-intel interact easybuil CD 0:21 2 p02-intel[01-02]
1557 p03-amd interact opr0019 R 0:57 1 p03-amd01
By default, this shows the job ID, partition, name of the job, job owner's username, job state, how long has the job been already running, number of allocated nodes, and a list of allocated nodes.
To view jobs only belonging to a particular user, you can either use --user=<username>, or --me, which serves as an alias for --user=$USER, to shows only your jobs:
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1559 p01-arm interact opr0019 R 3:37 8 p01-arm[01-08]
1560 p02-intel interact easybuil R 0:05 2 p02-intel[01-02]
1557 p03-amd interact opr0019 R 10:22 1 p03-amd01
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1559 p01-arm interact opr0019 R 4:04 8 p01-arm[01-08]
1557 p03-amd interact opr0019 R 10:49 1 p03-amd01
squeue also allows for printing information about specific jobs using the --jobs flag:
$ squeue --jobs 1557,1560
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1560 p02-intel interact easybuil R 2:09 2 p02-intel[01-02]
1557 p03-amd interact opr0019 R 12:26 1 p03-amd01
For more information about the squeue command, its flags, and formatting options, see the manual, either by using the man squeue command or online.
Job State Codes Overview¶
| Code | Job State | Explanation |
|---|---|---|
| BF | BOOT_FAIL | Job terminated due to launch failure, typically due to a hardware failure (e.g. unable to boot the node or block and the job can not be requeued). |
| CA | CANCELLED | Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated. |
| CD | COMPLETED | Job has terminated all processes on all nodes with an exit code of zero. |
| CF | CONFIGURING | Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting). |
| CG | COMPLETING | Job is in the process of completing. Some processes on some nodes may still be active. |
| DL | DEADLINE | Job terminated on deadline. |
| F | FAILED | Job terminated with non-zero exit code or other failure condition. |
| NF | NODE_FAIL | Job terminated due to failure of one or more allocated nodes. |
| OOM | OUT_OF_MEMORY | Job experienced out of memory error. |
| PD | PENDING | Job is awaiting resource allocation. |
| PR | PREEMPTED | Job terminated due to preemption. |
| R | RUNNING | Job currently has an allocation. |
| RD | RESV_DEL_HOLD | Job is being held after requested reservation was deleted. |
| RF | REQUEUE_FED | Job is being requeued by a federation. |
| RH | REQUEUE_HOLD | Held job is being requeued. |
| RQ | REQUEUED | Completing job is being requeued. |
| RS | RESIZING | Job is about to change size. |
| RV | REVOKED | Sibling was removed from cluster due to other cluster starting the job. |
| SI | SIGNALING | Job is being signaled. |
| SE | SPECIAL_EXIT | The job was requeued in a special state. This state can be set by users, typically in EpilogSlurmctld, if the job has terminated with a particular exit value. |
| SO | STAGE_OUT | Job is staging out files. |
| ST | STOPPED | Job has an allocation, but execution has been stopped with SIGSTOP signal. CPUS have been retained by this job. |
| S | SUSPENDED | Job has an allocation, but execution has been suspended and CPUs have been released for other jobs. |
| TO | TIMEOUT | Job terminated upon reaching its time limit. |
Job Cancelation¶
scancel is used to send signals or cancel jobs. For example, to cancel a specific job, you would run:
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1563 p01-arm interact opr0019 PD 0:00 3 (Resources)
1562 p01-arm interact opr0019 R 0:20 8 p01-arm[01-08]
1561 p03-amd interact opr0019 R 0:25 1 p03-amd01
$ scancel 1562
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1562 p01-arm interact opr0019 CA 0:57 8 p01-arm[01-08]
1563 p01-arm interact opr0019 R 0:03 3 p01-arm[01-03]
1561 p03-amd interact opr0019 R 1:06 1 p03-amd01
To cancel multiple jobs, simply list all of their job IDs:
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1562 p01-arm interact opr0019 CA 0:57 8 p01-arm[01-08]
1564 p01-arm interact opr0019 PD 0:00 8 (Resources)
1563 p01-arm interact opr0019 R 3:31 3 p01-arm[01-03]
1561 p03-amd interact opr0019 R 4:34 1 p03-amd01
1565 p03-amd interact opr0019 R 0:07 1 p03-amd01
$ scancel 1562 1563 1561 1565 1564
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1562 p01-arm interact opr0019 CA 0:57 8 p01-arm[01-08]
1563 p01-arm interact opr0019 CA 4:01 3 p01-arm[01-03]
1564 p01-arm interact opr0019 CA 0:00 8
1561 p03-amd interact opr0019 CA 5:04 1 p03-amd01
1565 p03-amd interact opr0019 CA 0:37 1 p03-amd01
Slurm also allows for canceling only jobs which fulfill certain criteria, for example, the partition to which they have been submitted, or their job state (or a mixture of both):
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1569 p01-arm interact opr0019 PD 0:00 3 (Resources)
1571 p01-arm interact opr0019 PD 0:00 3 (Priority)
1568 p01-arm interact opr0019 R 1:52 8 p01-arm[01-08]
1566 p03-amd interact opr0019 R 1:55 1 p03-amd01
1567 p03-amd interact opr0019 R 1:53 1 p03-amd01
1570 p03-amd interact opr0019 R 0:26 1 p03-amd01
$ scancel --partition p03-amd
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1569 p01-arm interact opr0019 PD 0:00 3 (Resources)
1571 p01-arm interact opr0019 PD 0:00 3 (Priority)
1568 p01-arm interact opr0019 R 3:08 8 p01-arm[01-08]
1566 p03-amd interact opr0019 CA 2:42 1 p03-amd01
1567 p03-amd interact opr0019 CA 2:40 1 p03-amd01
1570 p03-amd interact opr0019 CA 1:13 1 p03-amd01
$ scancel --partition p01-arm --state R
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1568 p01-arm interact opr0019 CA 4:29 8 p01-arm[01-08]
1569 p01-arm interact opr0019 R 0:07 3 p01-arm[01-03]
1571 p01-arm interact opr0019 R 0:07 3 p01-arm[04-06]
1566 p03-amd interact opr0019 CA 2:42 1 p03-amd01
1567 p03-amd interact opr0019 CA 2:40 1 p03-amd01
1570 p03-amd interact opr0019 CA 1:13 1 p03-amd01
For more information about the scancel command, its flags, and formatting options, see the manual, either by using the man scancel command or online.
Accessing Detailed Job Information and Advanced Job Options¶
Note
Please note that most of the scontrol functionality is reserved for system administrator's use only.
scontrol is a program used for Slurm configuration, and as such can be used for accessing detailed job information and advanced functions, such as holding, releasing, requeueing, and suspending jobs.
To view detailed job information, you can use scontrol show job <job_id>, for example:
$ scontrol show job 1571
UserId=opr0019(5856) GroupId=opr0019(6432) MCS_label=N/A
Priority=4294901692 Nice=0 Account=easybuild QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
DerivedExitCode=0:0
RunTime=00:59:02 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2023-02-13T12:47:01 EligibleTime=2023-02-13T12:47:01
AccrueTime=2023-02-13T12:47:01
StartTime=2023-02-13T12:49:51 EndTime=2023-02-13T14:49:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2023-02-13T12:49:51 Scheduler=Main
Partition=p01-arm AllocNode:Sid=login:1122068
ReqNodeList=(null) ExcNodeList=(null)
NodeList=p01-arm[04-06]
BatchHost=p01-arm04
NumNodes=3 NumCPUs=144 NumTasks=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=144,node=3,billing=144
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
JOB_GRES=(null)
Nodes=p01-arm[04-06] CPU_IDs=0-47 Mem=0 GRES=
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=(null)
WorkDir=/home/opr0019
Power=
Note
sbatch, salloc, and srun both support the -H, or --hold flag to submit a job directly in a held state. However, they can only be released via scontrol release <job_id>.
Sometimes you may want to temporarily prevent your job from running. For this reason, you can tell Slurm to hold your job; this will temporarily change its job priority to zero. Once you are sure you want your jobs to execute again, you can tell Slurm to release them back to the queue:
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1572 p01-arm interact opr0019 PD 0:00 8 (Resources)
1576 p01-arm interact opr0019 PD 0:00 8 (Priority)
1569 p01-arm interact opr0019 R 1:14:11 3 p01-arm[01-03]
1571 p01-arm interact opr0019 R 1:14:11 3 p01-arm[04-06]
$ scontrol hold 1572 1576
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1576 p01-arm interact opr0019 PD 0:00 8 (JobHeldUser)
1572 p01-arm interact opr0019 PD 0:00 8 (JobHeldUser)
1569 p01-arm interact opr0019 R 1:14:32 3 p01-arm[01-03]
1571 p01-arm interact opr0019 R 1:14:32 3 p01-arm[04-06]
$ scontrol release 1572 1576
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1572 p01-arm interact opr0019 PD 0:00 8 (Resources)
1576 p01-arm interact opr0019 PD 0:00 8 (Priority)
1569 p01-arm interact opr0019 R 1:14:39 3 p01-arm[01-03]
1571 p01-arm interact opr0019 R 1:14:39 3 p01-arm[04-06]
scontrol also offers an interactive mode, where you can run commands in quick succession:
$ scontrol
scontrol: hold 1572
scontrol: release 1572
scontrol: exit
$
For more information about the scontrol command, its flags and formatting options, see manual, either by using the man scontrol command or online.
Capacity Computing¶
Job arrays offer a mechanism for submitting and managing collections of similar jobs quicky and easily by specifying an additional -a, or --array=<indexes> parameter. Jobs have the same initial options (such as size, time limit, etc.), but also have several additional environment variables set:
| Variable | Value |
|---|---|
SLURM_ARRAY_JOB_ID |
Equals to the first job ID of the array. |
SLURM_ARRAY_TASK_ID |
Equals to the job array index value. |
SLURM_ARRAY_TASK_COUNT |
Equals to the number of tasks in the job array. |
SLURM_ARRAY_TASK_MAX |
Equals to the highest job array index value. |
SLURM_ARRAY_TASK_MIN |
Equals to the lowest job array index value. |
SLURM_ARRAY_TASK_STEP |
Equals to the step size of task IDs. |
Consider following batch script, batchscript.sh:
#!/bin/sh
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -N 1
set | grep -e "^SLURM_JOB_ID" -e "^SLURM_ARRAY"
exit
We can use it to submit 3 jobs in an array
sbatch --array=1-3 batchscript.sh
which would result in 3 output files, by default called slurm-${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}.out
$ ls -l slurm*.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_1.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_2.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_3.out
with the following contents:
$ cat slurm-1632_1.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1633
$ cat slurm-1632_2.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1634
$ cat slurm-1632_3.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1632
For more information about job arrays, see Job Array Support section of the official Slurm documentation.