Skip to content

Capacity Computing


In many cases, it is useful to submit a huge (>100) number of computational jobs into the PBS queue system. A huge number of (small) jobs is one of the most effective ways to execute embarrassingly parallel calculations, achieving the best runtime, throughput, and computer utilization.

However, executing a huge number of jobs via the PBS queue may strain the system. This strain may result in slow response to commands, inefficient scheduling, and overall degradation of performance and user experience for all users. For this reason, the number of jobs is limited to 100 jobs per user, 4,000 jobs and subjobs per user, 1,500 subjobs per job array.


Follow one of the procedures below, in case you wish to schedule more than 100 jobs at a time.

  • Use Job arrays when running a huge number of multithread (bound to one node only) or multinode (multithread across several nodes) jobs.
  • Use HyperQueue when running a huge number of multithread jobs. HyperQueue can help overcome the limits of job arrays.


  1. A user is allowed to submit at most 100 jobs. Each job may be a job array.
  2. The array size is at most 1,000 subjobs.

Job Arrays


A huge number of jobs may easily be submitted and managed as a job array.

A job array is a compact representation of many jobs called subjobs. Subjobs share the same job script, and have the same values for all attributes and resources, with the following exceptions:

  • each subjob has a unique index, $PBS_ARRAY_INDEX
  • job Identifiers of subjobs only differ by their indices
  • the state of subjobs can differ (R, Q, etc.)

All subjobs within a job array have the same scheduling priority and schedule as independent jobs. An entire job array is submitted through a single qsub command and may be managed by qdel, qalter, qhold, qrls, and qsig commands as a single job.

Shared Jobscript

All subjobs in a job array use the very same single jobscript. Each subjob runs its own instance of the jobscript. The instances execute different work controlled by the $PBS_ARRAY_INDEX variable.


Assume we have 900 input files with the name of each beginning with "file" (e.g. file001, ..., file900). Assume we would like to use each of these input files with myprog.x program executable, each as a separate job.

First, we create a tasklist file (or subjobs list), listing all tasks (subjobs) - all input files in our example:

$ find . -name 'file*' > tasklist

Then we create a jobscript:

#PBS -A OPEN-00-00
#PBS -q qprod
#PBS -l select=1,walltime=02:00:00

# change to scratch directory
mkdir -p $SCRDIR
cd $SCRDIR || exit

# get individual tasks from tasklist with index from PBS JOB ARRAY
TASK=$(sed -n "${PBS_ARRAY_INDEX}p" $PBS_O_WORKDIR/tasklist)

# copy input file and executable to scratch
cp $PBS_O_WORKDIR/myprog.x .

# execute the calculation
./myprog.x < input > output

# copy output file to submit directory
cp output $PBS_O_WORKDIR/$TASK.out

In this example, the submit directory contains the 900 input files, the myprog.x executable, and the jobscript file. As an input for each run, we take the filename of the input file from the created tasklist file. We copy the input file to the local scratch memory /lscratch/$PBS_JOBID, execute the myprog.x and copy the output file back to the submit directory, under the $TASK.out name. The myprog.x executable runs on one node only and must use threads to run in parallel. Be aware, that if the myprog.x is not multithreaded, then all the jobs are run as single-thread programs in a sequential manner. Due to the allocation of the whole node, the accounted time is equal to the usage of the whole node, while using only 1/16 of the node.

If running a huge number of parallel multicore (in means of multinode multithread, e.g. MPI enabled) jobs is needed, then a job array approach should be used. The main difference, as compared to the previous examples using one node, is that the local scratch memory should not be used (as it is not shared between nodes) and MPI or other techniques for parallel multinode processing has to be used properly.

Submit the Job Array

To submit the job array, use the qsub -J command. The 900 jobs of the example above may be submitted like this:

$ qsub -N JOBNAME -J 1-900 jobscript

In this example, we submit a job array of 900 subjobs. Each subjob will run on one full node and is assumed to take less than 2 hours (note the #PBS directives in the beginning of the jobscript file, do not forget to set your valid PROJECT_ID and desired queue).

Sometimes for testing purposes, you may need to submit a one-element only array. This is not allowed by PBSPro, but there is a workaround:

$ qsub -N JOBNAME -J 9-10:2 jobscript

This will only choose the lower index (9 in this example) for submitting/running your job.

Manage the Job Array

Check status of the job array using the qstat command.

$ qstat -a 12345[].dm2

                                                            Req'd Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time S Time
--------------- -------- --  |---|---| ------ --- --- ------ ----- - -----
12345[].dm2     user2    qprod    xx          13516   1 16    --  00:50 B 00:02

When the status is B, it means that some subjobs are already running. Check the status of the first 100 subjobs using the qstat command.

$ qstat -a 12345[1-100].dm2

                                                            Req'd Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time S Time
--------------- -------- --  |---|---| ------ --- --- ------ ----- - -----
12345[1].dm2    user2    qprod    xx          13516   1 16    --  00:50 R 00:02
12345[2].dm2    user2    qprod    xx          13516   1 16    --  00:50 R 00:02
12345[3].dm2    user2    qprod    xx          13516   1 16    --  00:50 R 00:01
12345[4].dm2    user2    qprod    xx          13516   1 16    --  00:50 Q   --
     .             .        .      .             .    .   .     .    .   .    .
     ,             .        .      .             .    .   .     .    .   .    .
12345[100].dm2 user2    qprod    xx          13516   1 16    --  00:50 Q   --

Delete the entire job array. Running subjobs will be killed, queueing subjobs will be deleted.

$ qdel 12345[].dm2

Deleting large job arrays may take a while. Display status information for all user's jobs, job arrays, and subjobs.

$ qstat -u $USER -t

Display status information for all user's subjobs.

$ qstat -u $USER -tJ

For more information on job arrays, see the PBSPro Users guide.


Download the examples in, illustrating the above listed ways to run a huge number of jobs. We recommend trying out the examples before using this for running production jobs.

Unzip the archive in an empty directory on cluster and follow the instructions in the README file-

$ unzip
$ cat README


HyperQueue lets you build a computation plan consisting of a large amount of tasks and then execute it transparently over a system like SLURM/PBS. It dynamically groups tasks into PBS jobs and distributes them to fully utilize allocated nodes. You thus do not have to manually aggregate your tasks into PBS jobs.

Find more about HyperQueue in its documentation.


  • Transparent task execution on top of a Slurm/PBS cluster

  • Automatic task distribution amongst jobs, nodes, and cores

  • Automatic submission of PBS/Slurm jobs

  • Dynamic load balancing across jobs

  • Work-stealing scheduler

  • NUMA-aware, core planning, task priorities, task arrays
  • Nodes and tasks may be added/removed on the fly

  • Scalable

  • Low overhead per task (~100μs)

  • Handles hundreds of nodes and millions of tasks
  • Output streaming avoids creating many files on network filesystems

  • Easy deployment

  • Single binary, no installation, depends only on libc

  • No elevated privileges required


  • On Barbora and Karolina, you can simply load the HyperQueue module:

    $ ml HyperQueue
  • If you want to install/compile HyperQueue manually, follow the steps on the official webpage.


Starting the Server

To use HyperQueue, you first have to start the HyperQueue server. It is a long-lived process that is supposed to be running on a login node. You can start it with the following command:

$ hq server start

Submitting Computation

Once the HyperQueue server is running, you can submit jobs into it. Here are a few examples of job submissions. You can find more information in the documentation.

  • Submit a simple job (command echo 'Hello world' in this case)

    $ hq submit echo 'Hello world'
  • Submit a job with 10000 tasks

    $ hq submit --array 1-10000

Once you start some jobs, you can observe their status using the following commands:

# Display status of a single job
$ hq job <job-id>

# Display status of all jobs
$ hq jobs


Before the jobs can start executing, you have to provide HyperQueue with some computational resources.

Providing Computational Resources

Before HyperQueue can execute your jobs, it needs to have access to some computational resources. You can provide these by starting HyperQueue workers which connect to the server and execute your jobs. The workers should run on computing nodes, therefore they should be started inside PBS jobs.

There are two ways of providing computational resources.

  • Allocate PBS jobs automatically

    HyperQueue can automatically submit PBS jobs with workers on your behalf. This system is called automatic allocation. After the server is started, you can add a new automatic allocation queue using the hq alloc add command:

    $ hq alloc add pbs -- -qqprod -AAccount1

    After you run this command, HQ will automatically start submitting PBS jobs on your behalf once some HQ jobs are submitted.

  • Manually start PBS jobs with HQ workers

    With the following command, you can submit a PBS job that will start a single HQ worker which will connect to a running HQ server.

    $ qsub <qsub-params> -- /bin/bash -l -c "$(which hq) worker start"


For debugging purposes, you can also start the worker e.g. on a login node, simply by running $ hq worker start. Do not use such worker for any long-running computations though.


Here you can see the architecture of HyperQueue. The user submits jobs into the server which schedules them onto a set of workers running on compute nodes.