Skip to content

Allocation of vnodes on qgpu


The qgpu queue on Karolina takes advantage of the division of nodes into vnodes. Accelerated node equipped with two 64-core processors and eight GPU cards is treated as eight vnodes, each containing 16 CPU cores and 1 GPU card. Vnodes can be allocated to jobs individually –⁠ through precise definition of resource list at job submission, you may allocate varying number of resources/GPU cards according to your needs.

Vnodes and Security

Division of nodes into vnodes was implemented to be as secure as possible, but it is still a "multi-user mode", which means that if two users allocate a portion of the same node, they can see each other's running processes. If this solution is inconvenient for you, consider allocating a whole node.

Selection Statement and Chunks

Requested resources are specified using a selection statement:

-l select=[<N>:]<chunk>[+[<N>:]<chunk> ...]

N specifies the number of chunks; if not specified then N = 1.
chunk declares the value of each resource in a set of resources which are to be allocated as a unit to a job.

  • chunk is seen by the MPI as one node.
  • Multiple chunks are then seen as multiple nodes.
  • Maximum chunk size is equal to the size of a full physical node (8 GPU cards, 128 cores)

Default chunk for the qgpu queue is configured to contain 1 GPU card and 16 CPU cores, i.e. ncpus=16:ngpus=1.

  • ncpus specifies number of CPU cores
  • ngpus specifies number of GPU cards

Allocating Single GPU

Single GPU can be allocated in an interactive session using

qsub -q qgpu -A OPEN-00-00 -l select=1 -I

or simply

qsub -q qgpu -A OPEN-00-00 -I

In this case, the ngpus parameter is optional, since it defaults to 1. You can verify your allocation either in the PBS using the qstat command, or by checking the number of allocated GPU cards in the CUDA_VISIBLE_DEVICES variable:

$ qstat -F json -f $PBS_JOBID | grep exec_vnode


The output shows that you have been allocated vnode acn53[0].

Allocating Single Accelerated Node

Security tip

Allocating a whole node prevents other users from seeing your running processes.

Single accelerated node can be allocated in an interactive session using

qsub -q qgpu -A OPEN-00-00 -l select=8 -I

Setting select=8 automatically allocates a whole accelerated node and sets mpiproc. So for N full nodes, set select to N x 8. However, note that it may take some time before your jobs are executed if the required amount of full nodes isn't available.

Allocating Multiple GPUs

Security risk

If two users allocate a portion of the same node, they can see each other's running processes. When required for security reasons, consider allocating a whole node.

Again, the following examples use only the selection statement, so no additional setting is required.

qsub -q qgpu -A OPEN-00-00 -l select=2 -I

In this example two chunks will be allocated on the same node, if possible.

qsub -q qgpu -A OPEN-00-00 -l select=16 -I

This example allocates two whole accelerated nodes.

Multiple vnodes within the same chunk can be allocated using the ngpus parameter. For example, to allocate 2 vnodes in an interactive mode, run

qsub -q qgpu -A OPEN-00-00 -l select=1:ngpus=2:mpiprocs=2 -I

Remember to set the number of mpiprocs equal to that of ngpus to spawn an according number of MPI processes.

To verify the correctness:

$ qstat -F json -f $PBS_JOBID | grep exec_vnode

$ echo $CUDA_VISIBLE_DEVICES | tr ',' '\n'

The number of chunks to allocate is specified in the select parameter. For example, to allocate 2 chunks, each with 4 GPUs, run

qsub -q qgpu -A OPEN-00-00 -l select=2:ngpus=4:mpiprocs=4 -I

To verify the correctness:

$ cat > <<EOF

$ chmod +x
$ ml OpenMPI/4.1.4-GCC-11.3.0
$ mpirun ./ | tr ',' '\n' | sort | uniq