Resources Allocation Policy¶
Job Queue Policies¶
The resources are allocated to the job in a fair-share fashion, subject to constraints set by the queue and the resources available to the Project. The Fair-share system of Anselm ensures that individual users may consume approximately equal amounts of resources per week. Detailed information can be found in the Job scheduling section. The resources are accessible via several queues for queueing the jobs. The queues provide prioritized and exclusive access to the computational resources. The following table provides the queue partitioning overview:
Check the queue status at https://extranet.it4i.cz/rsweb/anselm/
|queue||active project||project resources||nodes||min ncpus||priority||authorization||walltime|
|qexp||no||none required||209 nodes||1||150||no||1 h|
|qprod||yes||> 0||180 nodes w/o accelerator||16||0||no||24/48 h|
|qlong||yes||> 0||180 nodes w/o accelerator||16||0||no||72/144 h|
|qnvidia, qmic||yes||> 0||23 nvidia nodes, 4 mic nodes||16||200||yes||24/48 h|
|qfat||yes||> 0||2 fat nodes||16||200||yes||24/144 h|
|qfree||yes||< 120% of allocation||180 w/o accelerator||16||-1024||no||12 h|
The qfree queue is not free of charge. Normal accounting applies. However, it allows for utilization of free resources, once a project has exhausted all its allocated computational resources. This does not apply to Director's Discretion projects (DD projects) by default. Usage of qfree after exhaustion of DD projects' computational resources is allowed after request for this queue.
The qexp queue is equipped with nodes which do not have exactly the same CPU clock speed. Should you need the nodes to have exactly the same CPU speed, you have to select the proper nodes during the PSB job submission.
- qexp, the Express queue: This queue is dedicated to testing and running very small jobs. It is not required to specify a project to enter the qexp. There are always 2 nodes reserved for this queue (w/o accelerators), a maximum 8 nodes are available via the qexp for a particular user, from a pool of nodes containing Nvidia accelerated nodes (cn181-203), MIC accelerated nodes (cn204-207) and Fat nodes with 512GB of RAM (cn208-209). This enables us to test and tune accelerated code and code with higher RAM requirements. The nodes may be allocated on a per core basis. No special authorization is required to use qexp. The maximum runtime in qexp is 1 hour.
- qprod, the Production queue: This queue is intended for normal production runs. It is required that an active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue, except the reserved ones. 178 nodes without accelerators are included. Full nodes, 16 cores per node, are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qprod is 48 hours.
- qlong, the Long queue: This queue is intended for long production runs. It is required that an active project with nonzero remaining resources is specified to enter the qlong. Only 60 nodes without acceleration may be accessed via the qlong queue. Full nodes, 16 cores per node, are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qlong is 144 hours (three times that of the standard qprod time - 3 x 48 h).
- qnvidia, qmic, qfat, the Dedicated queues: The queue qnvidia is dedicated to accessing the Nvidia accelerated nodes, the qmic to accessing MIC nodes and qfat the Fat nodes. It is required that an active project with nonzero remaining resources is specified to enter these queues. 23 nvidia, 4 mic, and 2 fat nodes are included. Full nodes, 16 cores per node, are allocated. The queues run with very high priority, the jobs will be scheduled before the jobs coming from the qexp queue. An PI needs to explicitly ask support for authorization to enter the dedicated queues for all users associated with her/his project.
- qfree, The Free resource queue: The queue qfree is intended for utilization of free resources, after a project has exhausted all of its allocated computational resources (Does not apply to DD projects by default; DD projects have to request persmission to use qfree after exhaustion of computational resources). It is required that active project is specified to enter the queue. Consumed resources will be accounted to the Project. Access to the qfree queue is automatically removed if consumed resources exceed 120% of the resources allocated to the Project. Only 180 nodes without accelerators may be accessed from this queue. Full nodes, 16 cores per node, are allocated. The queue runs with very low priority and no special authorization is required to use it. The maximum runtime in qfree is 12 hours.
The job wall clock time defaults to half the maximum time, see the table above. Longer wall time limits can be set manually, see examples.
Jobs that exceed the reserved wall clock time (Req'd Time) get killed automatically. The wall clock time limit can be changed for queuing jobs (state Q) using the qalter command, however it cannot be changed for a running job (state R).
Anselm users may check the current queue configuration here.
Check the status of jobs, queues and compute nodes here.
Display the queue status on Anselm:
$ qstat -q
The PBS allocation overview may be obtained also using the rspbs command:
$ rspbs Usage: rspbs [options] Options: --version show program's version number and exit -h, --help show this help message and exit --get-node-ncpu-chart Print chart of allocated ncpus per node --summary Print summary --get-server-details Print server --get-queues Print queues --get-queues-details Print queues details --get-reservations Print reservations --get-reservations-details Print reservations details --get-nodes Print nodes of PBS complex --get-nodeset Print nodeset of PBS complex --get-nodes-details Print nodes details --get-jobs Print jobs --get-jobs-details Print jobs details --get-jobs-check-params Print jobid, job state, session_id, user, nodes --get-users Print users of jobs --get-allocated-nodes Print allocated nodes of jobs --get-allocated-nodeset Print allocated nodeset of jobs --get-node-users Print node users --get-node-jobs Print node jobs --get-node-ncpus Print number of ncpus per node --get-node-allocated-ncpus Print number of allocated ncpus per node --get-node-qlist Print node qlist --get-node-ibswitch Print node ibswitch --get-user-nodes Print user nodes --get-user-nodeset Print user nodeset --get-user-jobs Print user jobs --get-user-jobc Print number of jobs per user --get-user-nodec Print number of allocated nodes per user --get-user-ncpus Print number of allocated ncpus per user --get-qlist-nodes Print qlist nodes --get-qlist-nodeset Print qlist nodeset --get-ibswitch-nodes Print ibswitch nodes --get-ibswitch-nodeset Print ibswitch nodeset --state=STATE Only for given job state --jobid=JOBID Only for given job ID --user=USER Only for given user --node=NODE Only for given node --nodestate=NODESTATE Only for given node state (affects only --get-node* --get-qlist-* --get-ibswitch-* actions) --incl-finished Include finished jobs
Resource Accounting Policy¶
Wall-Clock Core-Hours WCH¶
The wall-clock core-hours (WCH) are the basic metric of computer utilization time. 1 wall-clock core-hour is defined as 1 processor core allocated for 1 hour of wall-clock time. Allocating a full node (16 cores Anselm, 24 cores Salomon) for 1 hour amounts to 16 wall-clock core-hours (Anselm) or 24 wall-clock core-hours (Salomon).
Normalized Core-Hours NCH¶
The resources subject to accounting are the normalized core-hours (NCH). The normalized core-hours are obtained from WCH by applying a normalization factor:
All jobs are accounted in normalized core-hours, using factor F valid at the time of the execution:
|Salomon||1.00||2017-09-11 to 2019-03-01|
|Anselm||0.65||2017-09-11 to 2019-03-01|
The accounting runs whenever the computational cores are allocated via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation.
The allocations are requested/granted in normalized core-hours NCH.
Whenever the term core-hour is used in this documentation, we mean the normalized core-hour, NCH.
The normalized core-hours were introduced to treat systems of different age on equal footing. Normalized core-hour is an accounting tool to discount the legacy systems. The past (before 2017-09-11) F factors are all 1.0. In future, the factors F will be updated, as new systems are installed. Factors F are expected to only decrease in time.
See examples in the Job submission and execution section.
Check how many core-hours have been consumed. The command it4ifree is available on cluster login nodes.
$ it4ifree Projects I am participating in ============================== PID Days left Total Used WCHs Used NCHs WCHs by me NCHs by me Free ---------- ----------- ------- ----------- ----------- ------------ ------------ ------- OPEN-XX-XX 323 0 5169947 5169947 50001 50001 1292555 Projects I am Primarily Investigating ===================================== PID Login Used WCHs Used NCHs ---------- ---------- ----------- ----------- OPEN-XX-XX user1 376670 376670 user2 4793277 4793277 Legend ====== WCH = Wall-clock Core Hour NCH = Normalized Core Hour
The it4ifree command is a part of
it4i.portal.clients package, located here.