PBS job queues resource management
Home > CentOS > CentOS 7.x > CentOS 7.x Rocks cluster 7.0 > PBS job queues resource management
In order to restrict amount of resources available to users we need to do resource management on queues and assign users / nodes or groups to queues.
Resource management for the overall Queue
- Set Maximum CPU for the queue
- qmgr -c "set queue testq1 resources_max.ncpus=20"
- Set Maximum Memory for queue
- qmgr -c "set queue testq1 resources_max.mem=5gb"
- Existing limits can also be removed via:
- qmgr -c "unset queue testq1 resources_max.ncpus"
- qmgr -c "unset queue testq1 resources_max.mem"
Adding nodes to queues
This can be done if we want to restrict nodes to specific queue. Ideally this is not required if queues are configured at user level or group level as explained later:
- Add node to a specific queue
- qmgr -c "set node node-name queue=queue-name"
- Remove nodes from a specific queue
- qmgr -c "unset node node-name queue"
User resource management for the Queue
- Enable user restrictions for the queue
- qmgr -c "set queue testq1 acl_user_enable = True"
- Add users to the queue
- qmgr -c 'set queue testq1 acl_users = "user1, user2"'
- Add more users to the queue
- qmgr -c 'set queue testq1 acl_users += "user3, user7"'
- Remove single user from the queue
- qmgr -c "set queue testq1 acl_users -= user3"
- Remove all users from the queue
- qmgr -c "unset queue testq1 acl_users"
- Disable user restrictions for the queue
- qmgr -c "unset queue testq1 acl_user_enable"
Set / Unset Maximum CPU per user
- Set Maximum CPU for single user
- qmgr -c "set queue testq1 max_run_res.ncpus = [u:user3=10]"
- Set Maximum CPU for multiple users
- qmgr -c 'set queue testq1 max_run_res.ncpus = "[u:user2=10],[u:user3=12]"'
- If Maximum CPU set for users and need to add more users to the same rule
- qmgr -c "set queue testq1 max_run_res.ncpus += [u:user9=10]"
- Remove Max CPU for the user
- qmgr -c "set queue testq1 max_run_res.ncpus -= [u:user9=10]"
Set / Unset Maximum Memory per user
- Set Maximum Memory for single user
- qmgr -c "set queue testq1 max_run_res.mem = [u:user1=4g]"
- Set Maximum Memory for multiple users
- qmgr -c 'set queue testq1 max_run_res.mem = "[u:user1=4g],[u:user2=3g]"'
- If Maximum Memory set for users and need to add more users to the same rule
- qmgr -c "set queue testq1 max_run_res.mem += [u:user9=4g]"
- Remove Max memory for the user
- qmgr -c "set queue testq1 max_run_res.mem -= [u:user9=4g]"
User level resource restriction example
Create test queue using:
qmgr -c "create queue testq1 queue_type=execution"
qmgr -c "set queue testq1 started=true"
qmgr -c "set queue testq1 enabled=true"
qmgr -c "set queue testq1 resources_default.nodes=1"
qmgr -c "set queue testq1 resources_default.walltime=3600"
qmgr -c "set queue testq1 resources_max.ncpus=3"
qmgr -c "set queue testq1 resources_max.mem=2gb"
qmgr -c "set queue testq1 acl_user_enable = True"
qmgr -c 'set queue testq1 acl_users = "user1"'
Then if we run following as user1, then we get error of insufficient resources in queue.
#!/bin/bash
#PBS -N job
#PBS -q testq1
#PBS -V
#PBS -l nodes=2:ppn=2
/opt/openmpi/bin/mpirun -machinefile $PBS_NODEFILE -np 4 /usr/bin/stress --cpu 1 --vm-bytes 10G --timeout 60s
Group resource management for the queue
On the same queue we should not apply both user level and group level limit. Only one of the two should be used.
- Enable Group restriction for the queue
- qmgr -c "set queue testq1 acl_group_enable = True"
- Add group to the queue
- qmgr -c "set queue testq1 acl_groups = group1"
- Add more groups to the queue
- qmgr -c "set queue testq1 acl_groups += group2"
- Remove group from the queue
- qmgr -c "set queue testq1 acl_groups -= group2"
- Disable Group restriction for the queue
- qmgr -c "unset queue testq1 acl_group_enable"
Set / Unset Maximum CPU at group level
- Set Maximum CPU for single group
- qmgr -c "set queue testq1 max_run_res.ncpus = [g:group1=10]"
- Set Maximum CPU for multiple groups
- qmgr -c 'set queue testq1 max_run_res.ncpus = "[g:group1=10],[g:group2=12]"'
- Add more groups to Maximum CPU rule
- qmgr -c "set queue testq1 max_run_res.ncpus += [g:group3=5]"
- Remove Max CPU for the group
- qmgr -c "set queue testq1 max_run_res.ncpus -= [g:group3=5]"
Set / Unset Maximum Memory at group level
- Set Maximum Memory for single group
- qmgr -c "set queue testq1 max_run_res.mem = [g:group1=4g]"
- Set Maximum Memory for multiple group
- qmgr -c 'set queue testq1 max_run_res.mem = "[g:group1=4g],[g:group2=3g]"'
- Add more groups to Maximum Memory rule
- qmgr -c "set queue testq1 max_run_res.mem += [g:group9=4g]"
- Remove Max memory for the group
- qmgr -c "set queue testq1 max_run_res.mem -= [g:group9=4g]"
Setting maximum execution time (walltime) limits for user or queue
We can configure walltime limit on queue and user level as follows:.
Below is the walltime limit configuration for the queue
qmgr -c "set queue testq1 resources_max.walltime = 00:01:00"
Below is the walltime limit configuration for the user
qmgr -c "set queue testq1 max_run_res.walltime = [u:user1=60]"
For some reason same walltime limit is not working at group level:
qmgr -c "set queue testq2 max_run_res.walltime = [g:group2=3600]"
Home > CentOS > CentOS 7.x > CentOS 7.x Rocks cluster 7.0 > PBS job queues resource management