PBS job queues resource management

From Notes_Wiki
Revision as of 09:16, 8 July 2022 by Saurabh (talk | contribs) (Created page with "Home > CentOS > CentOS 7.x > CentOS 7.x Rocks cluster 7.0 > PBS job queues resource management In order to restrict amount of resources available to users we need to do resource management on queues and assign users / nodes or groups to queues. =Resource management for the overall Queue= * Set Maximum CPU for the queue *:<pre> *:: qmgr -c "set queue testq1 resources_max.ncpus=20" *:</pre> * Set Maximum Memory for queue *:<pre> *:: qmgr...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Home > CentOS > CentOS 7.x > CentOS 7.x Rocks cluster 7.0 > PBS job queues resource management

In order to restrict amount of resources available to users we need to do resource management on queues and assign users / nodes or groups to queues.

Resource management for the overall Queue

  • Set Maximum CPU for the queue
    qmgr -c "set queue testq1 resources_max.ncpus=20"
  • Set Maximum Memory for queue
    qmgr -c "set queue testq1 resources_max.mem=5gb"
  • Existing limits can also be removed via:
    qmgr -c "unset queue testq1 resources_max.ncpus"
    qmgr -c "unset queue testq1 resources_max.mem"


Adding nodes to queues

This can be done if we want to restrict nodes to specific queue. Ideally this is not required if queues are configured at user level or group level as explained later:

  • Add node to a specific queue
    qmgr -c "set node node-name queue=queue-name"
  • Remove nodes from a specific queue
    qmgr -c "unset node node-name queue"


User management for the Queue

  • Enable user restrictions for the queue
    qmgr -c "set queue testq1 acl_user_enable = True"
  • Add users to the queue
    qmgr -c 'set queue testq1 acl_users = "user1, user2"'
  • Add more users to the queue
    qmgr -c 'set queue testq1 acl_users += "user3, user7"'
  • Remove single user from the queue
    qmgr -c "set queue testq1 acl_users -= user3"
  • Remove all users from the queue
    qmgr -c "unset queue testq1 acl_users"
  • Disable user restrictions for the queue
    qmgr -c "unset queue testq1 acl_user_enable"


Set / Unset Maximum CPU per user

  • Set Maximum CPU for single user
    qmgr -c "set queue testq1 max_run_res.ncpus = [u:user3=10]"
  • Set Maximum CPU for multiple users
    qmgr -c 'set queue testq1 max_run_res.ncpus = "[u:user2=10],[u:user3=12]"'
  • If Maximum CPU set for users and need to add more users to the same rule
    qmgr -c "set queue testq1 max_run_res.ncpus += [u:user9=10]"
  • Remove Max CPU for the user
    qmgr -c "set queue testq1 max_run_res.ncpus -= [u:user9=10]"


Set / Unset Maximum Memory per user

  • Set Maximum Memory for single user
    qmgr -c "set queue testq1 max_run_res.mem = [u:user1=4g]"
  • Set Maximum Memory for multiple users
    qmgr -c 'set queue testq1 max_run_res.mem = "[u:user1=4g],[u:user2=3g]"'
  • If Maximum Memory set for users and need to add more users to the same rule
    qmgr -c "set queue testq1 max_run_res.mem += [u:user9=4g]"
  • Remove Max memory for the user
    qmgr -c "set queue testq1 max_run_res.mem -= [u:user9=4g]"


User level resource restriction example

Create test queue using:

qmgr -c "create queue testq1 queue_type=execution"
qmgr -c "set queue testq1 started=true"
qmgr -c "set queue testq1 enabled=true"
qmgr -c "set queue testq1 resources_default.nodes=1"
qmgr -c "set queue testq1 resources_default.walltime=3600"

qmgr -c "set queue testq1 resources_max.ncpus=3"
qmgr -c "set queue testq1 resources_max.mem=2gb"

qmgr -c "set queue testq1 acl_user_enable = True"
qmgr -c 'set queue testq1 acl_users = "user1"'

Then if we run following as user1, then we get error of insufficient resources in queue.

#!/bin/bash

#PBS -N job
#PBS -q testq1
#PBS -V
#PBS -l nodes=2:ppn=2

/opt/openmpi/bin/mpirun -machinefile $PBS_NODEFILE -np 4 /usr/bin/stress --cpu 1 --vm-bytes 10G --timeout 60s


Home > CentOS > CentOS 7.x > CentOS 7.x Rocks cluster 7.0 > PBS job queues resource management