hwNodesΒΆ
hwNodes will provide you with a high level overview of the status of all DMOG comptue nodes.
An example output:
[gp27@login1 [dmog] scripts]$ hwNodes
NODELIST CPUS S:C:T MEMORY ALLOCMEM FREE_MEM GRES GRES_USED CPUS(A/I/O/T) CPU_LOAD STATE REASON
======================================================================================================================================================
gpu01 64 2:32:1 256000 0 39990 gpu:2 gpu:0 0/64/0/64 0.00 idle none
gpu02 64 2:32:1 256000 0 142873 gpu:2 gpu:0 0/64/0/64 0.00 idle none
gpu03 64 2:32:1 256000 102400 72185 gpu:2 gpu:1 1/63/0/64 0.68 mixed none
gpu04 64 2:32:1 256000 0 85584 gpu:2 gpu:0 0/64/0/64 0.00 idle none
gpu05 64 2:32:1 256000 0 102740 gpu:2 gpu:0 0/64/0/64 0.00 idle none
node01 64 2:32:1 514048 185168 484965 (null) gpu:0 64/0/0/64 14.47 allocated none
node02 64 2:32:1 514048 131072 248131 (null) gpu:0 32/0/32/64 33.00 draining Kill task failed
node03 64 2:32:1 514048 54520 410692 (null) gpu:0 64/0/0/64 61.06 allocated none
node04 64 2:32:1 514048 59040 302075 (null) gpu:0 64/0/0/64 52.00 allocated none
node05 64 2:32:1 514048 262144 407828 (null) gpu:0 64/0/0/64 65.00 allocated none