Hi, We encounter an annoying issue here. Basically within the same partition we have two types of nodes, one with 24 cores and the other one with 28 cores, so we use node feature to distinguish them, savio2_c24 and savio2_c28. Slurm is reporting the proper configuration and feature from all angles, such as sinfo, scontrol, etc. But if I try to request the resource by node feature, only the savio2_c24 node feature is being honored, savio2_c28 will give this error,
srun: error: Unable to allocate resources: Requested node configuration is not available The commands that I ran were, 1. Good $ srun -p savio2 -C savio2_c24 -N1 -t 2:0 --pty bash 2. Bad $ srun -p savio2 -C savio2_c28 -N1 -t 2:0 --pty bash srun: error: Unable to allocate resources: Requested node configuration is not available $ sinfo -p savio2 -o "%N|%b"|less NODELIST|ACTIVE_FEATURES n0027.savio2,n0028.savio2,n0029.savio2,n0030.savio2,n0031.savio2,n0032.savio2,n0033.savio2,n0034.savio2,n0035.savio2,n0036.savio2,n0037.savio2,n0038.savio2,n0039.savio2,n0040.savio2,n0041.savio2,n0042.savio2,n0043.savio2,n0044.savio2,n0045.savio2,n0046.savio2,n0047.savio2,n0048.savio2,n0049.savio2,n0050.savio2,n0051.savio2,n0052.savio2,n0053.savio2,n0054.savio2,n0055.savio2,n0056.savio2,n0057.savio2,n0058.savio2,n0059.savio2,n0060.savio2,n0061.savio2,n0062.savio2,n0063.savio2,n0064.savio2,n0065.savio2,n0066.savio2,n0067.savio2,n0068.savio2,n0069.savio2,n0070.savio2,n0071.savio2,n0072.savio2,n0073.savio2,n0074.savio2,n0075.savio2,n0076.savio2,n0077.savio2,n0078.savio2,n0079.savio2,n0080.savio2,n0081.savio2,n0082.savio2,n0083.savio2,n0084.savio2,n0085.savio2,n0086.savio2,n0087.savio2,n0088.savio2,n0089.savio2,n0090.savio2,n0091.savio2,n0092.savio2,n0093.savio2,n0094.savio2,n0095.savio2,n0096.savio2,n0097.savio2,n0098.savio2,n0099.savio2,n0100.savio2,n0101.savio2,n0102.savio2,n0103.savio2,n0104.savio2,n0105.savio2,n0106.savio2,n0107.savio2,n0108.savio2,n0109.savio2,n0110.savio2,n0111.savio2,n0112.savio2,n0113.savio2,n0114.savio2,n0115.savio2,n0116.savio2,n0117.savio2,n0118.savio2,n0119.savio2,n0120.savio2,n0121.savio2,n0122.savio2,n0123.savio2,n0124.savio2,n0125.savio2,n0126.savio2,n0127.savio2,n0128.savio2,n0129.savio2,n0130.savio2,n0131.savio2,n0132.savio2,n0133.savio2,n0134.savio2,n0135.savio2,n0136.savio2,n0137.savio2,n0138.savio2,n0139.savio2,n0140.savio2,n0141.savio2,n0142.savio2,n0143.savio2,n0144.savio2,n0145.savio2,n0146.savio2,n0147.savio2,n0148.savio2,n0149.savio2,n0150.savio2,n0151.savio2,n0152.savio2,n0153.savio2,n0154.savio2,n0155.savio2,n0156.savio2,n0157.savio2,n0158.savio2,n0159.savio2,n0160.savio2,n0161.savio2,n0162.savio2,n0183.savio2,n0184.savio2,n0185.savio2,n0186.savio2|savio2,savio2_c24 n0187.savio2,n0188.savio2,n0189.savio2,n0190.savio2,n0191.savio2,n0192.savio2,n0193.savio2,n0194.savio2,n0195.savio2,n0196.savio2,n0197.savio2,n0198.savio2,n0199.savio2,n0200.savio2,n0201.savio2,n0202.savio2|savio2,savio2_c28 $ scontrol show node n0187.savio2 NodeName=n0187.savio2 Arch=x86_64 CoresPerSocket=14 CPUAlloc=0 CPUErr=0 CPUTot=28 CPULoad=0.02 AvailableFeatures=savio2,savio2_c28 ActiveFeatures=savio2,savio2_c28 Slurmctld debugging shows, [2017-05-25T11:34:36.646] _build_node_list: No nodes satisfy job 1343450 requirements in partition savio2 [2017-05-25T11:34:36.646] _slurm_rpc_allocate_resources: Requested node configuration is not available Any suggestions on how to debug or fix this issue? Thanks, Yong Qin
