> On 12/08/16 10:06, Christopher Samuel wrote:
>
>> Not had any reports of problems (that have been relayed to me at least)
>> but can't check because those nodes are busy at the moment, sorry!
>
> One node just became free and I can confirm this works:
>
> [samuel@spartan Ping]$ srun -p physical --gres=gpu:1 --pty -u /bin/bash -i -l
> srun: job 33361 queued and waiting for resources
> srun: job 33361 has been allocated resources
> [samuel@spartan-gpu002 ~]$ logout
> [samuel@spartan Ping]$
Thanks, Chris.
I ran an strace -f on the slurmd process and kicked one of these off. Still not
really any closer to figuring out where on earth it is getting that device
number (though I have seen the format of the number change even — sometimes
it’s got XXX:YYYYYY and sometimes it’s YYYYYY. It appears to go through the
devices on the list, then go through adding devices to the allow list, and then
adds this mystery device to both the allow and deny lists in a slightly
different form.
I’d post the output, but it seems like there’s a possibility that it might
contain my munge key. Here’s a short example from right where things appear to
go wrong, anyway:
pid 11767]
open("/sys/fs/cgroup/devices/slurm/uid_109366/job_5377709/devices.allow",
O_WRONLY) = 10
[pid 11767] write(10, "c 202:0 rwm", 11) = 11
[pid 11767] close(10) = 0
[pid 11767] poll([{fd=3, events=POLLOUT}], 1, 5000) = 1 ([{fd=3,
revents=POLLOUT}])
[pid 11767] fstat(3, {st_mode=S_IFREG|0600, st_size=28617, ...}) = 0
[pid 11767] write(3, "[2016-08-11T22:53:50.869] [53777"..., 75) = 75
[pid 11767]
open("/sys/fs/cgroup/devices/slurm/uid_109366/job_5377709/devices.allow",
O_WRONLY) = 10
[pid 11767] write(10, "501835 rwm", 10) = -1 EINVAL (Invalid argument)
[pid 11767] close(10) = 0
[pid 11767] poll([{fd=3, events=POLLOUT}], 1, 5000) = 1 ([{fd=3,
revents=POLLOUT}])
[pid 11767] fstat(3, {st_mode=S_IFREG|0600, st_size=28692, ...}) = 0
[pid 11767] write(3, "[2016-08-11T22:53:50.869] [53777"..., 89) = 89
[pid 11767]
open("/sys/fs/cgroup/devices/slurm/uid_109366/job_5377709/devices.deny",
O_WRONLY) = 10
[pid 11767] write(10, "501835 rwm501835 rwm", 20) = -1 EINVAL (Invalid argument)
[pid 11767] close(10) = 0
[pid 11767] umask(022) = 022
[pid 11767] mkdir("/sys/fs/cgroup/devices/slurm/uid_109366/job_5377709/step_0",
0755) = 0
[pid 11767] umask(022) = 022
[pid 11767] chown("/sys/fs/cgroup/devices/slurm/uid_109366/job_5377709/step_0",
109366, 109366) = 0
[pid 11767]
open("/sys/fs/cgroup/devices/slurm/uid_109366/job_5377709/step_0/notify_on_release",
O_WRONLY) = 10
[pid 11767] write(10, "1", 1) = 1
[pid 11767] close(10) = 0
[pid 11767] open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such
device or address)
[pid 11767] writev(2, [{"*** Error in `", 14}, {"slurmstepd: [5377709.0]", 23},
{"': ", 3}, {"malloc(): memory corruption", 27}, {": 0x", 4},
{"0000000001dc91c0", 16}, {" ***\n", 5}], 7) = 92
[pid 11767] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x2ab279a31000
[pid 11767] open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such
device or address)
[pid 11767] writev(2, [{"*** Error in `", 14}, {"slurmstepd: [5377709.0]", 23},
{"': ", 3}, {"malloc(): memory corruption", 27}, {": 0x", 4},
{"0000000001dc91c0", 16}, {" ***\n", 5}], 7) = 92
[pid 11767] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x2ab279a32000
[pid 11767] munmap(0x2ab279a31000, 4096) = 0
[pid 11767] futex(0x2ab27ac3cd20, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 11768] <... nanosleep resumed> 0x2ab279b3dcf0) = 0
[pid 11768] futex(0x7fa17c, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 11769] <... nanosleep resumed> NULL) = 0
[pid 11769] nanosleep({0, 999999000}, NULL) = 0
[pid 11769] nanosleep({1, 0}, NULL) = 0
[pid 11769] nanosleep({1, 0}, NULL) = 0
[pid 11769] nanosleep({1, 0}, NULL) = 0
Anyone got any ideas? I guess I’m going to have to try this on a second system
to make sure that something about the system itself is not the problem (it’s
stateless, but this is the only machine I’ve tested this particular combination
on as this is the one that’s available for testing).
--
____
|| \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - [email protected]
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'
