> On Aug 11, 2016, at 5:50 PM, Kilian Cavalotti > <[email protected]> wrote: > > On Thu, Aug 11, 2016 at 12:46 PM, Ryan Novosielski <[email protected]> > wrote: >> I’ll try adding the Gres debugging, but is there some way to figure out what >> this alleged device “819275” is (this number will change with each job). > > Weird, indeed. /dev/nv* devices should be 195:x, and slurmd should log > something like this: > Allowing access to device c 195:0 rwm > Not allowing access to device c 195:1 rwm > Not allowing access to device c 195:2 rwm > Not allowing access to device c 195:3 rwm > > The fact that it's "actively" allowing access to bogus device 819275 > makes me think it considers it as the actual GPU device. Except it got > the wrong major for it.
I really don’t know where it’s getting that from. I tried a find /dev -ls and neither the string 819275 or 819 or 275 appear anywhere. Like I said, though, that number will change from invocation to invocation. I wonder if this is some unexpected systemd/cgroup interaction that hasn’t been noticed before? I don’t know how many people are using RHEL/CentOS 7 at this point /and/ bothering with cgroups (I’m surprised at user group meetings of one sort or another to hear how few people do this). > What does "ls -al /dev/nv*" look like on the GPU node? [root@gpu002 ~]# ls -la /dev/nv* crw-rw-rw- 1 root root 195, 0 Aug 10 19:39 /dev/nvidia0 crw-rw-rw- 1 root root 195, 1 Aug 10 19:39 /dev/nvidia1 crw-rw-rw- 1 root root 195, 2 Aug 10 19:39 /dev/nvidia2 crw-rw-rw- 1 root root 195, 3 Aug 10 19:39 /dev/nvidia3 crw-rw-rw- 1 root root 195, 4 Aug 10 19:39 /dev/nvidia4 crw-rw-rw- 1 root root 195, 5 Aug 10 19:39 /dev/nvidia5 crw-rw-rw- 1 root root 195, 6 Aug 10 19:39 /dev/nvidia6 crw-rw-rw- 1 root root 195, 7 Aug 10 19:39 /dev/nvidia7 crw-rw-rw- 1 root root 195, 255 Aug 10 19:39 /dev/nvidiactl crw------- 1 root root 10, 144 Aug 10 19:39 /dev/nvram > And which version of Slurm is it? 15.08.4 — I’ve considered trying 15.08.latest before I continue on with this, but I figured someone will tell me if this is some odd behavior that was dealt with already (excuse the laziness — stretched a little thin). NVIDIA driver is 352.63, in case that makes any difference. One thing I guess I have to consider is that this is a node that is also refusing to boot unless I remove “quiet” from the boot command line, but I guess I was thinking that that is probably related to something else odd about the boot process/Warewulf and not a problem with the system that would cause other screwy behavior. Unfortunately, I don’t have another system to try it out on at the moment, unfortunately (not one with real GPUs anyway). Happened to notice some of this going on in dmesg too: [66169.268646] slurmstepd[20020]: segfault at 5a ip 00002af2b0bef4bd sp 00007ffca00dafe0 error 4 in libc-2.17.so[2af2b0b73000+1b6000] [66533.200868] slurmstepd[20158]: segfault at 5a ip 00002b9f05ad74bd sp 00007ffd35f8b060 error 4 in libc-2.17.so[2b9f05a5b000+1b6000] [66748.038193] slurmstepd[20251]: segfault at 5a ip 00002ba4c80e24bd sp 00007fff3b770990 error 4 in libc-2.17.so[2ba4c8066000+1b6000] [67278.028235] slurmstepd[20413]: segfault at 5a ip 00002aeb6a0f74bd sp 00007fffa3594840 error 4 in libc-2.17.so[2aeb6a07b000+1b6000] [67864.519436] slurmstepd[20728]: segfault at 5a ip 00002b355de974bd sp 00007fffdf239c70 error 4 in libc-2.17.so[2b355de1b000+1b6000] -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - [email protected] || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'
signature.asc
Description: Message signed with OpenPGP using GPGMail
