I have system that will not accept qrsh jobs. qstat shows the job in
the r state:
174955 0.50934 QRLOGIN coffman r 02/22/2012 09:59:21 all.q@cs424
But the login never goes through.
a simple qsub will work fine.
>From the qmaster spool/messages file:
02/22/2012 10:00:21|worker|geshadow|W|job 174955.1 failed on host
cs424 assumedly after job because: job 174955.1 died through signal
KILL (9)
I was seeing the following in the spool/messages file on the exec host:
02/22/2012 09:39:47| main|cs424|E|shepherd of job 174535.1 exited
with exit status = 25
I restarted execd using a softstop and start and now nothing is being logged.
There are 2 active jobs on the exec host, so I can't reboot it.
There are some comments about the 25 status error in sge_conf.
max_advance_reservations - If the max_advance_reservations limit is
exceeded by an Advance Reservation request then the submission
command exits with exit status 25 and an appropriate error message
max_u_jobs - If the max_u_jobs limit is exceeded by a job
submission then the submission command exits with exit status 25 and
an appropriate error message.
max_jobs - If the max_jobs limit is exceeded by a job submission
then the submission command exits with exit status 25 and an
appropriate error message.
None of these are the issue. Any thoughts on how to debug this?
I also tried running strace against the sgeexecd and did not see
anything interesting. A grep of the jid produces the following:
mkdir("active_jobs/174996.1", 0755) = 0
mkdir("/tmp/174996.1.all.q", 0755) = 0
chown("/tmp/174996.1.all.q", 26927, 20) = 0
open("/opt/grid-6.2u5/ftcrnd/spool/cs424/active_jobs/174996.1/pe_hostfile",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
open("/opt/grid-6.2u5/ftcrnd/spool/cs424/active_jobs/174996.1/environment",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
open("active_jobs/174996.1/config", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
chdir("active_jobs/174996.1") = 0
stat("active_jobs/174996.1/addgrpid", {st_mode=S_IFREG|0644,
st_size=6, ...}) = 0
open("active_jobs/174996.1/addgrpid", O_RDONLY) = 4
stat("active_jobs/174996.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("active_jobs/174996.1/pid", O_RDONLY) = 6
stat("active_jobs/174996.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("active_jobs/174996.1/config", O_RDONLY) = 4
open("active_jobs/174996.1/exit_status", O_RDONLY) = 4
open("active_jobs/174996.1/error", O_RDONLY) = 4
open("active_jobs/174996.1/pid", O_RDONLY) = 4
open("active_jobs/174996.1/usage", O_RDONLY) = 4
stat("active_jobs/174996.1/checkpointed", 0x7ffff75e7280) = -1 ENOENT
(No such file or directory)
stat("active_jobs/174996.1/noresources", 0x7ffff75e7280) = -1 ENOENT
(No such file or directory)
open("active_jobs/174996.1", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4
lstat("active_jobs/174996.1/error", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
unlink("active_jobs/174996.1/error") = 0
lstat("active_jobs/174996.1/config", {st_mode=S_IFREG|0644,
st_size=2167, ...}) = 0
unlink("active_jobs/174996.1/config") = 0
lstat("active_jobs/174996.1/usage", {st_mode=S_IFREG|0644,
st_size=309, ...}) = 0
unlink("active_jobs/174996.1/usage") = 0
lstat("active_jobs/174996.1/environment", {st_mode=S_IFREG|0644,
st_size=1364, ...}) = 0
unlink("active_jobs/174996.1/environment") = 0
lstat("active_jobs/174996.1/pe_hostfile", {st_mode=S_IFREG|0644,
st_size=66, ...}) = 0
unlink("active_jobs/174996.1/pe_hostfile") = 0
lstat("active_jobs/174996.1/addgrpid", {st_mode=S_IFREG|0644,
st_size=6, ...}) = 0
unlink("active_jobs/174996.1/addgrpid") = 0
lstat("active_jobs/174996.1/trace", {st_mode=S_IFREG|0644,
st_size=5979, ...}) = 0
unlink("active_jobs/174996.1/trace") = 0
lstat("active_jobs/174996.1/job_pid", {st_mode=S_IFREG|0644,
st_size=6, ...}) = 0
unlink("active_jobs/174996.1/job_pid") = 0
lstat("active_jobs/174996.1/pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
unlink("active_jobs/174996.1/pid") = 0
lstat("active_jobs/174996.1/shepherd_about_to_exit",
{st_mode=S_IFREG|0644, st_size=0, ...}) = 0
unlink("active_jobs/174996.1/shepherd_about_to_exit") = 0
lstat("active_jobs/174996.1/exit_status", {st_mode=S_IFREG|0644,
st_size=2, ...}) = 0
unlink("active_jobs/174996.1/exit_status") = 0
rmdir("active_jobs/174996.1") = 0
open("/tmp/174996.1.all.q", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4
rmdir("/tmp/174996.1.all.q") = 0
Thanks.
--
-MichaelC
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
