Hi Team,
Our users jobs are exiting with below error for random nodes. could you
please help us to resolve this issue?
[root@bng1grcdc200 output.228472]# cat user_script.stderr
[bng1grcdc181:08381] [[54933,0],0] ORTE_ERROR_LOG: The specified
application failed to start in file plm_lsf_module.c at line 308
[bng1grcdc181:08381] lsb_launch failed: 0
------------------------------------------------------------------------
--
A daemon (pid unknown) died unexpectedly on signal 1 while attempting
to
launch so we are aborting.
There may be more information reported by the environment (see above).
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have
the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
------------------------------------------------------------------------
--
------------------------------------------------------------------------
--
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
------------------------------------------------------------------------
--
------------------------------------------------------------------------
--
mpirun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
------------------------------------------------------------------------
--
bng1grcdc172 - daemon did not report back when launched
bng1grcdc154 - daemon did not report back when launched
bng1grcdc198 - daemon did not report back when launched
bng1grcdc183 - daemon did not report back when launched
bng1grcdc187 - daemon did not report back when launched
bng1grcdc196 - daemon did not report back when launched
bng1grcdc153 - daemon did not report back when launched
bng1grcdc173 - daemon did not report back when launched
bng1grcdc193 - daemon did not report back when launched
bng1grcdc185 - daemon did not report back when launched
bng1grcdc176 - daemon did not report back when launched
bng1grcdc190 - daemon did not report back when launched
bng1grcdc194 - daemon did not report back when launched
bng1grcdc156 - daemon did not report back when launched
Thanks,
Bharati Singh
