Hi Siegmar,
The attached JRE log shows very important information.
When JRE loads the MPI class, JNI_OnLoad function in
libmpi_java.so (Open MPI library; written in C) is called.
And probably mca_base_var_cache_files function passes NULL
to asprintf function. I don't know how this situation occurs.
You may be able to track this by inserting debug printf in
Open MPI code shown in the stack trace, or by using gdb or
something.
hs_err_pid13080.log:
----------------------------------------------------------------
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
si_addr=0x0000000000000000
Stack: [0xffffffff7b400000,0xffffffff7b500000], sp=0xffffffff7b4fc730, free
space=1009k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0x3c7f0] strlen+0x50
C [libc.so.1+0xaf640] vsnprintf+0x84
C [libc.so.1+0xaadb4] vasprintf+0x20
C [libc.so.1+0xaaf04] asprintf+0x28
C [libopen-pal.so.0.0.0+0xaf3cc] mca_base_var_cache_files+0x160
C [libopen-pal.so.0.0.0+0xaed90] mca_base_var_init+0x4e8
C [libopen-pal.so.0.0.0+0xb260c] register_variable+0x214
C [libopen-pal.so.0.0.0+0xb36a0] mca_base_var_register+0x104
C [libmpi_java.so.0.0.0+0x221e8] JNI_OnLoad+0x128
C [libjava.so+0x10860] Java_java_lang_ClassLoader_00024NativeLibrary_load+0xb8
j java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+-665819
j java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+0
j java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+328
j java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+290
j java.lang.Runtime.loadLibrary0(Ljava/lang/Class;Ljava/lang/String;)V+54
j java.lang.System.loadLibrary(Ljava/lang/String;)V+7
j mpi.MPI.<clinit>()V+28
----------------------------------------------------------------
Regards,
Takahiro
> Hello Oscar,
>
> do you have time to look into my problem? Probably Takahiro has a
> point and gdb behaves differently on Solaris and Linux, so that
> the differing outputs have no meaning. I tried to debug my Java
> program, but without success so far, because I wasn't able to get
> into the Java program to set a breakpoint or to see the code. Have
> you succeeded to debug a mpiJava program? If so, how must I call
> gdb (I normally use "gdb mipexec" and then "run -np 1 java ...")?
> What can I do to get helpful information to track the error down?
> I have attached the error log file. Perhaps you can see if something
> is going wrong with the Java interface. Thank you very much for your
> help and any hints for the usage of gdb with mpiJava in advance.
> Please let me know if I can provide anything else.
>
>
> Kind regards
>
> Siegmar
>
>
> > > I think that it must have to do with MPI, because everything
> > > works fine on Linux and my Java program works fine with an older
> > > MPI version (openmpi-1.8.2a1r31804) as well.
> >
> > Yes. I also think it must have to do with MPI.
> > But java process side, not mpiexec process side.
> >
> > When you run Java MPI program via mpiexec, a mpiexec process
> > process launch a java process. When the java process (your
> > Java program) calls a MPI method, native part (written in C/C++)
> > of the MPI library is called. It runs in java process, not in
> > mpiexec process. I suspect that part.
> >
> > > On Solaris things are different.
> >
> > Are you saying the following difference?
> > After this line,
> > > 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> > Linux shows
> > > orte_job_state_to_str (state=1)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> > > 217 switch(state) {
> > but Solaris shows
> > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> > > 122 if (NULL == name) {
> >
> > Each macro is defined as:
> >
> > #define ORTE_ACTIVATE_JOB_STATE(j, s) \
> > do { \
> > orte_job_t *shadow=(j); \
> > opal_output_verbose(1, orte_state_base_framework.framework_output, \
> > "%s ACTIVATE JOB %s STATE %s AT %s:%d", \
> > ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \
> > (NULL == shadow) ? "NULL" : \
> > ORTE_JOBID_PRINT(shadow->jobid),
> > \
> > orte_job_state_to_str((s)), \
> > __FILE__, __LINE__); \
> > orte_state.activate_job_state(shadow, (s)); \
> > } while(0);
> >
> > #define ORTE_NAME_PRINT(n) \
> > orte_util_print_name_args(n)
> >
> > #define ORTE_JOBID_PRINT(n) \
> > orte_util_print_jobids(n)
> >
> > I'm not sure, but I think the gdb on Solaris steps into
> > orte_util_print_name_args, but gdb on Linux doesn't step into
> > orte_util_print_name_args and orte_util_print_jobids for some
> > reason, or orte_job_state_to_str is evaluated before them.
> >
> > So I think it's not an important difference.
> >
> > You showed the following lines.
> > > > > orterun (argc=5, argv=0xffffffff7fffe0d8)
> > > > > at
> ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084
> > > > > 1084 while (orte_event_base_active) {
> > > > > (gdb)
> > > > > 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > > > (gdb)
> >
> > I'm not familiar with this code but I think this part (in mpiexec
> > process) is only waiting the java process to terminate (normally
> > or abnormally). So I think the problem is not in a mpiexec process
> > but in a java process.
> >
> > Regards,
> > Takahiro
> >
> > > Hi Takahiro,
> > >
> > > > mpiexec and java run as distinct processes. Your JRE message
> > > > says java process raises SEGV. So you should trace the java
> > > > process, not the mpiexec process. And more, your JRE message
> > > > says the crash happened outside the Java Virtual Machine in
> > > > native code. So usual Java program debugger is useless.
> > > > You should trace native code part of the java process.
> > > > Unfortunately I don't know how to debug such one.
> > >
> > > I think that it must have to do with MPI, because everything
> > > works fine on Linux and my Java program works fine with an older
> > > MPI version (openmpi-1.8.2a1r31804) as well.
> > >
> > > linpc1 x 112 mpiexec -np 1 java InitFinalizeMain
> > > Hello!
> > > linpc1 x 113
> > >
> > > Therefore I single stepped through the program on Linux as well
> > > and found a difference launching the process. On Linux I get the
> > > following sequence.
> > >
> > > Breakpoint 1, rsh_launch (jdata=0x614aa0)
> > > at
> ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876
> > > 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) {
> > > (gdb) s
> > > 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> > > (gdb) s
> > > orte_job_state_to_str (state=1)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> > > 217 switch(state) {
> > > (gdb)
> > > 221 return "PENDING INIT";
> > > (gdb)
> > > 317 }
> > > (gdb)
> > > orte_util_print_jobids (job=4294967295)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> > > 170 ptr = get_print_name_buffer();
> > > (gdb)
> > >
> > >
> > >
> > > On Solaris things are different.
> > >
> > > Breakpoint 1, rsh_launch (jdata=0x100125250)
> > > at
> ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876
> > > 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) {
> > > (gdb) s
> > > 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> > > (gdb) s
> > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> > > 122 if (NULL == name) {
> > > (gdb)
> > > 142 job = orte_util_print_jobids(name->jobid);
> > > (gdb)
> > > orte_util_print_jobids (job=2673410048)
> > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> > > 170 ptr = get_print_name_buffer();
> > > (gdb)
> > >
> > >
> > >
> > > Is this normal or is it the reason for the crash on Solaris?
> > >
> > >
> > > Kind regards
> > >
> > > Siegmar
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > The log file output by JRE may help you.
> > > > > # An error report file with more information is saved as:
> > > > > #
> /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid13080.log
> > > >
> > > > Regards,
> > > > Takahiro
> > > >
> > > > > Hi,
> > > > >
> > > > > I installed openmpi-dev-124-g91e9686 on Solaris 10 Sparc with
> > > > > gcc-4.9.1 to track down the error with my small Java program.
> > > > > I started single stepping in orterun.c at line 1081 and
> > > > > continued until I got the segmentation fault. I get
> > > > > "jdata = 0x0" in version openmpi-1.8.2a1r31804, which is the
> > > > > last one which works with Java in my environment, while I get
> > > > > "jdata = 0x100125250" in this version. Unfortunately I don't
> > > > > know which files or variables are important to look at. Perhaps
> > > > > somebody can look at the following lines of code and tell me,
> > > > > which information I should provide to solve the problem. I know
> > > > > that Solaris isn't any longer on your list of supported systems,
> > > > > but perhaps we can get it working again, if you tell me what
> > > > > you need and I do the debugging.
> > > > >
> > > > > /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
> > > > > GNU gdb (GDB) 7.6.1
> > > > > ...
> > > > > (gdb) run -np 1 java InitFinalizeMain
> > > > > Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec \
> > > > > -np 1 java InitFinalizeMain
> > > > > [Thread debugging using libthread_db enabled]
> > > > > [New Thread 1 (LWP 1)]
> > > > > [New LWP 2 ]
> > > > > #
> > > > > # A fatal error has been detected by the Java Runtime Environment:
> > > > > #
> > > > > # SIGSEGV (0xb) at pc=0xffffffff7ea3c7f0, pid=13064, tid=2
> > > > > ...
> > > > > [LWP 2 exited]
> > > > > [New Thread 2 ]
> > > > > [Switching to Thread 1 (LWP 1)]
> > > > > sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be
> > > > > found to satisfy query
> > > > > (gdb) thread 1
> > > > > [Switching to thread 1 (LWP 1 )]
> > > > > #0 0xffffffff7f6173d0 in rtld_db_dlactivity () from
> /usr/lib/sparcv9/ld.so.1
> > > > > (gdb) b orterun.c:1081
> > > > > Breakpoint 1 at 0x1000070dc: file
> > > > > ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c,
> > > > > line
> 1081.
> > > > > (gdb) r
> > > > > The program being debugged has been started already.
> > > > > Start it from the beginning? (y or n) y
> > > > >
> > > > > Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 1
> > > > > java
> > > > > InitFinalizeMain
> > > > > [Thread debugging using libthread_db enabled]
> > > > > [New Thread 1 (LWP 1)]
> > > > > [New LWP 2 ]
> > > > > [Switching to Thread 1 (LWP 1)]
> > > > >
> > > > > Breakpoint 1, orterun (argc=5, argv=0xffffffff7fffe0d8)
> > > > > at
> ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1081
> > > > > 1081 rc = orte_plm.spawn(jdata);
> > > > > (gdb) print jdata
> > > > > $1 = (orte_job_t *) 0x100125250
> > > > > (gdb) s
> > > > > rsh_launch (jdata=0x100125250)
> > > > > at
> > > > >
> ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876
> > > > > 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) {
> > > > > (gdb) s
> > > > > 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> > > > > 122 if (NULL == name) {
> > > > > (gdb)
> > > > > 142 job = orte_util_print_jobids(name->jobid);
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=2502885376) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> > > > > 170 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd990)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=2502885376) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172
> > > > > 172 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 182 if (ORTE_JOBID_INVALID == job) {
> > > > > (gdb)
> > > > > 184 } else if (ORTE_JOBID_WILDCARD == job) {
> > > > > (gdb)
> > > > > 187 tmp1 = ORTE_JOB_FAMILY((unsigned long)job);
> > > > > (gdb)
> > > > > 188 tmp2 = ORTE_LOCAL_JOBID((unsigned long)job);
> > > > > (gdb)
> > > > > 189 snprintf(ptr->buffers[ptr->cntr++],
> > > > > (gdb)
> > > > > 193 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 194 }
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:143
> > > > > 143 vpid = orte_util_print_vpids(name->vpid);
> > > > > (gdb)
> > > > > orte_util_print_vpids (vpid=0) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:260
> > > > > 260 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd9a0)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_vpids (vpid=0) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:262
> > > > > 262 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 268 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 272 if (ORTE_VPID_INVALID == vpid) {
> > > > > (gdb)
> > > > > 274 } else if (ORTE_VPID_WILDCARD == vpid) {
> > > > > (gdb)
> > > > > 277 snprintf(ptr->buffers[ptr->cntr++],
> > > > > (gdb)
> > > > > 281 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 282 }
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:146
> > > > > 146 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffda60)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:148
> > > > > 148 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 154 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 158 snprintf(ptr->buffers[ptr->cntr++],
> > > > > (gdb)
> > > > > 162 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 163 }
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=4294967295) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> > > > > 170 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffda60)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=4294967295) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172
> > > > > 172 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 182 if (ORTE_JOBID_INVALID == job) {
> > > > > (gdb)
> > > > > 183 snprintf(ptr->buffers[ptr->cntr++],
> > > > > ORTE_PRINT_NAME_ARGS_MAX_SIZE, "[INVALID]");
> > > > > (gdb)
> > > > > 193 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 194 }
> > > > > (gdb)
> > > > > orte_job_state_to_str (state=1) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> > > > > 217 switch(state) {
> > > > > (gdb)
> > > > > 221 return "PENDING INIT";
> > > > > (gdb)
> > > > > 317 }
> > > > > (gdb)
> > > > > opal_output_verbose (level=1, output_id=0,
> > > > > format=0xffffffff7f14dd98 <orte_job_states>
> > > > > "\336\257\276\355\336\257\276\355")
> > > > > at ../../../openmpi-dev-124-g91e9686/opal/util/output.c:373
> > > > > 373 va_start(arglist, format);
> > > > > (gdb)
> > > > > 369 {
> > > > > (gdb)
> > > > > 370 if (output_id >= 0 && output_id < OPAL_OUTPUT_MAX_STREAMS
> > > > > &&
> > > > > (gdb)
> > > > > 377 }
> > > > > (gdb)
> > > > > orte_state_base_activate_job_state (jdata=0x100125250, state=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:33
> > > > > 33 opal_list_item_t *itm, *any=NULL, *error=NULL;
> > > > > (gdb)
> > > > > 37 for (itm = opal_list_get_first(&orte_job_states);
> > > > > (gdb)
> > > > > opal_list_get_first (list=0xffffffff7f14dd98 <orte_job_states>)
> > > > > at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_list.h:320
> > > > > 320 opal_list_item_t* item =
> > > > > (opal_list_item_t*)list->opal_list_sentinel.opal_list_next;
> > > > > (gdb)
> > > > > 324 assert(1 == item->opal_list_item_refcount);
> > > > > (gdb)
> > > > > 325 assert( list == item->opal_list_item_belong_to );
> > > > > (gdb)
> > > > > 328 return item;
> > > > > (gdb)
> > > > > 329 }
> > > > > (gdb)
> > > > > orte_state_base_activate_job_state (jdata=0x100125250, state=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:38
> > > > > 38 itm != opal_list_get_end(&orte_job_states);
> > > > > (gdb)
> > > > > opal_list_get_end (list=0xffffffff7f14dd98 <orte_job_states>)
> > > > > at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_list.h:399
> > > > > 399 return &(list->opal_list_sentinel);
> > > > > (gdb)
> > > > > 400 }
> > > > > (gdb)
> > > > > orte_state_base_activate_job_state (jdata=0x100125250, state=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:37
> > > > > 37 for (itm = opal_list_get_first(&orte_job_states);
> > > > > (gdb)
> > > > > 40 s = (orte_state_t*)itm;
> > > > > (gdb)
> > > > > 41 if (s->job_state == ORTE_JOB_STATE_ANY) {
> > > > > (gdb)
> > > > > 45 if (s->job_state == ORTE_JOB_STATE_ERROR) {
> > > > > (gdb)
> > > > > 48 if (s->job_state == state) {
> > > > > (gdb)
> > > > > 49 OPAL_OUTPUT_VERBOSE((1,
> > > > > orte_state_base_framework.framework_output,
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
> > > > > 122 if (NULL == name) {
> > > > > (gdb)
> > > > > 142 job = orte_util_print_jobids(name->jobid);
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=2502885376) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> > > > > 170 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd880)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=2502885376) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172
> > > > > 172 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 182 if (ORTE_JOBID_INVALID == job) {
> > > > > (gdb)
> > > > > 184 } else if (ORTE_JOBID_WILDCARD == job) {
> > > > > (gdb)
> > > > > 187 tmp1 = ORTE_JOB_FAMILY((unsigned long)job);
> > > > > (gdb)
> > > > > 188 tmp2 = ORTE_LOCAL_JOBID((unsigned long)job);
> > > > > (gdb)
> > > > > 189 snprintf(ptr->buffers[ptr->cntr++],
> > > > > (gdb)
> > > > > 193 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 194 }
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:143
> > > > > 143 vpid = orte_util_print_vpids(name->vpid);
> > > > > (gdb)
> > > > > orte_util_print_vpids (vpid=0) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:260
> > > > > 260 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd890)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_vpids (vpid=0) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:262
> > > > > 262 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 268 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 272 if (ORTE_VPID_INVALID == vpid) {
> > > > > (gdb)
> > > > > 274 } else if (ORTE_VPID_WILDCARD == vpid) {
> > > > > (gdb)
> > > > > 277 snprintf(ptr->buffers[ptr->cntr++],
> > > > > (gdb)
> > > > > 281 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 282 }
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:146
> > > > > 146 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd950)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_name_args (name=0x100118380 <orte_process_info+104>)
> > > > > at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:148
> > > > > 148 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 154 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 158 snprintf(ptr->buffers[ptr->cntr++],
> > > > > (gdb)
> > > > > 162 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 163 }
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=4294967295) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170
> > > > > 170 ptr = get_print_name_buffer();
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92
> > > > > 92 if (!fns_init) {
> > > > > (gdb)
> > > > > 101 ret = opal_tsd_getspecific(print_args_tsd_key,
> (void**)&ptr);
> > > > > (gdb)
> > > > > opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd950)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163
> > > > > 163 *valuep = pthread_getspecific(key);
> > > > > (gdb)
> > > > > 164 return OPAL_SUCCESS;
> > > > > (gdb)
> > > > > 165 }
> > > > > (gdb)
> > > > > get_print_name_buffer () at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102
> > > > > 102 if (OPAL_SUCCESS != ret) return NULL;
> > > > > (gdb)
> > > > > 104 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 113 return (orte_print_args_buffers_t*) ptr;
> > > > > (gdb)
> > > > > 114 }
> > > > > (gdb)
> > > > > orte_util_print_jobids (job=4294967295) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172
> > > > > 172 if (NULL == ptr) {
> > > > > (gdb)
> > > > > 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) {
> > > > > (gdb)
> > > > > 182 if (ORTE_JOBID_INVALID == job) {
> > > > > (gdb)
> > > > > 183 snprintf(ptr->buffers[ptr->cntr++],
> > > > > ORTE_PRINT_NAME_ARGS_MAX_SIZE, "[INVALID]");
> > > > > (gdb)
> > > > > 193 return ptr->buffers[ptr->cntr-1];
> > > > > (gdb)
> > > > > 194 }
> > > > > (gdb)
> > > > > orte_job_state_to_str (state=1) at
> > > > > ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
> > > > > 217 switch(state) {
> > > > > (gdb)
> > > > > 221 return "PENDING INIT";
> > > > > (gdb)
> > > > > 317 }
> > > > > (gdb)
> > > > > opal_output_verbose (level=1, output_id=-1, format=0x1 <Address 0x1
> > > > > out
> of
> > > > > bounds>)
> > > > > at ../../../openmpi-dev-124-g91e9686/opal/util/output.c:373
> > > > > 373 va_start(arglist, format);
> > > > > (gdb)
> > > > > 369 {
> > > > > (gdb)
> > > > > 370 if (output_id >= 0 && output_id < OPAL_OUTPUT_MAX_STREAMS
> > > > > &&
> > > > > (gdb)
> > > > > 377 }
> > > > > (gdb)
> > > > > orte_state_base_activate_job_state (jdata=0x100125250, state=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:54
> > > > > 54 if (NULL == s->cbfunc) {
> > > > > (gdb)
> > > > > 62 caddy = OBJ_NEW(orte_state_caddy_t);
> > > > > (gdb)
> > > > > opal_obj_new_debug (type=0xffffffff7f14c7d8
> > > > > <orte_state_caddy_t_class>,
> > > > > file=0xffffffff7f034c08
> > > > >
> "../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c",
> > > > > line=62) at
> ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:249
> > > > > 249 opal_object_t* object = opal_obj_new(type);
> > > > > (gdb)
> > > > > opal_obj_new (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>)
> > > > > at
> > > > > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:465
> > > > > 465 assert(cls->cls_sizeof >= sizeof(opal_object_t));
> > > > > (gdb)
> > > > > 470 object = (opal_object_t *) malloc(cls->cls_sizeof);
> > > > > (gdb)
> > > > > 472 if (0 == cls->cls_initialized) {
> > > > > (gdb)
> > > > > 473 opal_class_initialize(cls);
> > > > > (gdb)
> > > > > opal_class_initialize (cls=0xffffffff7f14c7d8
> <orte_state_caddy_t_class>)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:79
> > > > > 79 assert(cls);
> > > > > (gdb)
> > > > > 84 if (1 == cls->cls_initialized) {
> > > > > (gdb)
> > > > > 87 opal_atomic_lock(&class_lock);
> > > > > (gdb)
> > > > > opal_atomic_lock (lock=0xffffffff7ee89bf0 <class_lock>)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:397
> > > > > 397 while( !opal_atomic_cmpset_acq_32( &(lock->u.lock),
> > > > > (gdb)
> > > > > opal_atomic_cmpset_acq_32 (addr=0xffffffff7ee89bf0 <class_lock>,
> oldval=0,
> > > > > newval=1)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:107
> > > > > 107 rc = opal_atomic_cmpset_32(addr, oldval, newval);
> > > > > (gdb)
> > > > > opal_atomic_cmpset_32 (addr=0xffffffff7ee89bf0 <class_lock>,
> > > > > oldval=0,
> newval=1)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:93
> > > > > 93 int32_t ret = newval;
> > > > > (gdb)
> > > > > 95 __asm__ __volatile__("casa [%1] " ASI_P ", %2, %0"
> > > > > (gdb)
> > > > > 98 return (ret == oldval);
> > > > > (gdb)
> > > > > 99 }
> > > > > (gdb)
> > > > > opal_atomic_cmpset_acq_32 (addr=0xffffffff7ee89bf0 <class_lock>,
> oldval=0,
> > > > > newval=1)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:108
> > > > > 108 opal_atomic_rmb();
> > > > > (gdb)
> > > > > opal_atomic_rmb () at
> > > > > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:63
> > > > > 63 MEMBAR("#LoadLoad");
> > > > > (gdb)
> > > > > 64 }
> > > > > (gdb)
> > > > > opal_atomic_cmpset_acq_32 (addr=0xffffffff7ee89bf0 <class_lock>,
> oldval=0,
> > > > > newval=1)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:110
> > > > > 110 return rc;
> > > > > (gdb)
> > > > > 111 }
> > > > > (gdb)
> > > > > opal_atomic_lock (lock=0xffffffff7ee89bf0 <class_lock>)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:403
> > > > > 403 }
> > > > > (gdb)
> > > > > opal_class_initialize (cls=0xffffffff7f14c7d8
> <orte_state_caddy_t_class>)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:93
> > > > > 93 if (1 == cls->cls_initialized) {
> > > > > (gdb)
> > > > > 103 cls->cls_depth = 0;
> > > > > (gdb)
> > > > > 104 cls_construct_array_count = 0;
> > > > > (gdb)
> > > > > 105 cls_destruct_array_count = 0;
> > > > > (gdb)
> > > > > 106 for (c = cls; c; c = c->cls_parent) {
> > > > > (gdb)
> > > > > 107 if( NULL != c->cls_construct ) {
> > > > > (gdb)
> > > > > 108 cls_construct_array_count++;
> > > > > (gdb)
> > > > > 110 if( NULL != c->cls_destruct ) {
> > > > > (gdb)
> > > > > 111 cls_destruct_array_count++;
> > > > > (gdb)
> > > > > 113 cls->cls_depth++;
> > > > > (gdb)
> > > > > 106 for (c = cls; c; c = c->cls_parent) {
> > > > > (gdb)
> > > > > 107 if( NULL != c->cls_construct ) {
> > > > > (gdb)
> > > > > 110 if( NULL != c->cls_destruct ) {
> > > > > (gdb)
> > > > > 113 cls->cls_depth++;
> > > > > (gdb)
> > > > > 106 for (c = cls; c; c = c->cls_parent) {
> > > > > (gdb)
> > > > > 122 (void
> (**)(opal_object_t*))malloc((cls_construct_array_count +
> > > > > (gdb)
> > > > > 123
> cls_destruct_array_count + 2)
> > > > > *
> > > > > (gdb)
> > > > > 122 (void
> (**)(opal_object_t*))malloc((cls_construct_array_count +
> > > > > (gdb)
> > > > > 121 cls->cls_construct_array =
> > > > > (gdb)
> > > > > 125 if (NULL == cls->cls_construct_array) {
> > > > > (gdb)
> > > > > 130 cls->cls_construct_array + cls_construct_array_count
> > > > > +
> 1;
> > > > > (gdb)
> > > > > 129 cls->cls_destruct_array =
> > > > > (gdb)
> > > > > 136 cls_construct_array = cls->cls_construct_array +
> > > > > cls_construct_array_count;
> > > > > (gdb)
> > > > > 137 cls_destruct_array = cls->cls_destruct_array;
> > > > > (gdb)
> > > > > 139 c = cls;
> > > > > (gdb)
> > > > > 140 *cls_construct_array = NULL; /* end marker for the
> constructors */
> > > > > (gdb)
> > > > > 141 for (i = 0; i < cls->cls_depth; i++) {
> > > > > (gdb)
> > > > > 142 if( NULL != c->cls_construct ) {
> > > > > (gdb)
> > > > > 143 --cls_construct_array;
> > > > > (gdb)
> > > > > 144 *cls_construct_array = c->cls_construct;
> > > > > (gdb)
> > > > > 146 if( NULL != c->cls_destruct ) {
> > > > > (gdb)
> > > > > 147 *cls_destruct_array = c->cls_destruct;
> > > > > (gdb)
> > > > > 148 cls_destruct_array++;
> > > > > (gdb)
> > > > > 150 c = c->cls_parent;
> > > > > (gdb)
> > > > > 141 for (i = 0; i < cls->cls_depth; i++) {
> > > > > (gdb)
> > > > > 142 if( NULL != c->cls_construct ) {
> > > > > (gdb)
> > > > > 146 if( NULL != c->cls_destruct ) {
> > > > > (gdb)
> > > > > 150 c = c->cls_parent;
> > > > > (gdb)
> > > > > 141 for (i = 0; i < cls->cls_depth; i++) {
> > > > > (gdb)
> > > > > 152 *cls_destruct_array = NULL; /* end marker for the
> destructors */
> > > > > (gdb)
> > > > > 154 cls->cls_initialized = 1;
> > > > > (gdb)
> > > > > 155 save_class(cls);
> > > > > (gdb)
> > > > > save_class (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:188
> > > > > 188 if (num_classes >= max_classes) {
> > > > > (gdb)
> > > > > 189 expand_array();
> > > > > (gdb)
> > > > > expand_array () at
> ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:201
> > > > > 201 max_classes += increment;
> > > > > (gdb)
> > > > > 202 classes = (void**)realloc(classes, sizeof(opal_class_t*)
> > > > > *
> > > > > max_classes);
> > > > > (gdb)
> > > > > 203 if (NULL == classes) {
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 208 classes[i] = NULL;
> > > > > (gdb)
> > > > > 207 for (i = num_classes; i < max_classes; ++i) {
> > > > > (gdb)
> > > > > 210 }
> > > > > (gdb)
> > > > > save_class (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:192
> > > > > 192 classes[num_classes] = cls->cls_construct_array;
> > > > > (gdb)
> > > > > 193 ++num_classes;
> > > > > (gdb)
> > > > > 194 }
> > > > > (gdb)
> > > > > opal_class_initialize (cls=0xffffffff7f14c7d8
> <orte_state_caddy_t_class>)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:159
> > > > > 159 opal_atomic_unlock(&class_lock);
> > > > > (gdb)
> > > > > opal_atomic_unlock (lock=0xffffffff7ee89bf0 <class_lock>)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:409
> > > > > 409 opal_atomic_wmb();
> > > > > (gdb)
> > > > > opal_atomic_wmb () at
> > > > > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:69
> > > > > 69 MEMBAR("#StoreStore");
> > > > > (gdb)
> > > > > 70 }
> > > > > (gdb)
> > > > > opal_atomic_unlock (lock=0xffffffff7ee89bf0 <class_lock>)
> > > > > at
> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:410
> > > > > 410 lock->u.lock=OPAL_ATOMIC_UNLOCKED;
> > > > > (gdb)
> > > > > 411 }
> > > > > (gdb)
> > > > > opal_class_initialize (cls=0xffffffff7f14c7d8
> <orte_state_caddy_t_class>)
> > > > > at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:160
> > > > > 160 }
> > > > > (gdb)
> > > > > opal_obj_new (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>)
> > > > > at
> > > > > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:475
> > > > > 475 if (NULL != object) {
> > > > > (gdb)
> > > > > 476 object->obj_class = cls;
> > > > > (gdb)
> > > > > 477 object->obj_reference_count = 1;
> > > > > (gdb)
> > > > > 478 opal_obj_run_constructors(object);
> > > > > (gdb)
> > > > > opal_obj_run_constructors (object=0x1001bfcf0)
> > > > > at
> > > > > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:420
> > > > > 420 assert(NULL != object->obj_class);
> > > > > (gdb)
> > > > > 422 cls_construct = object->obj_class->cls_construct_array;
> > > > > (gdb)
> > > > > 423 while( NULL != *cls_construct ) {
> > > > > (gdb)
> > > > > 424 (*cls_construct)(object);
> > > > > (gdb)
> > > > > orte_state_caddy_construct (caddy=0x1001bfcf0)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_frame.c:84
> > > > > 84 memset(&caddy->ev, 0, sizeof(opal_event_t));
> > > > > (gdb)
> > > > > 85 caddy->jdata = NULL;
> > > > > (gdb)
> > > > > 86 }
> > > > > (gdb)
> > > > > opal_obj_run_constructors (object=0x1001bfcf0)
> > > > > at
> > > > > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:425
> > > > > 425 cls_construct++;
> > > > > (gdb)
> > > > > 423 while( NULL != *cls_construct ) {
> > > > > (gdb)
> > > > > 427 }
> > > > > (gdb)
> > > > > opal_obj_new (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>)
> > > > > at
> > > > > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:480
> > > > > 480 return object;
> > > > > (gdb)
> > > > > 481 }
> > > > > (gdb)
> > > > > opal_obj_new_debug (type=0xffffffff7f14c7d8
> > > > > <orte_state_caddy_t_class>,
> > > > > file=0xffffffff7f034c08
> > > > >
> "../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c",
> > > > > line=62) at
> ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:250
> > > > > 250 object->obj_magic_id = OPAL_OBJ_MAGIC_ID;
> > > > > (gdb)
> > > > > 251 object->cls_init_file_name = file;
> > > > > (gdb)
> > > > > 252 object->cls_init_lineno = line;
> > > > > (gdb)
> > > > > 253 return object;
> > > > > (gdb)
> > > > > 254 }
> > > > > (gdb)
> > > > > orte_state_base_activate_job_state (jdata=0x100125250, state=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:63
> > > > > 63 if (NULL != jdata) {
> > > > > (gdb)
> > > > > 64 caddy->jdata = jdata;
> > > > > (gdb)
> > > > > 65 caddy->job_state = state;
> > > > > (gdb)
> > > > > 66 OBJ_RETAIN(jdata);
> > > > > (gdb)
> > > > > opal_obj_update (inc=1, object=0x100125250)
> > > > > at
> > > > > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:497
> > > > > 497 return opal_atomic_add_32(&(object->obj_reference_count),
> inc);
> > > > > (gdb)
> > > > > opal_atomic_add_32 (addr=0x100125260, delta=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:63
> > > > > 63 oldval = *addr;
> > > > > (gdb)
> > > > > 64 } while (0 == opal_atomic_cmpset_32(addr, oldval, oldval +
> delta));
> > > > > (gdb)
> > > > > opal_atomic_cmpset_32 (addr=0x100125260, oldval=1, newval=2)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:93
> > > > > 93 int32_t ret = newval;
> > > > > (gdb)
> > > > > 95 __asm__ __volatile__("casa [%1] " ASI_P ", %2, %0"
> > > > > (gdb)
> > > > > 98 return (ret == oldval);
> > > > > (gdb)
> > > > > 99 }
> > > > > (gdb)
> > > > > opal_atomic_add_32 (addr=0x100125260, delta=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:65
> > > > > 65 return (oldval + delta);
> > > > > (gdb)
> > > > > 66 }
> > > > > (gdb)
> > > > > orte_state_base_activate_job_state (jdata=0x100125250, state=1)
> > > > > at
> > > > >
> ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:66
> > > > > 66 OBJ_RETAIN(jdata);
> > > > > (gdb)
> > > > > 68 opal_event_set(orte_event_base, &caddy->ev, -1,
> > > > > OPAL_EV_WRITE, s->cbfunc, caddy);
> > > > > (gdb)
> > > > > 69 opal_event_set_priority(&caddy->ev, s->priority);
> > > > > (gdb)
> > > > > 70 opal_event_active(&caddy->ev, OPAL_EV_WRITE, 1);
> > > > > (gdb)
> > > > > 71 return;
> > > > > (gdb)
> > > > > 105 }
> > > > > (gdb)
> > > > > rsh_launch (jdata=0x100125250)
> > > > > at
> > > > >
> ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:883
> > > > > 883 return ORTE_SUCCESS;
> > > > > (gdb)
> > > > > 884 }
> > > > > (gdb)
> > > > > orterun (argc=5, argv=0xffffffff7fffe0d8)
> > > > > at
> ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084
> > > > > 1084 while (orte_event_base_active) {
> > > > > (gdb)
> > > > > 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > > > (gdb)
> > > > > 1084 while (orte_event_base_active) {
> > > > > (gdb)
> > > > > 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > > > (gdb)
> > > > > 1084 while (orte_event_base_active) {
> > > > > (gdb)
> > > > > 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > > > (gdb)
> > > > > #
> > > > > # A fatal error has been detected by the Java Runtime Environment:
> > > > > #
> > > > > # SIGSEGV (0xb) at pc=0xffffffff7ea3c7f0, pid=13080, tid=2
> > > > > #
> > > > > # JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build
> 1.8.0-b132)
> > > > > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode
> solaris-sparc
> > > > > compressed oops)
> > > > > # Problematic frame:
> > > > > # 1084 while (orte_event_base_active) {
> > > > > (gdb)
> > > > > 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> > > > > (gdb)
> > > > > C [libc.so.1+0x3c7f0] strlen+0x50
> > > > > #
> > > > > # Failed to write core dump. Core dumps have been disabled. To enable
> core
> > > > > dumping, try "ulimit -c unlimited" before starting Java again
> > > > > #
> > > > > # An error report file with more information is saved as:
> > > > > #
> /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid13080.log
> > > > > #
> > > > > # If you would like to submit a bug report, please visit:
> > > > > # http://bugreport.sun.com/bugreport/crash.jsp
> > > > > # The crash happened outside the Java Virtual Machine in native code.
> > > > > # See problematic frame for where to report the bug.
> > > > > #
> > > > >
> --------------------------------------------------------------------------
> > > > > mpiexec noticed that process rank 0 with PID 0 on node tyr exited on
> signal 6
> > > > > (Abort).
> > > > >
> --------------------------------------------------------------------------
> > > > > 1084 while (orte_event_base_active) {
> > > > > (gdb)
> > > > > 1089 orte_odls.kill_local_procs(NULL);
> > > > > (gdb)
> > > > >
> > > > >
> > > > > Thank you very much for any help in advance.
> > > > >
> > > > > Kind regards
> > > > >
> > > > > Siegmar
