the JVM sets its own signal handlers, and it is important openmpi dones not override them. this is what previously happened with PSM (infinipath) but this has been solved since. you might be linking with a third party library that hijacks signal handlers and cause the crash (which would explain why I cannot reproduce the issue)
the master branch has a revamped memory patcher (compared to v2.x or v1.10), and that could have some bad interactions with the JVM, so you might also give v2.x a try Cheers, Gilles On Friday, July 8, 2016, Gundram Leifert <gundram.leif...@uni-rostock.de> wrote: > You made the best of it... thanks a lot! > > Whithout MPI it runs. > Just adding MPI.init() causes the crash! > > maybe I installed something wrong... > > install newest automake, autoconf, m4, libtoolize in right order and same > prefix > check out ompi, > autogen > configure with same prefix, pointing to the same jdk, I later use > make > make install > > I will test some different configurations of ./configure... > > > On 07/08/2016 01:40 PM, Gilles Gouaillardet wrote: > > I am running out of ideas ... > > what if you do not run within slurm ? > what if you do not use '-cp executor.jar' > or what if you configure without --disable-dlopen --disable-mca-dso ? > > if you > mpirun -np 1 ... > then MPI_Bcast and MPI_Barrier are basically no-op, so it is really weird > your program is still crashing. an other test is to comment out MPI_Bcast > and MPI_Barrier and try again with -np 1 > > Cheers, > > Gilles > > On Friday, July 8, 2016, Gundram Leifert < > <javascript:_e(%7B%7D,'cvml','gundram.leif...@uni-rostock.de');> > gundram.leif...@uni-rostock.de > <javascript:_e(%7B%7D,'cvml','gundram.leif...@uni-rostock.de');>> wrote: > >> In any cases the same error. >> this is my code: >> >> salloc -n 3 >> export IPATH_NO_BACKTRACE >> ulimit -s 10240 >> mpirun -np 3 java -cp executor.jar >> de.uros.citlab.executor.test.TestSendBigFiles2 >> >> >> also for 1 or two cores, the process crashes. >> >> >> On 07/08/2016 12:32 PM, Gilles Gouaillardet wrote: >> >> you can try >> export IPATH_NO_BACKTRACE >> before invoking mpirun (that should not be needed though) >> >> an other test is to >> ulimit -s 10240 >> before invoking mpirun. >> >> btw, do you use mpirun or srun ? >> >> can you reproduce the crash with 1 or 2 tasks ? >> >> Cheers, >> >> Gilles >> >> On Friday, July 8, 2016, Gundram Leifert <gundram.leif...@uni-rostock.de >> <javascript:_e(%7B%7D,'cvml','gundram.leif...@uni-rostock.de');>> wrote: >> >>> Hello, >>> >>> configure: >>> ./configure --enable-mpi-java --with-jdk-dir=/home/gl069/bin/jdk1.7.0_25 >>> --disable-dlopen --disable-mca-dso >>> >>> >>> 1 node with 3 cores. I use SLURM to allocate one node. I changed --mem, >>> but it has no effect. >>> salloc -n 3 >>> >>> >>> core file size (blocks, -c) 0 >>> data seg size (kbytes, -d) unlimited >>> scheduling priority (-e) 0 >>> file size (blocks, -f) unlimited >>> pending signals (-i) 256564 >>> max locked memory (kbytes, -l) unlimited >>> max memory size (kbytes, -m) unlimited >>> open files (-n) 100000 >>> pipe size (512 bytes, -p) 8 >>> POSIX message queues (bytes, -q) 819200 >>> real-time priority (-r) 0 >>> stack size (kbytes, -s) unlimited >>> cpu time (seconds, -t) unlimited >>> max user processes (-u) 4096 >>> virtual memory (kbytes, -v) unlimited >>> file locks (-x) unlimited >>> >>> uname -a >>> Linux titan01.service 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 >>> 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >>> >>> cat /etc/system-release >>> CentOS Linux release 7.2.1511 (Core) >>> >>> what else do you need? >>> >>> Cheers, Gundram >>> >>> On 07/07/2016 10:05 AM, Gilles Gouaillardet wrote: >>> >>> Gundram, >>> >>> >>> can you please provide more information on your environment : >>> >>> - configure command line >>> >>> - OS >>> >>> - memory available >>> >>> - ulimit -a >>> >>> - number of nodes >>> >>> - number of tasks used >>> >>> - interconnect used (if any) >>> >>> - batch manager (if any) >>> >>> >>> Cheers, >>> >>> >>> Gilles >>> On 7/7/2016 4:17 PM, Gundram Leifert wrote: >>> >>> Hello Gilles, >>> >>> I tried you code and it crashes after 3-15 iterations (see (1)). It is >>> always the same error (only the "94" varies). >>> >>> Meanwhile I think Java and MPI use the same memory because when I delete >>> the hash-call, the program runs sometimes more than 9k iterations. >>> When it crashes, there are different lines (see (2) and (3)). The >>> crashes also occurs on rank 0. >>> >>> ##### (1)##### >>> # Problematic frame: >>> # J 94 C2 de.uros.citlab.executor.test.TestSendBigFiles2.hashcode([BI)I >>> (42 bytes) @ 0x00002b03242dc9c4 [0x00002b03242dc860+0x164] >>> >>> #####(2)##### >>> # Problematic frame: >>> # V [libjvm.so+0x68d0f6] >>> JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, >>> Thread*)+0xb6 >>> >>> #####(3)##### >>> # Problematic frame: >>> # V [libjvm.so+0x4183bf] >>> ThreadInVMfromNative::ThreadInVMfromNative(JavaThread*)+0x4f >>> >>> Any more idea? >>> >>> On 07/07/2016 03:00 AM, Gilles Gouaillardet wrote: >>> >>> Gundram, >>> >>> >>> fwiw, i cannot reproduce the issue on my box >>> >>> - centos 7 >>> >>> - java version "1.8.0_71" >>> Java(TM) SE Runtime Environment (build 1.8.0_71-b15) >>> Java HotSpot(TM) 64-Bit Server VM (build 25.71-b15, mixed mode) >>> >>> >>> i noticed on non zero rank saveMem is allocated at each iteration. >>> ideally, the garbage collector can take care of that and this should not >>> be an issue. >>> >>> would you mind giving the attached file a try ? >>> >>> Cheers, >>> >>> Gilles >>> >>> On 7/7/2016 7:41 AM, Gilles Gouaillardet wrote: >>> >>> I will have a look at it today >>> >>> how did you configure OpenMPI ? >>> >>> Cheers, >>> >>> Gilles >>> >>> On Thursday, July 7, 2016, Gundram Leifert < >>> gundram.leif...@uni-rostock.de> wrote: >>> >>>> Hello Giles, >>>> >>>> thank you for your hints! I did 3 changes, unfortunately the same error >>>> occures: >>>> >>>> update ompi: >>>> commit ae8444682f0a7aa158caea08800542ce9874455e >>>> Author: Ralph Castain <r...@open-mpi.org> >>>> <javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');> >>>> Date: Tue Jul 5 20:07:16 2016 -0700 >>>> >>>> update java: >>>> java version "1.8.0_92" >>>> Java(TM) SE Runtime Environment (build 1.8.0_92-b14) >>>> Java HotSpot(TM) Server VM (build 25.92-b14, mixed mode) >>>> >>>> delete hashcode-lines. >>>> >>>> Now I get this error message - to 100%, after different number of >>>> iterations (15-300): >>>> >>>> 0/ 3:length = 100000000 >>>> 0/ 3:bcast length done (length = 100000000) >>>> 1/ 3:bcast length done (length = 100000000) >>>> 2/ 3:bcast length done (length = 100000000) >>>> # >>>> # A fatal error has been detected by the Java Runtime Environment: >>>> # >>>> # SIGSEGV (0xb) at pc=0x00002b3d022fcd24, pid=16578, >>>> tid=0x00002b3d29716700 >>>> # >>>> # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build >>>> 1.8.0_92-b14) >>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode >>>> linux-amd64 compressed oops) >>>> # Problematic frame: >>>> # V [libjvm.so+0x414d24] ciEnv::get_field_by_index(ciInstanceKlass*, >>>> int)+0x94 >>>> # >>>> # Failed to write core dump. Core dumps have been disabled. To enable >>>> core dumping, try "ulimit -c unlimited" before starting Java again >>>> # >>>> # An error report file with more information is saved as: >>>> # /home/gl069/ompi/bin/executor/hs_err_pid16578.log >>>> # >>>> # Compiler replay data is saved as: >>>> # /home/gl069/ompi/bin/executor/replay_pid16578.log >>>> # >>>> # If you would like to submit a bug report, please visit: >>>> # http://bugreport.java.com/bugreport/crash.jsp >>>> # >>>> [titan01:16578] *** Process received signal *** >>>> [titan01:16578] Signal: Aborted (6) >>>> [titan01:16578] Signal code: (-6) >>>> [titan01:16578] [ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x2b3d01500100] >>>> [titan01:16578] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2b3d01b5c5f7] >>>> [titan01:16578] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2b3d01b5dce8] >>>> [titan01:16578] [ 3] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x91e605)[0x2b3d02806605] >>>> [titan01:16578] [ 4] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0xabda63)[0x2b3d029a5a63] >>>> [titan01:16578] [ 5] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x14f)[0x2b3d0280be2f] >>>> [titan01:16578] [ 6] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x91a5c3)[0x2b3d028025c3] >>>> [titan01:16578] [ 7] /usr/lib64/libc.so.6(+0x35670)[0x2b3d01b5c670] >>>> [titan01:16578] [ 8] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x414d24)[0x2b3d022fcd24] >>>> [titan01:16578] [ 9] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x43c5ae)[0x2b3d023245ae] >>>> [titan01:16578] [10] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x369ade)[0x2b3d02251ade] >>>> [titan01:16578] [11] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36eda0)[0x2b3d02256da0] >>>> [titan01:16578] [12] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37091b)[0x2b3d0225891b] >>>> [titan01:16578] [13] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3712b6)[0x2b3d022592b6] >>>> [titan01:16578] [14] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36d2cf)[0x2b3d022552cf] >>>> [titan01:16578] [15] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36e412)[0x2b3d02256412] >>>> [titan01:16578] [16] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36ed8d)[0x2b3d02256d8d] >>>> [titan01:16578] [17] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37091b)[0x2b3d0225891b] >>>> [titan01:16578] [18] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3712b6)[0x2b3d022592b6] >>>> [titan01:16578] [19] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36d2cf)[0x2b3d022552cf] >>>> [titan01:16578] [20] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36e412)[0x2b3d02256412] >>>> [titan01:16578] [21] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36ed8d)[0x2b3d02256d8d] >>>> [titan01:16578] [22] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3708c2)[0x2b3d022588c2] >>>> [titan01:16578] [23] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3724e7)[0x2b3d0225a4e7] >>>> [titan01:16578] [24] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37a817)[0x2b3d02262817] >>>> [titan01:16578] [25] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37a92f)[0x2b3d0226292f] >>>> [titan01:16578] [26] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x358edb)[0x2b3d02240edb] >>>> [titan01:16578] [27] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x35929e)[0x2b3d0224129e] >>>> [titan01:16578] [28] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3593ce)[0x2b3d022413ce] >>>> [titan01:16578] [29] >>>> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x35973e)[0x2b3d0224173e] >>>> [titan01:16578] *** End of error message *** >>>> ------------------------------------------------------- >>>> Primary job terminated normally, but 1 process returned >>>> a non-zero exit code. Per user-direction, the job has been aborted. >>>> ------------------------------------------------------- >>>> >>>> -------------------------------------------------------------------------- >>>> mpirun noticed that process rank 2 with PID 0 on node titan01 exited on >>>> signal 6 (Aborted). >>>> >>>> -------------------------------------------------------------------------- >>>> >>>> I don't know if it is a problem of java or ompi - but the last years, >>>> java worked with no problems on my machine... >>>> >>>> Thank you for your tips in advance! >>>> Gundram >>>> >>>> On 07/06/2016 03:10 PM, Gilles Gouaillardet wrote: >>>> >>>> Note a race condition in MPI_Init has been fixed yesterday in the >>>> master. >>>> can you please update your OpenMPI and try again ? >>>> >>>> hopefully the hang will disappear. >>>> >>>> Can you reproduce the crash with a simpler (and ideally deterministic) >>>> version of your program. >>>> the crash occurs in hashcode, and this makes little sense to me. can >>>> you also update your jdk ? >>>> >>>> Cheers, >>>> >>>> Gilles >>>> >>>> On Wednesday, July 6, 2016, Gundram Leifert < >>>> gundram.leif...@uni-rostock.de >>>> <javascript:_e(%7B%7D,'cvml','gundram.leif...@uni-rostock.de');>> >>>> wrote: >>>> >>>>> Hello Jason, >>>>> >>>>> thanks for your response! I thing it is another problem. I try to send >>>>> 100MB bytes. So there are not many tries (between 10 and 30). I realized >>>>> that the execution of this code can result 3 different errors: >>>>> >>>>> 1. most often the posted error message occures. >>>>> >>>>> 2. in <10% the cases i have a live lock. I can see 3 java-processes, >>>>> one with 200% and two with 100% processor utilization. After ~15 minutes >>>>> without new system outputs this error occurs. >>>>> >>>>> >>>>> [thread 47499823949568 also had an error] >>>>> # A fatal error has been detected by the Java Runtime Environment: >>>>> # >>>>> # Internal Error (safepoint.cpp:317), pid=24256, tid=47500347131648 >>>>> # guarantee(PageArmed == 0) failed: invariant >>>>> # >>>>> # JRE version: 7.0_25-b15 >>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode >>>>> linux-amd64 compressed oops) >>>>> # Failed to write core dump. Core dumps have been disabled. To enable >>>>> core dumping, try "ulimit -c unlimited" before starting Java again >>>>> # >>>>> # An error report file with more information is saved as: >>>>> # /home/gl069/ompi/bin/executor/hs_err_pid24256.log >>>>> # >>>>> # If you would like to submit a bug report, please visit: >>>>> # <http://bugreport.sun.com/bugreport/crash.jsp> >>>>> http://bugreport.sun.com/bugreport/crash.jsp >>>>> # >>>>> [titan01:24256] *** Process received signal *** >>>>> [titan01:24256] Signal: Aborted (6) >>>>> [titan01:24256] Signal code: (-6) >>>>> [titan01:24256] [ 0] >>>>> /usr/lib64/libpthread.so.0(+0xf100)[0x2b336a324100] >>>>> [titan01:24256] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2b336a9815f7] >>>>> [titan01:24256] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2b336a982ce8] >>>>> [titan01:24256] [ 3] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x742ac5)[0x2b336b44fac5] >>>>> [titan01:24256] [ 4] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a2137)[0x2b336b5af137] >>>>> [titan01:24256] [ 5] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x407262)[0x2b336b114262] >>>>> [titan01:24256] [ 6] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x7c6c34)[0x2b336b4d3c34] >>>>> [titan01:24256] [ 7] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a9c17)[0x2b336b5b6c17] >>>>> [titan01:24256] [ 8] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8aa2c0)[0x2b336b5b72c0] >>>>> [titan01:24256] [ 9] >>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x744270)[0x2b336b451270] >>>>> [titan01:24256] [10] >>>>> /usr/lib64/libpthread.so.0(+0x7dc5)[0x2b336a31cdc5] >>>>> [titan01:24256] [11] /usr/lib64/libc.so.6(clone+0x6d)[0x2b336aa4228d] >>>>> [titan01:24256] *** End of error message *** >>>>> ------------------------------------------------------- >>>>> Primary job terminated normally, but 1 process returned >>>>> a non-zero exit code. Per user-direction, the job has been aborted. >>>>> ------------------------------------------------------- >>>>> >>>>> -------------------------------------------------------------------------- >>>>> mpirun noticed that process rank 0 with PID 0 on node titan01 exited >>>>> on signal 6 (Aborted). >>>>> >>>>> -------------------------------------------------------------------------- >>>>> >>>>> >>>>> 3. in <10% the cases i have a dead lock while MPI.init. This stays for >>>>> more than 15 minutes without returning with an error message... >>>>> >>>>> Can I enable some debug-flags to see what happens on C / OpenMPI side? >>>>> >>>>> Thanks in advance for your help! >>>>> Gundram Leifert >>>>> >>>>> >>>>> On 07/05/2016 06:05 PM, Jason Maldonis wrote: >>>>> >>>>> After reading your thread looks like it may be related to an issue I >>>>> had a few weeks ago (I'm a novice though). Maybe my thread will be of >>>>> help: >>>>> <https://www.open-mpi.org/community/lists/users/2016/06/29425.php> >>>>> https://www.open-mpi.org/community/lists/users/2016/06/29425.php >>>>> >>>>> When you say "After a specific number of repetitions the process >>>>> either hangs up or returns with a SIGSEGV." does you mean that a single >>>>> call hangs, or that at some point during the for loop a call hangs? If you >>>>> mean the latter, then it might relate to my issue. Otherwise my thread >>>>> probably won't be helpful. >>>>> >>>>> Jason Maldonis >>>>> Research Assistant of Professor Paul Voyles >>>>> Materials Science Grad Student >>>>> University of Wisconsin, Madison >>>>> 1509 University Ave, Rm M142 >>>>> Madison, WI 53706 >>>>> maldo...@wisc.edu <javascript:_e(%7B%7D,'cvml','maldo...@wisc.edu');> >>>>> 608-295-5532 >>>>> >>>>> On Tue, Jul 5, 2016 at 9:58 AM, Gundram Leifert < >>>>> gundram.leif...@uni-rostock.de >>>>> <javascript:_e(%7B%7D,'cvml','gundram.leif...@uni-rostock.de');>> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I try to send many byte-arrays via broadcast. After a specific number >>>>>> of repetitions the process either hangs up or returns with a SIGSEGV. >>>>>> Does >>>>>> any one can help me solving the problem: >>>>>> >>>>>> ########## The code: >>>>>> >>>>>> import java.util.Random; >>>>>> import mpi.*; >>>>>> >>>>>> public class TestSendBigFiles { >>>>>> >>>>>> public static void log(String msg) { >>>>>> try { >>>>>> System.err.println(String.format("%2d/%2d:%s", >>>>>> MPI.COMM_WORLD.getRank(), MPI.COMM_WORLD.getSize(), msg)); >>>>>> } catch (MPIException ex) { >>>>>> System.err.println(String.format("%2s/%2s:%s", "?", "?", >>>>>> msg)); >>>>>> } >>>>>> } >>>>>> >>>>>> private static int hashcode(byte[] bytearray) { >>>>>> if (bytearray == null) { >>>>>> return 0; >>>>>> } >>>>>> int hash = 39; >>>>>> for (int i = 0; i < bytearray.length; i++) { >>>>>> byte b = bytearray[i]; >>>>>> hash = hash * 7 + (int) b; >>>>>> } >>>>>> return hash; >>>>>> } >>>>>> >>>>>> public static void main(String args[]) throws MPIException { >>>>>> log("start main"); >>>>>> MPI.Init(args); >>>>>> try { >>>>>> log("initialized done"); >>>>>> byte[] saveMem = new byte[100000000]; >>>>>> MPI.COMM_WORLD.barrier(); >>>>>> Random r = new Random(); >>>>>> r.nextBytes(saveMem); >>>>>> if (MPI.COMM_WORLD.getRank() == 0) { >>>>>> for (int i = 0; i < 1000; i++) { >>>>>> saveMem[r.nextInt(saveMem.length)]++; >>>>>> log("i = " + i); >>>>>> int[] lengthData = new int[]{saveMem.length}; >>>>>> log("object hash = " + hashcode(saveMem)); >>>>>> log("length = " + lengthData[0]); >>>>>> MPI.COMM_WORLD.bcast(lengthData, 1, MPI.INT, 0); >>>>>> log("bcast length done (length = " + >>>>>> lengthData[0] + ")"); >>>>>> MPI.COMM_WORLD.barrier(); >>>>>> MPI.COMM_WORLD.bcast(saveMem, lengthData[0], >>>>>> MPI.BYTE, 0); >>>>>> log("bcast data done"); >>>>>> MPI.COMM_WORLD.barrier(); >>>>>> } >>>>>> MPI.COMM_WORLD.bcast(new int[]{0}, 1, MPI.INT, 0); >>>>>> } else { >>>>>> while (true) { >>>>>> int[] lengthData = new int[1]; >>>>>> MPI.COMM_WORLD.bcast(lengthData, 1, MPI.INT, 0); >>>>>> log("bcast length done (length = " + >>>>>> lengthData[0] + ")"); >>>>>> if (lengthData[0] == 0) { >>>>>> break; >>>>>> } >>>>>> MPI.COMM_WORLD.barrier(); >>>>>> saveMem = new byte[lengthData[0]]; >>>>>> MPI.COMM_WORLD.bcast(saveMem, saveMem.length, >>>>>> MPI.BYTE, 0); >>>>>> log("bcast data done"); >>>>>> MPI.COMM_WORLD.barrier(); >>>>>> log("object hash = " + hashcode(saveMem)); >>>>>> } >>>>>> } >>>>>> MPI.COMM_WORLD.barrier(); >>>>>> } catch (MPIException ex) { >>>>>> System.out.println("caugth error." + ex); >>>>>> log(ex.getMessage()); >>>>>> } catch (RuntimeException ex) { >>>>>> System.out.println("caugth error." + ex); >>>>>> log(ex.getMessage()); >>>>>> } finally { >>>>>> MPI.Finalize(); >>>>>> } >>>>>> >>>>>> } >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> ############ The Error (if it does not just hang up): >>>>>> >>>>>> # >>>>>> # A fatal error has been detected by the Java Runtime Environment: >>>>>> # >>>>>> # SIGSEGV (0xb) at pc=0x00002b7e9c86e3a1, pid=1172, >>>>>> tid=47822674495232 >>>>>> # >>>>>> # >>>>>> # A fatal error has been detected by the Java Runtime Environment: >>>>>> # JRE version: 7.0_25-b15 >>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode >>>>>> linux-amd64 compressed oops) >>>>>> # Problematic frame: >>>>>> # # >>>>>> # SIGSEGV (0xb) at pc=0x00002af69c0693a1, pid=1173, >>>>>> tid=47238546896640 >>>>>> # >>>>>> # JRE version: 7.0_25-b15 >>>>>> J de.uros.citlab.executor.test.TestSendBigFiles.hashcode([B)I >>>>>> # >>>>>> # Failed to write core dump. Core dumps have been disabled. To enable >>>>>> core dumping, try "ulimit -c unlimited" before starting Java again >>>>>> # >>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode >>>>>> linux-amd64 compressed oops) >>>>>> # Problematic frame: >>>>>> # J de.uros.citlab.executor.test.TestSendBigFiles.hashcode([B)I >>>>>> # >>>>>> # Failed to write core dump. Core dumps have been disabled. To enable >>>>>> core dumping, try "ulimit -c unlimited" before starting Java again >>>>>> # >>>>>> # An error report file with more information is saved as: >>>>>> # /home/gl069/ompi/bin/executor/hs_err_pid1172.log >>>>>> # An error report file with more information is saved as: >>>>>> # /home/gl069/ompi/bin/executor/hs_err_pid1173.log >>>>>> # >>>>>> # If you would like to submit a bug report, please visit: >>>>>> # <http://bugreport.sun.com/bugreport/crash.jsp> >>>>>> http://bugreport.sun.com/bugreport/crash.jsp >>>>>> # >>>>>> # >>>>>> # If you would like to submit a bug report, please visit: >>>>>> # <http://bugreport.sun.com/bugreport/crash.jsp> >>>>>> http://bugreport.sun.com/bugreport/crash.jsp >>>>>> # >>>>>> [titan01:01172] *** Process received signal *** >>>>>> [titan01:01172] Signal: Aborted (6) >>>>>> [titan01:01172] Signal code: (-6) >>>>>> [titan01:01173] *** Process received signal *** >>>>>> [titan01:01173] Signal: Aborted (6) >>>>>> [titan01:01173] Signal code: (-6) >>>>>> [titan01:01172] [ 0] >>>>>> /usr/lib64/libpthread.so.0(+0xf100)[0x2b7e9596a100] >>>>>> [titan01:01172] [ 1] >>>>>> /usr/lib64/libc.so.6(gsignal+0x37)[0x2b7e95fc75f7] >>>>>> [titan01:01172] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2b7e95fc8ce8] >>>>>> [titan01:01172] [ 3] >>>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x742ac5)[0x2b7e96a95ac5] >>>>>> [titan01:01172] [ 4] >>>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a2137)[0x2b7e96bf5137] >>>>>> [titan01:01172] [ 5] >>>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x140)[0x2b7e96a995e0] >>>>>> [titan01:01172] [ 6] [titan01:01173] [ 0] >>>>>> /usr/lib64/libpthread.so.0(+0xf100)[0x2af694ded100] >>>>>> [titan01:01173] [ 1] /usr/lib64/libc.so.6(+0x35670)[0x2b7e95fc7670] >>>>>> [titan01:01172] [ 7] [0x2b7e9c86e3a1] >>>>>> [titan01:01172] *** End of error message *** >>>>>> /usr/lib64/libc.so.6(gsignal+0x37)[0x2af69544a5f7] >>>>>> [titan01:01173] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2af69544bce8] >>>>>> [titan01:01173] [ 3] >>>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x742ac5)[0x2af695f18ac5] >>>>>> [titan01:01173] [ 4] >>>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a2137)[0x2af696078137] >>>>>> [titan01:01173] [ 5] >>>>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x140)[0x2af695f1c5e0] >>>>>> [titan01:01173] [ 6] /usr/lib64/libc.so.6(+0x35670)[0x2af69544a670] >>>>>> [titan01:01173] [ 7] [0x2af69c0693a1] >>>>>> [titan01:01173] *** End of error message *** >>>>>> ------------------------------------------------------- >>>>>> Primary job terminated normally, but 1 process returned >>>>>> a non-zero exit code. Per user-direction, the job has been aborted. >>>>>> ------------------------------------------------------- >>>>>> >>>>>> -------------------------------------------------------------------------- >>>>>> mpirun noticed that process rank 1 with PID 0 on node titan01 exited >>>>>> on signal 6 (Aborted). >>>>>> >>>>>> >>>>>> ########CONFIGURATION: >>>>>> I used the ompi master sources from github: >>>>>> commit 267821f0dd405b5f4370017a287d9a49f92e734a >>>>>> Author: Gilles Gouaillardet <gil...@rist.or.jp >>>>>> <javascript:_e(%7B%7D,'cvml','gil...@rist.or.jp');>> >>>>>> Date: Tue Jul 5 13:47:50 2016 +0900 >>>>>> >>>>>> ./configure --enable-mpi-java >>>>>> --with-jdk-dir=/home/gl069/bin/jdk1.7.0_25 --disable-dlopen >>>>>> --disable-mca-dso >>>>>> >>>>>> Thanks a lot for your help! >>>>>> Gundram >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');> >>>>>> Subscription: <https://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>> https://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> <http://www.open-mpi.org/community/lists/users/2016/07/29584.php> >>>>>> http://www.open-mpi.org/community/lists/users/2016/07/29584.php >>>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing listus...@open-mpi.org >>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2016/07/29585.php >>>>> >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> users mailing listus...@open-mpi.org >>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2016/07/29587.php >>>> >>>> >>>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/07/29589.php >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/07/29590.php >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/07/29592.php >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/07/29593.php >>> >>> >>> >> >> _______________________________________________ >> users mailing listus...@open-mpi.org >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/07/29601.php >> >> >> > > _______________________________________________ > users mailing listus...@open-mpi.org > <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/07/29603.php > > >
