I "unlimited" my stack space- got a different error, which maybe is a clue..
Im not sure how to vary the rank, like you suggested, so if you have a tip
that would be great.
Here is the new error:
[macmanes:05298] *** Process received signal ***
[macmanes:05298] Signal: Segmentation fault (11)
[macmanes:05298] Signal code: Address not mapped (1)
[macmanes:05298] Failing at address: 0x2ba2e9d9c00c
[macmanes:05298] [ 0] /lib/libpthread.so.0 [0x2ba2b27ce190]
[macmanes:05298] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05298] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05298] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05298] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05298] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05298] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05298] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2ba2b29f9abd]
[macmanes:05298] [ 8] mb [0x410949]
[macmanes:05298] *** End of error message ***
[macmanes:05299] *** Process received signal ***
[macmanes:05299] Signal: Segmentation fault (11)
[macmanes:05299] Signal code: Address not mapped (1)
[macmanes:05299] Failing at address: 0x2b089e31600c
[macmanes:05299] [ 0] /lib/libpthread.so.0 [0x2b0866d48190]
[macmanes:05299] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05299] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05299] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05299] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05299] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05299] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05299] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b0866f73abd]
[macmanes:05299] [ 8] mb [0x410949]
[macmanes:05299] *** End of error message ***
[macmanes:05300] *** Process received signal ***
[macmanes:05300] Signal: Segmentation fault (11)
[macmanes:05300] Signal code: Address not mapped (1)
[macmanes:05300] Failing at address: 0x2b1fa264200c
[macmanes:05300] [ 0] /lib/libpthread.so.0 [0x2b1f6b074190]
[macmanes:05300] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05300] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05300] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05300] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05300] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05300] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05300] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b1f6b29fabd]
[macmanes:05300] [ 8] mb [0x410949]
[macmanes:05300] *** End of error message ***
[macmanes:05301] *** Process received signal ***
[macmanes:05301] Signal: Segmentation fault (11)
[macmanes:05301] Signal code: Address not mapped (1)
[macmanes:05301] Failing at address: 0x2b69f7c3300c
[macmanes:05301] [ 0] /lib/libpthread.so.0 [0x2b69c0665190]
[macmanes:05301] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05301] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05301] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05301] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05301] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05301] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05301] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b69c0890abd]
[macmanes:05301] [ 8] mb [0x410949]
[macmanes:05301] *** End of error message ***
[macmanes:05302] *** Process received signal ***
[macmanes:05302] Signal: Segmentation fault (11)
[macmanes:05302] Signal code: Address not mapped (1)
[macmanes:05302] Failing at address: 0x2b923066b00c
[macmanes:05302] [ 0] /lib/libpthread.so.0 [0x2b91f909d190]
[macmanes:05302] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05302] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05302] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05302] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05302] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05302] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05302] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b91f92c8abd]
[macmanes:05302] [ 8] mb [0x410949]
[macmanes:05302] *** End of error message ***
[macmanes:05303] *** Process received signal ***
[macmanes:05303] Signal: Segmentation fault (11)
[macmanes:05303] Signal code: Address not mapped (1)
[macmanes:05303] Failing at address: 0x2b36bc08c00c
[macmanes:05303] [ 0] /lib/libpthread.so.0 [0x2b3684abe190]
[macmanes:05303] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05303] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05303] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05303] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05303] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05303] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05303] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b3684ce9abd]
[macmanes:05303] [ 8] mb [0x410949]
[macmanes:05303] *** End of error message ***
[macmanes:05304] *** Process received signal ***
[macmanes:05304] Signal: Segmentation fault (11)
[macmanes:05304] Signal code: Address not mapped (1)
[macmanes:05304] Failing at address: 0x2ac048ece00c
[macmanes:05304] [ 0] /lib/libpthread.so.0 [0x2ac011900190]
[macmanes:05304] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05304] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05304] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05304] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05304] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05304] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05304] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2ac011b2babd]
[macmanes:05304] [ 8] mb [0x410949]
[macmanes:05304] *** End of error message ***
[macmanes:05305] *** Process received signal ***
[macmanes:05305] Signal: Segmentation fault (11)
[macmanes:05305] Signal code: Address not mapped (1)
[macmanes:05305] Failing at address: 0x2ad1bd22900c
[macmanes:05305] [ 0] /lib/libpthread.so.0 [0x2ad185c5b190]
[macmanes:05305] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05305] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05305] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05305] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05305] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05305] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05305] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2ad185e86abd]
[macmanes:05305] [ 8] mb [0x410949]
[macmanes:05305] *** End of error message ***
[macmanes:05306] *** Process received signal ***
[macmanes:05306] Signal: Segmentation fault (11)
[macmanes:05306] Signal code: Address not mapped (1)
[macmanes:05306] Failing at address: 0x2aff7d85000c
[macmanes:05306] [ 0] /lib/libpthread.so.0 [0x2aff46282190]
[macmanes:05306] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05306] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05306] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05306] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05306] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05306] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05306] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2aff464adabd]
[macmanes:05306] [ 8] mb [0x410949]
[macmanes:05306] *** End of error message ***
[macmanes:05307] *** Process received signal ***
[macmanes:05307] Signal: Segmentation fault (11)
[macmanes:05307] Signal code: Address not mapped (1)
[macmanes:05307] Failing at address: 0x2b8b4104000c
[macmanes:05307] [ 0] /lib/libpthread.so.0 [0x2b8b09a72190]
[macmanes:05307] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05307] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05307] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05307] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05307] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05307] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05307] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b8b09c9dabd]
[macmanes:05307] [ 8] mb [0x410949]
[macmanes:05307] *** End of error message ***
[macmanes:05308] *** Process received signal ***
[macmanes:05308] Signal: Segmentation fault (11)
[macmanes:05308] Signal code: Address not mapped (1)
[macmanes:05308] Failing at address: 0x2ad33273400c
[macmanes:05308] [ 0] /lib/libpthread.so.0 [0x2ad2fb166190]
[macmanes:05308] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05308] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05308] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05308] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05308] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05308] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05308] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2ad2fb391abd]
[macmanes:05308] [ 8] mb [0x410949]
[macmanes:05308] *** End of error message ***
[macmanes:05309] *** Process received signal ***
[macmanes:05309] Signal: Segmentation fault (11)
[macmanes:05309] Signal code: Address not mapped (1)
[macmanes:05309] Failing at address: 0x2b5e4da9100c
[macmanes:05309] [ 0] /lib/libpthread.so.0 [0x2b5e164c3190]
[macmanes:05309] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05309] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05309] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05309] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05309] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05309] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05309] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b5e166eeabd]
[macmanes:05309] [ 8] mb [0x410949]
[macmanes:05309] *** End of error message ***
[macmanes:05310] *** Process received signal ***
[macmanes:05310] Signal: Segmentation fault (11)
[macmanes:05310] Signal code: Address not mapped (1)
[macmanes:05310] Failing at address: 0x2b7b2a94300c
[macmanes:05311] *** Process received signal ***
[macmanes:05311] Signal: Segmentation fault (11)
[macmanes:05311] Signal code: Address not mapped (1)
[macmanes:05311] Failing at address: 0x2b9e2bf4b00c
[macmanes:05311] [ 0] /lib/libpthread.so.0 [0x2b9df497d190]
[macmanes:05311] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05311] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05311] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05311] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05311] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05311] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05311] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b9df4ba8abd]
[macmanes:05311] [ 8] mb [0x410949]
[macmanes:05311] *** End of error message ***
[macmanes:05312] *** Process received signal ***
[macmanes:05312] Signal: Segmentation fault (11)
[macmanes:05312] Signal code: Address not mapped (1)
[macmanes:05312] Failing at address: 0x2b756bf1b00c
[macmanes:05312] [ 0] /lib/libpthread.so.0 [0x2b753494d190]
[macmanes:05312] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05312] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05312] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05312] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05312] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05312] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05312] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b7534b78abd]
[macmanes:05312] [ 8] mb [0x410949]
[macmanes:05312] *** End of error message ***
Defining charset called gene1000
[macmanes:05310] [ 0] /lib/libpthread.so.0 [0x2b7af3375190]
[macmanes:05310] [ 1] mb(DoCharset+0x187) [0x41d9a7]
[macmanes:05310] [ 2] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05310] [ 3] mb(DoExecute+0x67f) [0x42f81f]
[macmanes:05310] [ 4] mb(ParseCommand+0x2b2) [0x42edc2]
[macmanes:05310] [ 5] mb(CommandLine+0x17e) [0x4137de]
[macmanes:05310] [ 6] mb(main+0x82) [0x413ad2]
[macmanes:05310] [ 7] /lib/libc.so.6(__libc_start_main+0xfd)
[0x2b7af35a0abd]
[macmanes:05310] [ 8] mb [0x410949]
[macmanes:05310] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 9 with PID 5307 on node macmanes exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
2 total processes killed (some possibly by mpirun during cleanup)
macmanes@macmanes:~/mrbayes$
_________________________________
Matthew MacManes
PhD Candidate
University of California- Berkeley
Museum of Vertebrate Zoology
Phone: 510-495-5833
Lab Website: http://ib.berkeley.edu/labs/lacey
Personal Website: http://macmanes.com/
On Thu, Mar 11, 2010 at 07:42, Peter Kjellstrom <c...@nsc.liu.se> wrote:
> On Thursday 11 March 2010, Matthew MacManes wrote:
> > Can anybody tell me if this is an error associated with openmpi, versus
> an
> > issue with the program I am running (MRBAYES,
> > https://sourceforge.net/projects/mrbayes/)
> >
> > We are trying to run a large simulated dataset using 1,000,000 bases
> > divided up into 1000 genes, 5 taxa.. An error is occurring, but we are
> not
> > sure why. We are using the MPI version of MRBAYES v3.2-cvs on a linux
> > 16core 24GB RAM machine. It does not appear as if the program runs out of
> > memory (max memory usage is 13gb). Maybe this is an OpenMPI problem and
> > not related to MrBayes...
> >
> > See snippet of error message below. Can anybody give me any hints about
> the
> > source of the problem?
> >
> > I am using OPENMPI version 1.4.1.
> >
> > ...
> > Defining charset called gene997
> > Defining charset called gene998
> > Defining charset called gene999
> > Defining charset called gene1000
> > Defining partition called Genes
> > [macmanes:02546] *** Process received signal ***
> > [macmanes:02546] Signal: Segmentation fault (11)
> > [macmanes:02546] Signal code: Address not mapped (1)
> > [macmanes:02546] Failing at address: (nil)
> > [macmanes:02546] [ 0] /lib/libpthread.so.0 [0x7ffd0f322190]
> > [macmanes:02546] *** End of error message ***
> >
> --------------------------------------------------------------------------
> > mpirun noticed that process rank 13 with PID 2546 on node macmanes exited
> > on signal 11 (Segmentation fault).
>
> On of the ranks got a "Segmentation fault". This would typically indicate a
> problem with the app not the MPI. Maybe you ran out of stack space?
> (ulimit -s).
>
> Have you tried a different/lower number of ranks?
>
> /Peter
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
