Before v1.10, the default is ROMIO, and you can force OMPIO with
mpirun --mca io ompio ...
From v2, the default is OMPIO (unless you are running on lustre iirc),
and you can force ROMIO with
mpirun --mca io ^ompio ...
maybe that can help for the time being
Cheers,
Gilles
----- Original Message -----
Hello,
Which MPI Version are you using?
This looks for me like it triggers https://github.com/open-mpi/ompi/issues/2399
You can check if you are running into this problem by playing around
with the mca_io_ompio_cycle_buffer_size parameter.
Best
Christoph Niethammer
--
Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart
Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer
----- Original Message -----
From: "Nils Moschuering" <ni...@ipp.mpg.de>
To: "Open MPI Users" <users@lists.open-mpi.org>
Sent: Friday, April 28, 2017 12:51:50 PM
Subject: [OMPI users] MPI I/O gives undefined behavior if the amount
of bytes described by a filetype reaches 2^32
Dear OpenMPI Mailing List,
I have a problem with MPI I/O running on more than 1 rank using very
large filetypes. In order to reproduce the problem please take advantage
of the attached program "mpi_io_test.c". After compilation it should be
run on 2 nodes.
The program will do the following for a variety of different
parameters:
1. Create an elementary datatype (commonly refered to as etype in the
MPI Standard) of a specific size given by the parameter bsize (in
multiple of bytes). This datatype is called blk_filetype .
2. Create a complex filetype, which is different for each rank. This
filetype divides the file into a number of blocks given by parameter nr_
blocks of size bsize . Each rank only gets access to a subarray
containing
nr_blocks_per_rank = nr_blocks / size
blocks (where size is the number of participating ranks). The
respective subarray of each rank starts at
rank * nr_blocks_per_rank
This guarantees that the regions of the different ranks don't overlap.
The resulting datatype is called full_filetype .
3. Allocate enough memory on each rank, in order to be able to write a
whole block.
4. Fill the allocated memory with the rank number to be able to check
the resulting file for correctness.
5. Open a file named fname and set the view using the previously
generated blk_filetype and full_filetype .
6. Write one block on each rank, using the collective routine.
7. Clean up.
The above will be repeated for different values of bsize and nr_blocks
. Please note, that there is no overflow of the used basic dataype int .
The output is verified using
hexdump fname
which performs a hexdump of the file. This tool collects consecutive
equal lines in a file into one output line. The resulting output of a
call to hexdump is given by a structure comparable to the following
00000000 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |.............
...|
*
1f400000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............
...|
*
3e800000 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 |.............
...|
*
5dc00000
This example is to be read in the following manner:
-From byte 00000000 to 1f400000 (which is equal to 500 Mib) the file
contains the value 01 in each byte.
-From byte 1f400000 to 3e800000 (which is equal to 1000 Mib) the file
contains the value 00 in each byte.
-From byte 3e800000 to 5dc00000 (which is equal to 1500 Mib) the file
contains the value 02 in each byte.
-The file ends here.
This is the correct output of the above outlined program with
parameters
bsize=500*1023*1024
nr_blocks=4
running on 2 ranks. The attached file contains a lot of tests for
different cases. These were made to pinpoint the source of the problem
and to exclude different other, potentially important, factors.
I deem an output wrong if it doesn't follow from the parameters or if
the program crashes on execution.
The only difference between OpenMPI and Intel MPI, according to my
tests, is in the different behavior on error: OpenMPI will mostly write
wrong data but won't crash, whereas Intel MPI mostly crashes.
The tests and their results are defined in comments in the source.
The final conclusions, I derive from the tests, are the following:
1. If the filetype used in the view is set in a way that it describes
an amount of bytes equaling or exceeding 2^32 = 4Gib the code produces
wrong output. For values slightly smaller (the second example with fname
="test_8_blocks" uses a total filetype size of 4000 MiB which is smaller
than 4Gib) the code works as expected.
2. The act of actually writing the described regions is not important.
When the filetype describes an area >= 4Gib but only writes to regions
much smaller than that, the code still produces undefined behavior (
please refer to the 6th example with fname="test_too_large_blocks" in
order to see an example).
3. It doesn't matter if the block size or the amount of blocks pushes
the filetype over the 4 Gib (refer to the 5th and 6th example, with
filenames "test_16_blocks" and "test_too_large_blocks" respectively).
4. If the binary is launched using only one rank, the output is always
as expected (refer to the 3rd and 4th example, with filenames "test_too_
large_blocks_single" and "test_too_large_blocks_single_even_larger",
respectively).
There are, of course, many other things one could test.
It seems that the implementations use 32bit integer variables to
compute the byte composition inside the filetype. Since the filetype is
defined using two 32bit integer variables, this can easily lead to
integer overflows if the user supplies large values. It seems that no
implementation expects this problem and therefore they do not act
gracefully on its occurrence.
I looked at [ https://software.intel.com/en-us/node/528914 | ILP64 ]
Support, but it only adapts the function parameters and not the
internally used variables and it is also not available for C.
I looked at [
https://www.gnu.org/software/libc/manual/html_node/Program-Error-Signals.html#Program%20Error%20Signals
| integer
overflow ] (FPE_INTOVF_TRAP) trapping, which could help to
verify the source of the problem, but it doesn't seem to be possible for
C. Intel does [
https://software.intel.com/en-us/forums/intel-c-compiler/topic/306156
| not ] offer any built-in integer overflow trapping.
There are ways to circumvent this problem for most cases. It is only
unavoidable if the logic of a program contains complex, non-repeating
data structures with sizes of over (or equal) 4GiB. Even then, one could
split up the filetype and use a different displacement in two distinct
write calls.
Still, this problem violates the standard as it produces undefined
behavior even when using the API in a consistent way. The implementation
should at least provide a warning for the user, but should ideally use
larger datatypes in the filetype computations. When a user stumbles on
this problem, he will have a hard time to debug it.
Thank you very much for reading everything ;)
Kind Regards,
Nils
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
