Is the problem independent of the the number of MPI processes? (You
suggest this is the case.)
If so, does this problem show up even with np=1 (a single MPI process)?
If so, does the problem show up even if you turn MPI off?
If so, the problem would seem to be unrelated to the MPI implementation
(but possibly related to code that was introduced to parallelize).
Julia He wrote:
The OpenMPI version is
[julia.he@bob bin]$ mpirun --version
mpirun (Open MPI) 1.2.8
Report bugs to http://www.open-mpi.org/community/help/
The platform is
[julia.he@bob bin]$ uname -a
Linux bob.csi.cuny.edu 2.6.18-92.1.13.el5 #1 SMP Wed Sep 24 19:32:05
EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
The my_sub is a modification of Radiative Transfer code 6S.
http://6s.ltdri.org/ The 6S code takes angles, atmospheric conditions,
altitude, etc as inputs, and it returns top of the atmosphere
reflectance as the output. The code I provided is a pseudo code because
6S code consists of plenty of subroutines and the main program has 3219
lines.
What I need is to use MPI to parallel the jobs. So, each computing node
computes one set of the inputs. But I found that the returned value
were not correct after 570 instances. So, I passed the same inputs to
each computing node. But the problem still exist. The first 570
returned values are correct(also same in this case), but after 570 the
returned values are NaN.
Can someone give a hint because our system administrator can't help
with programming? But, I suspect if some setting in MPI prevents
computing more than certain times? I know it sounds weird. But I have
no clue why with the same inputs the returned value could be garbage
after 570 instances.
On
Tue, Aug 18, 2009 at 8:28 AM, Julia He <springwater...@yahoo.com>
wrote:
I found that the subroutine call inside a loop did not
return correct value after certain iterations. In order to simplify the
problem, the inputs to the subroutine are chosen to be constant, so the
output should be the same for every iteration on every computing node.
It is a fortran program, after the initialization the program goes like
this:
do i = 1, N
call my_sub(A, B, C, re)
print *, mypn, A, B, C, re
end do
where re is the output value of the my_sub, A, B, C are inputs to
my_sub.
570
is the number of correct iterations. If the combined instances does not
exceed 570, the output is fine. For example, if I requested 10
computing nodes and N were 40, so it gives 10*40=400 instances, the
output would be fine. But if the combined instances exceeded 570, the
first 570 is fine, but the rest will return NaN value. For example, if
the number of computing nodes were 20 and N were 40, which gives
20*40=800 instances, then the first 570 are fine, but the rest are NaN
value. |
|
|
