[OMPI users] Accessing to the send buffer

[Alberto Canestrelli]

Hi,
I have a problem with a fortran code that I have parallelized with MPI. 
I state in advance that I read the whole ebook "Mit Press - Mpi - The 
Complete Reference, Volume 1" and I took different MPI classes, so I 
have a discrete MPI knowledge. I was able to solve by myself all the 
errors I bumped into but now I am not able to find the bug of my code 
that provides erroneous results. Without entering in the details of my 
code, I think that the cause of the problem could be reletad to the 
following aspect highlighted in the above ebook (in the follow I copy 
and paste from the e-book):


A nonblocking post-send call indicates that the system may start copying 
data
out of the send buffer. The sender must not access any part of the send 
buffer
(neither for loads nor for STORES) after a nonblocking send operation is 
posted until

the complete send returns.
A nonblocking post-receive indicates that the system may start writing 
data into
the receive buffer. The receiver must not access any part of the receive 
buffer after
a nonblocking receive operation is posted, until the complete-receive 
returns.
Rationale. We prohibit read accesses to a send buffer while it is being 
used, even
though the send operation is not supposed to alter the content of this 
buffer. This
may seem more stringent than necessary, but the additional restriction 
causes little
loss of functionality and allows better performance on some systems- 
consider
the case where data transfer is done by a DMA engine that is not 
cache-coherent

with the main processor.End of rationale.

I use plenty of nonblocking post-send in my code. Is it really true that 
the sender must not access any part of the send buffer not even for 
STORES?  Or was it a MPI 1.0 issue?

Thanks.
alberto

Re: [OMPI users] OpenIB Error in ibv_create_srq

[Terry Dontje]

My guess is from the message below saying "(openib) BTL failed to 
initialize"  that the code is probably running over tcp.  To absolutely 
prove this you can specify to only use the openib, sm and self btls to 
eliminate the tcp btl.  To do that you add the following to the mpirun 
line "-mca btl openib,sm,self".  I believe with that specification the 
code will abort and not run to completion. 

What version of the OFED stack are you using?  I wonder if srq is 
supported on your system or not?


--td

Allen Barnett wrote:

Hi: A customer is attempting to run our OpenMPI 1.4.2-based application
on a cluster of machines running RHEL4 with the standard OFED stack. The
HCAs are identified as:

03:01.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1)
04:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev a1)

ibv_devinfo says that one port on the HCAs is active but the other is
down:

hca_id: mthca0
fw_ver: 3.0.2
node_guid:  0006:6a00:9800:4c78
sys_image_guid: 0006:6a00:9800:4c78
vendor_id:  0x066a
vendor_part_id: 23108
hw_ver: 0xA1
phys_port_cnt:  2
port:   1
state:  active (4)
max_mtu:2048 (4)
active_mtu: 2048 (4)
sm_lid: 1
port_lid:   26
port_lmc:   0x00

port:   2
state:  down (1)
max_mtu:2048 (4)
active_mtu: 512 (2)
sm_lid: 0
port_lid:   0
port_lmc:   0x00


 When the OMPI application is run, it prints the error message:


The OpenFabrics (openib) BTL failed to initialize while trying to
create an internal queue.  This typically indicates a failed
OpenFabrics installation, faulty hardware, or that Open MPI is
attempting to use a feature that is not supported on your hardware
(i.e., is a shared receive queue specified in the
btl_openib_receive_queues MCA parameter with a device that does not
support it?).  The failure occured here:

  Local host:  machine001.lan
  OMPI
source: /software/openmpi-1.4.2/ompi/mca/btl/openib/btl_openib.c:250
  Function:ibv_create_srq()
  Error:   Invalid argument (errno=22)
  Device:  mthca0

You may need to consult with your system administrator to get this
problem fixed.


The full log of a run with "btl_openib_verbose 1" is attached. My
application appears to run to completion, but I can't tell if it's just
running on TCP and not using the IB hardware.

I would appreciate any suggestions on how to proceed to fix this error.

Thanks,
Allen

  



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] Accessing to the send buffer

[Terry Dontje]

I believe it is definitely a no-no to STORE (write) into a send buffer 
while a send is posted.  I know there have been debate in the forum to 
relax LOADS (reads) from a send buffer.  I think OMPI can handle the 
latter case (LOADS).  On the posted receive side you open yourself up 
for some race conditions and overwrites if you do STORES or LOADS from a 
posted receive buffer.


--td

Alberto Canestrelli wrote:

Hi,
I have a problem with a fortran code that I have parallelized with 
MPI. I state in advance that I read the whole ebook "Mit Press - Mpi - 
The Complete Reference, Volume 1" and I took different MPI classes, so 
I have a discrete MPI knowledge. I was able to solve by myself all the 
errors I bumped into but now I am not able to find the bug of my code 
that provides erroneous results. Without entering in the details of my 
code, I think that the cause of the problem could be reletad to the 
following aspect highlighted in the above ebook (in the follow I copy 
and paste from the e-book):


A nonblocking post-send call indicates that the system may start 
copying data
out of the send buffer. The sender must not access any part of the 
send buffer
(neither for loads nor for STORES) after a nonblocking send operation 
is posted until

the complete send returns.
A nonblocking post-receive indicates that the system may start writing 
data into
the receive buffer. The receiver must not access any part of the 
receive buffer after
a nonblocking receive operation is posted, until the complete-receive 
returns.
Rationale. We prohibit read accesses to a send buffer while it is 
being used, even
though the send operation is not supposed to alter the content of this 
buffer. This
may seem more stringent than necessary, but the additional restriction 
causes little
loss of functionality and allows better performance on some systems- 
consider
the case where data transfer is done by a DMA engine that is not 
cache-coherent

with the main processor.End of rationale.

I use plenty of nonblocking post-send in my code. Is it really true 
that the sender must not access any part of the send buffer not even 
for STORES?  Or was it a MPI 1.0 issue?

Thanks.
alberto
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] Accessing to the send buffer

[Alberto Canestrelli]

Thanks,
it was late in the night yesterday and i highlighted STORES but I 
meanted to highlight LOADS! I know that
stores are not allowed when you are doing non blocking send-recv. But I 
was impressed about LOADS case. I always do some loads of the data
between all my ISEND-IRECVs and my WAITs. Could  you please confirm me 
that OMPI can handle the LOAD case? And if it cannot handle it, which 
could be the consequence? What could happen in the worst of the case 
when there is a data race in reading a data?

thanks
alberto

Il 02/08/2010 9.32, Alberto Canestrelli ha scritto:

I believe it is definitely a no-no to STORE (write) into a send buffer
while a send is posted. I know there have been debate in the forum to
relax LOADS (reads) from a send buffer. I think OMPI can handle the
latter case (LOADS). On the posted receive side you open yourself up
for some race conditions and overwrites if you do STORES or LOADS from a
posted receive buffer.

--td

Alberto Canestrelli wrote:

 Hi,
 I have a problem with a fortran code that I have parallelized with
 MPI. I state in advance that I read the whole ebook "Mit Press - Mpi -
 The Complete Reference, Volume 1" and I took different MPI classes, so
 I have a discrete MPI knowledge. I was able to solve by myself all the
 errors I bumped into but now I am not able to find the bug of my code
 that provides erroneous results. Without entering in the details of my
 code, I think that the cause of the problem could be reletad to the
 following aspect highlighted in the above ebook (in the follow I copy
 and paste from the e-book):

 A nonblocking post-send call indicates that the system may start
 copying data
 out of the send buffer. The sender must not access any part of the
 send buffer
 (neither for loads nor for STORES) after a nonblocking send operation
 is posted until
 the complete send returns.
 A nonblocking post-receive indicates that the system may start writing
 data into
 the receive buffer. The receiver must not access any part of the
 receive buffer after
 a nonblocking receive operation is posted, until the complete-receive
 returns.
 Rationale. We prohibit read accesses to a send buffer while it is
 being used, even
 though the send operation is not supposed to alter the content of this
 buffer. This
 may seem more stringent than necessary, but the additional restriction
 causes little
 loss of functionality and allows better performance on some systems-
 consider
 the case where data transfer is done by a DMA engine that is not
 cache-coherent
 with the main processor.End of rationale.

 I use plenty of nonblocking post-send in my code. Is it really true
 that the sender must not access any part of the send buffer not even
 for STORES? Or was it a MPI 1.0 issue?
 Thanks.
 alberto
 ___
 users mailing list
 users_at_[hidden]
 http://www.open-mpi.org/mailman/listinfo.cgi/users




--
**
Ing. Alberto Canestrelli
Università degli Studi di Padova,
Dipartimento di Ingegneria Idraulica, Marittima,
Ambientale e Geotecnica,
via Loredan 20, 35131 PADOVA (ITALY)
phone: +39 0498275438
fax:  +39 0498275446
mail:  canestre...@idra.unipd.it

***

Re: [OMPI users] Accessing to the send buffer

[Terry Dontje]

In the posted irecv case if you are reading from the posted receive 
buffer the problem is you may get one of three values:


1.  pre irecv value
2.  value received from the irecv in progress
3.  possibly garbage if you are unlucky enough to access memory that is 
at the same time being updated. 


--td
Alberto Canestrelli wrote:

Thanks,
it was late in the night yesterday and i highlighted STORES but I 
meanted to highlight LOADS! I know that
stores are not allowed when you are doing non blocking send-recv. But 
I was impressed about LOADS case. I always do some loads of the data
between all my ISEND-IRECVs and my WAITs. Could  you please confirm me 
that OMPI can handle the LOAD case? And if it cannot handle it, which 
could be the consequence? What could happen in the worst of the case 
when there is a data race in reading a data?

thanks
alberto

Il 02/08/2010 9.32, Alberto Canestrelli ha scritto:

I believe it is definitely a no-no to STORE (write) into a send buffer
while a send is posted. I know there have been debate in the forum to
relax LOADS (reads) from a send buffer. I think OMPI can handle the
latter case (LOADS). On the posted receive side you open yourself up
for some race conditions and overwrites if you do STORES or LOADS from a
posted receive buffer.

--td

Alberto Canestrelli wrote:

 Hi,
 I have a problem with a fortran code that I have parallelized with
 MPI. I state in advance that I read the whole ebook "Mit Press - Mpi -
 The Complete Reference, Volume 1" and I took different MPI classes, so
 I have a discrete MPI knowledge. I was able to solve by myself all the
 errors I bumped into but now I am not able to find the bug of my code
 that provides erroneous results. Without entering in the details of my
 code, I think that the cause of the problem could be reletad to the
 following aspect highlighted in the above ebook (in the follow I copy
 and paste from the e-book):

 A nonblocking post-send call indicates that the system may start
 copying data
 out of the send buffer. The sender must not access any part of the
 send buffer
 (neither for loads nor for STORES) after a nonblocking send operation
 is posted until
 the complete send returns.
 A nonblocking post-receive indicates that the system may start writing
 data into
 the receive buffer. The receiver must not access any part of the
 receive buffer after
 a nonblocking receive operation is posted, until the complete-receive
 returns.
 Rationale. We prohibit read accesses to a send buffer while it is
 being used, even
 though the send operation is not supposed to alter the content of this
 buffer. This
 may seem more stringent than necessary, but the additional restriction
 causes little
 loss of functionality and allows better performance on some systems-
 consider
 the case where data transfer is done by a DMA engine that is not
 cache-coherent
 with the main processor.End of rationale.

 I use plenty of nonblocking post-send in my code. Is it really true
 that the sender must not access any part of the send buffer not even
 for STORES? Or was it a MPI 1.0 issue?
 Thanks.
 alberto
 ___
 users mailing list
 users_at_[hidden]
 http://www.open-mpi.org/mailman/listinfo.cgi/users







--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] Accessing to the send buffer

[Alberto Canestrelli]

Thanks,
ok that is not my problem I never read a data from the posted receive 
before the correspondent WAIT. Now the last question is: what could 
happen if I  am reading the data from the posted send? I do it plenty of 
times! possible consequences?Can you guarantee me that this approach is 
safe?

thank you very much
Alberto

Il 02/08/2010 11.29, Alberto Canestrelli ha scritto:

In the posted irecv case if you are reading from the posted receive
buffer the problem is you may get one of three values:

1. pre irecv value
2. value received from the irecv in progress
3. possibly garbage if you are unlucky enough to access memory that is
at the same time being updated.

--td
Alberto Canestrelli wrote:

 Thanks,
 it was late in the night yesterday and i highlighted STORES but I
 meanted to highlight LOADS! I know that
 stores are not allowed when you are doing non blocking send-recv. But
 I was impressed about LOADS case. I always do some loads of the data
 between all my ISEND-IRECVs and my WAITs. Could you please confirm me
 that OMPI can handle the LOAD case? And if it cannot handle it, which
 could be the consequence? What could happen in the worst of the case
 when there is a data race in reading a data?
 thanks
 alberto

 Il 02/08/2010 9.32, Alberto Canestrelli ha scritto:
> I believe it is definitely a no-no to STORE (write) into a send buffer
> while a send is posted. I know there have been debate in the forum to
> relax LOADS (reads) from a send buffer. I think OMPI can handle the
> latter case (LOADS). On the posted receive side you open yourself up
> for some race conditions and overwrites if you do STORES or LOADS 
from a

> posted receive buffer.
>
> --td
>
> Alberto Canestrelli wrote:
>> Hi,
>> I have a problem with a fortran code that I have parallelized with
>> MPI. I state in advance that I read the whole ebook "Mit Press - 
Mpi -
>> The Complete Reference, Volume 1" and I took different MPI 
classes, so
>> I have a discrete MPI knowledge. I was able to solve by myself all 
the

>> errors I bumped into but now I am not able to find the bug of my code
>> that provides erroneous results. Without entering in the details 
of my

>> code, I think that the cause of the problem could be reletad to the
>> following aspect highlighted in the above ebook (in the follow I copy
>> and paste from the e-book):
>>
>> A nonblocking post-send call indicates that the system may start
>> copying data
>> out of the send buffer. The sender must not access any part of the
>> send buffer
>> (neither for loads nor for STORES) after a nonblocking send operation
>> is posted until
>> the complete send returns.
>> A nonblocking post-receive indicates that the system may start 
writing

>> data into
>> the receive buffer. The receiver must not access any part of the
>> receive buffer after
>> a nonblocking receive operation is posted, until the complete-receive
>> returns.
>> Rationale. We prohibit read accesses to a send buffer while it is
>> being used, even
>> though the send operation is not supposed to alter the content of 
this

>> buffer. This
>> may seem more stringent than necessary, but the additional 
restriction

>> causes little
>> loss of functionality and allows better performance on some systems-
>> consider
>> the case where data transfer is done by a DMA engine that is not
>> cache-coherent
>> with the main processor.End of rationale.
>>
>> I use plenty of nonblocking post-send in my code. Is it really true
>> that the sender must not access any part of the send buffer not even
>> for STORES? Or was it a MPI 1.0 issue?
>> Thanks.
>> alberto




--
**
Ing. Alberto Canestrelli
Università degli Studi di Padova,
Dipartimento di Ingegneria Idraulica, Marittima,
Ambientale e Geotecnica,
via Loredan 20, 35131 PADOVA (ITALY)
phone: +39 0498275438
fax:  +39 0498275446
mail:  canestre...@idra.unipd.it

***

Re: [OMPI users] Accessing to the send buffer

[Terry Dontje]

For OMPI I believe reading the data buffer given to a posted send will 
not cause any problems.


Anyone on the list care to disagree?

--td

Alberto Canestrelli wrote:

Thanks,
ok that is not my problem I never read a data from the posted receive 
before the correspondent WAIT. Now the last question is: what could 
happen if I  am reading the data from the posted send? I do it plenty 
of times! possible consequences?Can you guarantee me that this 
approach is safe?

thank you very much
Alberto

Il 02/08/2010 11.29, Alberto Canestrelli ha scritto:

In the posted irecv case if you are reading from the posted receive
buffer the problem is you may get one of three values:

1. pre irecv value
2. value received from the irecv in progress
3. possibly garbage if you are unlucky enough to access memory that is
at the same time being updated.

--td
Alberto Canestrelli wrote:

 Thanks,
 it was late in the night yesterday and i highlighted STORES but I
 meanted to highlight LOADS! I know that
 stores are not allowed when you are doing non blocking send-recv. But
 I was impressed about LOADS case. I always do some loads of the data
 between all my ISEND-IRECVs and my WAITs. Could you please confirm me
 that OMPI can handle the LOAD case? And if it cannot handle it, which
 could be the consequence? What could happen in the worst of the case
 when there is a data race in reading a data?
 thanks
 alberto

 Il 02/08/2010 9.32, Alberto Canestrelli ha scritto:
> I believe it is definitely a no-no to STORE (write) into a send 
buffer

> while a send is posted. I know there have been debate in the forum to
> relax LOADS (reads) from a send buffer. I think OMPI can handle the
> latter case (LOADS). On the posted receive side you open yourself up
> for some race conditions and overwrites if you do STORES or LOADS 
from a

> posted receive buffer.
>
> --td
>
> Alberto Canestrelli wrote:
>> Hi,
>> I have a problem with a fortran code that I have parallelized with
>> MPI. I state in advance that I read the whole ebook "Mit Press - 
Mpi -
>> The Complete Reference, Volume 1" and I took different MPI 
classes, so
>> I have a discrete MPI knowledge. I was able to solve by myself 
all the
>> errors I bumped into but now I am not able to find the bug of my 
code
>> that provides erroneous results. Without entering in the details 
of my

>> code, I think that the cause of the problem could be reletad to the
>> following aspect highlighted in the above ebook (in the follow I 
copy

>> and paste from the e-book):
>>
>> A nonblocking post-send call indicates that the system may start
>> copying data
>> out of the send buffer. The sender must not access any part of the
>> send buffer
>> (neither for loads nor for STORES) after a nonblocking send 
operation

>> is posted until
>> the complete send returns.
>> A nonblocking post-receive indicates that the system may start 
writing

>> data into
>> the receive buffer. The receiver must not access any part of the
>> receive buffer after
>> a nonblocking receive operation is posted, until the 
complete-receive

>> returns.
>> Rationale. We prohibit read accesses to a send buffer while it is
>> being used, even
>> though the send operation is not supposed to alter the content of 
this

>> buffer. This
>> may seem more stringent than necessary, but the additional 
restriction

>> causes little
>> loss of functionality and allows better performance on some systems-
>> consider
>> the case where data transfer is done by a DMA engine that is not
>> cache-coherent
>> with the main processor.End of rationale.
>>
>> I use plenty of nonblocking post-send in my code. Is it really true
>> that the sender must not access any part of the send buffer not even
>> for STORES? Or was it a MPI 1.0 issue?
>> Thanks.
>> alberto







--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] Accessing to the send buffer

[Richard Treumann]

For reading the data from an isend buffer to cause problems, the 
underlying hardware would need to have very unusual characteristic that 
the MPI implementation is exploiting.  People have imagined hardware 
characteristics that could make reading an Isend buffer a problem but I 
have never heard of real hardware where it would be.  (Imagine hardware 
where posting an ISEND causes addressability of the underlying memory to 
be handed over to an adapter until the send is done. The processor loses 
addressability.  No such real hardware I ever heard of but if there were, 
the send buffer access rule would be relevant)

The Forum has decided the send buffer rule is to restrictive.



Dick Treumann  -  MPI Team 
IBM Systems & Technology Group
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
Tele (845) 433-7846 Fax (845) 433-8363

Re: [OMPI users] OpenIB Error in ibv_create_srq

[Allen Barnett]

Hi Terry:
It is indeed the case that the openib BTL has not been initialized. I
ran with your tcp-disabled MCA option and it aborted in MPI_Init.

The OFED stack is what's included in RHEL4. It appears to be made up of
the RPMs:
openib-1.4-1.el4
opensm-3.2.5-1.el4
libibverbs-1.1.2-1.el4

How can I determine if srq is supported? Is there an MCA option to
defeat it? (Our in-house cluster has more recent Mellanox IB hardware
and is running this same IB stack and ompi 1.4.2 works OK, so I suspect
srq is supported by the OpenFabrics stack. Perhaps.)

Thanks,
Allen

On Mon, 2010-08-02 at 06:47 -0400, Terry Dontje wrote:
> My guess is from the message below saying "(openib) BTL failed to
> initialize"  that the code is probably running over tcp.  To
> absolutely prove this you can specify to only use the openib, sm and
> self btls to eliminate the tcp btl.  To do that you add the following
> to the mpirun line "-mca btl openib,sm,self".  I believe with that
> specification the code will abort and not run to completion.  
> 
> What version of the OFED stack are you using?  I wonder if srq is
> supported on your system or not?
> 
> --td
> 
> Allen Barnett wrote: 
> > Hi: A customer is attempting to run our OpenMPI 1.4.2-based application
> > on a cluster of machines running RHEL4 with the standard OFED stack. The
> > HCAs are identified as:
> > 
> > 03:01.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1)
> > 04:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev a1)
> > 
> > ibv_devinfo says that one port on the HCAs is active but the other is
> > down:
> > 
> > hca_id: mthca0
> > fw_ver: 3.0.2
> > node_guid:  0006:6a00:9800:4c78
> > sys_image_guid: 0006:6a00:9800:4c78
> > vendor_id:  0x066a
> > vendor_part_id: 23108
> > hw_ver: 0xA1
> > phys_port_cnt:  2
> > port:   1
> > state:  active (4)
> > max_mtu:2048 (4)
> > active_mtu: 2048 (4)
> > sm_lid: 1
> > port_lid:   26
> > port_lmc:   0x00
> > 
> > port:   2
> > state:  down (1)
> > max_mtu:2048 (4)
> > active_mtu: 512 (2)
> > sm_lid: 0
> > port_lid:   0
> > port_lmc:   0x00
> > 
> > 
> >  When the OMPI application is run, it prints the error message:
> > 
> > 
> > The OpenFabrics (openib) BTL failed to initialize while trying to
> > create an internal queue.  This typically indicates a failed
> > OpenFabrics installation, faulty hardware, or that Open MPI is
> > attempting to use a feature that is not supported on your hardware
> > (i.e., is a shared receive queue specified in the
> > btl_openib_receive_queues MCA parameter with a device that does not
> > support it?).  The failure occured here:
> > 
> >   Local host:  machine001.lan
> >   OMPI
> > source: /software/openmpi-1.4.2/ompi/mca/btl/openib/btl_openib.c:250
> >   Function:ibv_create_srq()
> >   Error:   Invalid argument (errno=22)
> >   Device:  mthca0
> > 
> > You may need to consult with your system administrator to get this
> > problem fixed.
> > 
> > 
> > The full log of a run with "btl_openib_verbose 1" is attached. My
> > application appears to run to completion, but I can't tell if it's just
> > running on TCP and not using the IB hardware.
> > 
> > I would appreciate any suggestions on how to proceed to fix this error.
> > 
> > Thanks,
> > Allen
> 

-- 
Allen Barnett
Transpire, Inc
E-Mail: al...@transpireinc.com

Re: [OMPI users] Fortran MPI Struct with Allocatable Array

[Eugene Loh]

I can't give you a complete answer, but I think this is less an MPI
question and more of a Fortran question.  The question is if you have a
Fortran derived type, one of whose components is a POINTER, what does
the data structure look like in linear memory?  I could imagine the
answer is implementation dependent.  Anyhow, here is a sample, non-MPI,
Fortran program that illustrates the question:

% cat b.f90
  type :: small
  integer, pointer :: array(:)
  end type small
  type(small) :: lala

  integer, pointer :: array(:)

  n = 20

  allocate( lala%array(n) )
  allocate(  array(n) )

  lala%array = (/ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20 /)
   array = (/ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20 /)

  call sub(lala)
  call sub(lala%array)
  call sub( array)
end

subroutine sub(x)
  integer x(20)
  write(6,*) x
end
% f90 b.f90
% a.out
 599376 20 4 599372 1 20 -4197508 1 2561 0 33 0 0 0 0 0 0 0 0 0
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
% 

So, your model of 20 consecutive words does not work if you pass the
derived type.  It does work if you pass the POINTER component.  This is
with Oracle (Sun) Studio Fortran.  Again, I can imagine the behavior
depends on the Fortran compiler.

I suspect what's going on is that a POINTER is a complicated data
structure that has all sorts of metadata in it, but if you pass a
POINTER the compiler knows to pass the thing you're pointing to rather
than the metadata itself.

Jeremy Roberts wrote:
I'm trying to parallelize a Fortran
code with rather complicated derived types full of pointer arrays. 
When I build the MPI type for sending, all the static components are
sent, but the pointer arrays are not (and retain initial values).  I
imagine this has to do with memory addresses when creating the MPI
struct, but I have no idea how to fix it.
  
I've included a simple code illustrating my issue below.  Any
suggestions?
  
program mpi_struct_example
  use mpi
  implicit none
  ! declarations
  type :: small
  real, pointer :: array(:)
  end type small
  type(small) :: lala
  integer :: stat, counts(1), types(1), ierr, iam, n=0, MPI_SMALL
  integer (kind=MPI_ADDRESS_KIND) :: displs(1)
  ! initialize MPI and get my rank
  call MPI_INIT( ierr )
  call MPI_COMM_RANK( MPI_COMM_WORLD, iam, ierr )
  n = 20
  allocate( lala%array(n) )
  lala%array = 2.0
  ! build block counts, displacements, and oldtypes
  counts = (/n/)
  displs = (/0/)
  types  = (/MPI_REAL/)
  ! make and commit new type
  call MPI_TYPE_CREATE_STRUCT( 1, counts, displs, types, MPI_SMALL,
ierr )
  call MPI_TYPE_COMMIT( MPI_SMALL, ierr )
  if (iam .eq. 0) then
    ! reset the value of the array
    lala%array  = 1.0 
    call MPI_SEND( lala, 1, MPI_SMALL, 1, 1, MPI_COMM_WORLD,
ierr)   ! this doesn't work
    !call MPI_SEND( lala%array, n, MPI_REAL, 1, 1,
MPI_COMM_WORLD, ierr) ! this does work
    write (*,*) "iam ",iam," and lala%array(1)  = ",
lala%array(1)
  else
    call MPI_RECV( lala, 1, MPI_SMALL, 0, 1, MPI_COMM_WORLD,
stat, ierr )   ! this doesn't work
    !call MPI_RECV( lala%array, n, MPI_REAL, 0, 1,
MPI_COMM_WORLD, stat, ierr ) ! this does work
    write (*,*) "iam ",iam," and lala%array(1)  = ",
lala%array(1), " ( should be 1.0)"
  end if
  call MPI_FINALIZE(ierr)
end program mpi_struct_example

Re: [OMPI users] Accessing to the send buffer

[Terry Frankcombe]

On Mon, 2010-08-02 at 11:36 -0400, Alberto Canestrelli wrote:
> Thanks,
> ok that is not my problem I never read a data from the posted receive 
> before the correspondent WAIT. Now the last question is: what could 
> happen if I  am reading the data from the posted send? I do it plenty of 
> times! possible consequences?Can you guarantee me that this approach is 
> safe?

Well, it seems from what you've posted that the standard says you should
not assume it's safe.  Don't you want to be standard-compliant?


> 
> Il 02/08/2010 11.29, Alberto Canestrelli ha scritto:
> > In the posted irecv case if you are reading from the posted receive
> > buffer the problem is you may get one of three values:
> >
> > 1. pre irecv value
> > 2. value received from the irecv in progress
> > 3. possibly garbage if you are unlucky enough to access memory that is
> > at the same time being updated.
> >
> > --td
> > Alberto Canestrelli wrote:
> >>  Thanks,
> >>  it was late in the night yesterday and i highlighted STORES but I
> >>  meanted to highlight LOADS! I know that
> >>  stores are not allowed when you are doing non blocking send-recv. But
> >>  I was impressed about LOADS case. I always do some loads of the data
> >>  between all my ISEND-IRECVs and my WAITs. Could you please confirm me
> >>  that OMPI can handle the LOAD case? And if it cannot handle it, which
> >>  could be the consequence? What could happen in the worst of the case
> >>  when there is a data race in reading a data?
> >>  thanks
> >>  alberto
> >>
> >>  Il 02/08/2010 9.32, Alberto Canestrelli ha scritto:
> >> > I believe it is definitely a no-no to STORE (write) into a send buffer
> >> > while a send is posted. I know there have been debate in the forum to
> >> > relax LOADS (reads) from a send buffer. I think OMPI can handle the
> >> > latter case (LOADS). On the posted receive side you open yourself up
> >> > for some race conditions and overwrites if you do STORES or LOADS 
> >> from a
> >> > posted receive buffer.
> >> >
> >> > --td
> >> >
> >> > Alberto Canestrelli wrote:
> >> >> Hi,
> >> >> I have a problem with a fortran code that I have parallelized with
> >> >> MPI. I state in advance that I read the whole ebook "Mit Press - 
> >> Mpi -
> >> >> The Complete Reference, Volume 1" and I took different MPI 
> >> classes, so
> >> >> I have a discrete MPI knowledge. I was able to solve by myself all 
> >> the
> >> >> errors I bumped into but now I am not able to find the bug of my code
> >> >> that provides erroneous results. Without entering in the details 
> >> of my
> >> >> code, I think that the cause of the problem could be reletad to the
> >> >> following aspect highlighted in the above ebook (in the follow I copy
> >> >> and paste from the e-book):
> >> >>
> >> >> A nonblocking post-send call indicates that the system may start
> >> >> copying data
> >> >> out of the send buffer. The sender must not access any part of the
> >> >> send buffer
> >> >> (neither for loads nor for STORES) after a nonblocking send operation
> >> >> is posted until
> >> >> the complete send returns.
> >> >> A nonblocking post-receive indicates that the system may start 
> >> writing
> >> >> data into
> >> >> the receive buffer. The receiver must not access any part of the
> >> >> receive buffer after
> >> >> a nonblocking receive operation is posted, until the complete-receive
> >> >> returns.
> >> >> Rationale. We prohibit read accesses to a send buffer while it is
> >> >> being used, even
> >> >> though the send operation is not supposed to alter the content of 
> >> this
> >> >> buffer. This
> >> >> may seem more stringent than necessary, but the additional 
> >> restriction
> >> >> causes little
> >> >> loss of functionality and allows better performance on some systems-
> >> >> consider
> >> >> the case where data transfer is done by a DMA engine that is not
> >> >> cache-coherent
> >> >> with the main processor.End of rationale.
> >> >>
> >> >> I use plenty of nonblocking post-send in my code. Is it really true
> >> >> that the sender must not access any part of the send buffer not even
> >> >> for STORES? Or was it a MPI 1.0 issue?
> >> >> Thanks.
> >> >> alberto
> >
> 
> -- 
> **
> Ing. Alberto Canestrelli
> Università degli Studi di Padova,
> Dipartimento di Ingegneria Idraulica, Marittima,
> Ambientale e Geotecnica,
> via Loredan 20, 35131 PADOVA (ITALY)
> phone: +39 0498275438
> fax:  +39 0498275446
> mail:  canestre...@idra.unipd.it
> 
> ***
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users