Re: [OMPI users] Infiniband performance Problem and stalling

[Randolph Pullen]

Paul I tried NetPipeMPI - (belatedly because their site was down down for a 
couple of days)

The results show a max of 7.4 Gb/s at 8388605 bytes which seems fine.

But my program still runs slowly and stalls occasionally.
I've using 1 buffer per process - I assume this is ok.
Is it of any significance that the  log_num_mtt and log_mtts_per_seg params 
where not set?
Is this a symptom of a broken install?

Reposting original message for clarity - its been a few days...
2nd posts are below this 1st section

I have a test rig comprising 2 i7 systems 8GB RAM with Melanox III HCA 10G cards
running Centos 5.7 Kernel 2.6.18-274
Open MPI 1.4.3
MLNX_OFED_LINUX-1.5.3-1.0.0.2 (OFED-1.5.3-1.0.0.2):
On a Cisco 24 pt switch

Normal performance is:
$ mpirun --mca btl openib,self -n 2 -hostfile mpi.hosts  PingPong
results in:
 Max rate = 958.388867 MB/sec  Min latency = 4.529953 usec
and:
$ mpirun --mca btl tcp,self -n 2 -hostfile mpi.hosts  PingPong
Max rate = 653.547293 MB/sec  Min latency = 19.550323 usec


My application exchanges about a gig of data between the processes with 2 
sender and 2 consumer processes on each node with 1 additional controler 
process on the starting node.
The program splits the data into 64K blocks and uses non blocking sends and 
receives with busy/sleep loops to monitor progress until completion.

My problem is I see better performance under IPoIB then I do on native IB 
(RDMA_CM).
My understanding is that IPoIB is limited to about 1G/s so I am at a loss to 
know why it is faster.

These 2 configurations are equivelant (about 8-10 seconds per cycle)
mpirun --mca btl_openib_flags 2 --mca mpi_leave_pinned 1 --mca btl tcp,self -H 
vh2,vh1 -np 9 --bycore prog
mpirun --mca btl_openib_flags 3 --mca mpi_leave_pinned 1 --mca btl tcp,self -H 
vh2,vh1 -np 9 --bycore prog

And this one produces similar run times but seems to degrade with repeated 
cycles:
mpirun --mca btl_openib_eager_limit 64 --mca mpi_leave_pinned 1 --mca btl 
openib,self -H vh2,vh1 -np 9 --bycore  prog

Other  btl_openib_flags settings result in much lower performance. 
Changing the first of the above configs to use openIB results in a 21 second 
run time at best.  Sometimes it takes up to 5 minutes.
With openib:
- Repeated cycles during a single run seem to slow down with each cycle.
- On occasions it seems to stall indefinately, waiting on a single receive. 

Any ideas appreciated.

Thanks in advance,
Randolph



 From: Randolph Pullen 
To: Paul Kapinos ; Open MPI Users 
 
Sent: Thursday, 30 August 2012 11:46 AM
Subject: Re: [OMPI users] Infiniband performance Problem and stalling
 

Interesting, the log_num_mtt and log_mtts_per_seg params where not set.
Setting them to utilise 2*8G of my RAM resulted in no change to the stalls or 
run time ie; (19,3) (20,2) (21,1) or (6,16). 
In all cases, OpenIB runs in twice the time it takes TCP,except if I push the 
small message max to 64K and force short messages.  Then the openib times are 
the same as TCP and no faster.

I'ms till at a loss as to why...



 From: Paul Kapinos 
To: Randolph Pullen ; Open MPI Users 
 
Sent: Tuesday, 28 August 2012 6:13 PM
Subject: Re: [OMPI users] Infiniband performance Problem and stalling
 
Randolph,
after reading this:

On 08/28/12 04:26, Randolph Pullen wrote:
> - On occasions it seems to stall indefinately, waiting on a single receive.

... I
 would make a blind guess: are you aware about IB card parameters for 
registered memory?
http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem

"Waiting forever" for a single operation is one of symptoms of the problem 
especially in 1.5.3.


best,
Paul

P.S. the lower performance with 'big' chinks is known phenomenon, cf.
http://www.scl.ameslab.gov/netpipe/
(image on bottom of the page). But the chunk size of 64k is fairly small




-- Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Accessing data member of MPI_File struct

[Ammar Ahmad Awan]

Dear All,

I am using a simple program to access MPI_File attributes. I know that the
API provides functions such as MPI_File_get_atomicity( ) but is there a way
to access them directly through code?

Example:

int atomicity;

// method 1
printf("atomicity : %d", MPI_File_get_atomicity(fh,&atomicity));

// method 2
printf("atomicity : %d", fh->atomicity);

Method 1 in the above code works fine as expected but method 2 give me the
following error

error: dereferencing pointer to incomplete type

Can anybody guide me on how to solve this problem.

Kind Regards
-- Ammar
--
Masters Student
Dept. Of Computer Engineering,
Kyung Hee University
--

Re: [OMPI users] error compiling openmpi-1.6.1 on Windows 7

[Shiqing Fan]

Hi Siegmar,

Could you please send the output of ompi_info command under you 64 bit 
env? And could you please also check if you have CCP or HPC pack 
installed? The incorrect configuration of that might cause Open MPI hanging.



Regards,
Shiqing

On 2012-08-29 12:33 PM, Siegmar Gross wrote:

Hi Shiqing,


It seems that the runtime environment is messed up with the different
versions of Open MPI. I suggest you completely remove all the
installations and install 1.6.1 again (just build the installation
project again). It should work without any problem under Cygwin too.

I removed openmpi-1.6 und rebuilt openmpi-1.6.1. Now I can compile and
run a program in 32-bit mode but have still some problems.

D:\...\prog\mpi\small_prog>mpicc init_finalize.c
Microsoft (R) 32-Bit C/C++-Optimierungscompiler Version
   16.00.40219.01 für 80x86
Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.

init_finalize.c
Microsoft (R) Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:init_finalize.exe
"/LIBPATH:C:\Program Files (x86)\openmpi-1.6.1\bin/../lib"
libmpi.lib
libopen-pal.lib
libopen-rte.lib
advapi32.lib
Ws2_32.lib
shlwapi.lib
init_finalize.obj

D:\...\prog\mpi\small_prog>mpiexec init_finalize.exe
-
Sorry!  You were supposed to get help about:
 invalid if_inexclude
But I couldn't open the help file:
 D:\...\prog\mpi\small_prog\..\share\openmpi\help-mpi-btl-tcp.txt:
 No such file or directory.  Sorry!
-

Hello!


Why does "mpiexec" look for the help file relativ to my current
program and not relative to itself? The file is part of the
package.

dir "c:\Program Files (x86)\openmpi-1.6.1\share\openmpi\help-mpi-btl-tcp.txt"
...
03.04.2012  16:30   631 help-mpi-btl-tcp.txt

Must I change something in the source code or can I set an environment
variable for help files? Why does "mpiexec" want to give some help at all?
I don't get this message for openmpi-1.5.1.


D:\...\prog\mpi\small_prog>mpicc init_finalize.c
Microsoft (R) 32-Bit C/C++-Optimierungscompiler Version 16.00.40219.01
   für 80x86
Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.

init_finalize.c
Microsoft (R) Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:init_finalize.exe
"/LIBPATH:C:/Program Files (x86)/openmpi-1.5.1/lib"
libmpi.lib
libopen-pal.lib
libopen-rte.lib
advapi32.lib
Ws2_32.lib
shlwapi.lib
init_finalize.obj

D:\...\prog\mpi\small_prog>mpiexec init_finalize.exe
Hello!
D:\...\prog\mpi\small_prog>





I can also compile in 64-bit mode but the program hangs.


D:\...\prog\mpi\small_prog>mpicc init_finalize.c

Microsoft (R) C/C++-Optimierungscompiler Version 16.00.40219.01 für x64
Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.

init_finalize.c
Microsoft (R) Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:init_finalize.exe
"/LIBPATH:C:\Program Files\openmpi-1.6.1\bin/../lib"
libmpi.lib
libopen-pal.lib
libopen-rte.lib
advapi32.lib
Ws2_32.lib
shlwapi.lib
init_finalize.obj

D:\...\prog\mpi\small_prog>mpiexec init_finalize.exe
^C
D:\...\prog\mpi\small_prog>


It's the same if I compile a 64-bit program with openmpi-1.5.1. The program
itself is very small.

#include 
#include 
#include "mpi.h"

int main (int argc, char *argv[])
{
   MPI_Init (&argc, &argv);
   printf ("Hello!\n");
   MPI_Finalize ();
   return EXIT_SUCCESS;
}


Any ideas why the 64-bit version hangs? Thank you very much for any help
in advance.


Kind regards

Siegmar


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
---
Shiqing Fan
High Performance Computing Center Stuttgart (HLRS)
Tel: ++49(0)711-685-87234  Nobelstrasse 19
Fax: ++49(0)711-685-65832  70569 Stuttgart
http://www.hlrs.de/organization/people/shiqing-fan/
email: f...@hlrs.de

[OMPI users] what is a "node"?

[Zbigniew Koza]

Hi,

consider this specification:

"Curie fat consists in 360 nodes which contains 4 eight cores CPU 
Nehalem-EX clocked at 2.27 GHz, let 32 cores / node and 11520 cores for 
the full fat configuration"


Suppose I would like to run some performance tests just on a single 
processor rather than 4 of them.

Is there a way to do this?
I'm afraid specifying that I need 1 cluster node with 8 MPI prcesses
will result in OS distributing these 8 processes among 4
processors forming the node, and this is not what I'm after.

Z Koza

[OMPI users] fork in Fortran

[sudhirs@]

Dear users,
 How to use fork(), vfork() type functions in Fortran programming ??

Thanking you in advance

-- 
Sudhir Kumar Sahoo
Ph.D Scholar
Dept. Of Chemistry
IIT Kanpur-208016

Re: [OMPI users] Accessing data member of MPI_File struct

[Jeff Squyres]

On Aug 30, 2012, at 5:05 AM, Ammar Ahmad Awan wrote:

> int atomicity;
> 
> // method 1
> printf("atomicity : %d", MPI_File_get_atomicity(fh,&atomicity));

I think you want:

int atomicity;
MPI_File_get_atomicity(fh, &atomicity);
printf("atomicity: %d\n", atomicity);

MPI_File is an opaque structure; you won't be able to access any of the fields 
inside of it directly.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] fork in Fortran

[Jeff Squyres]

Good question.  You might want to ask in some Fortran-based user forums.  :-)

This list is for support of Open MPI, not necessarily any direct language 
support.

But in that light, I'll warn you that fork and friends are not directly 
supported in MPI applications (e.g., it can cause problems if you're using 
OpenFabrics-based networks with MPI).  You should use MPI_COMM_SPAWN, 
instead... but that may be heavier weight than you want.


On Aug 30, 2012, at 7:45 AM, sudhirs@ wrote:

> Dear users,
> How to use fork(), vfork() type functions in Fortran programming ??
> 
> Thanking you in advance
> 
> -- 
> Sudhir Kumar Sahoo
> Ph.D Scholar
> Dept. Of Chemistry
> IIT Kanpur-208016
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] what is a "node"?

[Jeff Squyres]

In the OMPI v1.6 series, you can use the processor affinity options.  And you 
can use --report-bindings to show exactly where processes were bound.  For 
example:

-
% mpirun -np 4 --bind-to-core --report-bindings -bycore uptime
[svbu-mpi056:18904] MCW rank 0 bound to socket 0[core 0]: [B . . .][. . . .]
[svbu-mpi056:18904] MCW rank 1 bound to socket 0[core 1]: [. B . .][. . . .]
[svbu-mpi056:18904] MCW rank 2 bound to socket 0[core 2]: [. . B .][. . . .]
[svbu-mpi056:18904] MCW rank 3 bound to socket 0[core 3]: [. . . B][. . . .]
 05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
 05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
 05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
 05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
% 
-

I bound each process to a single core, and mapped them on a round-robin basis 
by core.  Hence, all 4 processes ended up on their own cores on a single 
processor socket.

The --report-bindings output shows that this particular machine has 2 sockets, 
each with 4 cores.



On Aug 30, 2012, at 5:37 AM, Zbigniew Koza wrote:

> Hi,
> 
> consider this specification:
> 
> "Curie fat consists in 360 nodes which contains 4 eight cores CPU Nehalem-EX 
> clocked at 2.27 GHz, let 32 cores / node and 11520 cores for the full fat 
> configuration"
> 
> Suppose I would like to run some performance tests just on a single processor 
> rather than 4 of them.
> Is there a way to do this?
> I'm afraid specifying that I need 1 cluster node with 8 MPI prcesses
> will result in OS distributing these 8 processes among 4
> processors forming the node, and this is not what I'm after.
> 
> Z Koza
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] OMPI 1.6.x Hang on khugepaged 100% CPU time

[Jeff Squyres]

On Aug 29, 2012, at 2:25 PM, Yong Qin wrote:

> This issue has been observed on OMPI 1.6 and 1.6.1 with openib btl but
> not on 1.4.5 (tcp btl is always fine). The application is VASP and
> only one specific dataset is identified during the testing, and the OS
> is SL 6.2 with kernel 2.6.32-220.23.1.el6.x86_64. The issue is that
> when a certain type of load is put on OMPI 1.6.x, khugepaged thread
> always runs with 100% CPU load, and it looks to me like that OMPI is
> waiting for some memory to be available thus appears to be hung.
> Reducing the per node processes would sometimes ease the problem a bit
> but not always. So I did some further testing by playing around with
> the kernel transparent hugepage support.
> 
> 1. Disable transparent hugepage support completely (echo never
>> /sys/kernel/mm/redhat_transparent_hugepage/enabled). This would allow
> the program to progress as normal (as in 1.4.5). Total run time for an
> iteration is 3036.03 s.

I'll admit that we have not tested using transparent hugepages.  I wonder if 
there's some kind of bad interaction going on here...

What exactly does changing this setting do?

> 2. Disable VM defrag effort (echo never
>> /sys/kernel/mm/redhat_transparent_hugepage/defrag). This allows the
> program to run as well, but the performance is horrible. The same
> iteration takes 4967.40 s.
> 
> 3. Disable defrag in khugepaged (echo no
>> /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag). This
> allows the program to run, and the performance is worse than #1 but
> better than #2. The same iteration takes 3348.10 s.
> 
> 4. Disable both VM defrag and khugepaged defrag (#2 + #3). Similar
> performance as #3.
> 
> So my question is, looks to me like this has to do with the memory
> management in the openib btl, are we using huge pages in 1.6.x? If
> that is true, is there a better way to resolve or workaround it within
> OMPI itself without disabling transparent hugepage support? We'd like
> to keep the hugepage support if possible.

Mellanox -- can you comment on this?

> Also is this related to the
> register memory imbalance issue that Jeff was mentioning recently
> (http://blogs.cisco.com/performance/registered-memory-imbalances/)
> because we definitely have this issue with this dataset from the
> symptoms that I can tell, but I wouldn't expect it to hang on
> khugepaged, or is this just a corner case?

It *could* be... but I really have no idea (haven't thought about huge page 
support w.r.t. registered memory exhaustion / imbalance).  Mellanox?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] error compiling openmpi-1.6.1 on Windows 7

[Siegmar Gross]

Hi Shiqing,

> Could you please send the output of ompi_info command under you 64 bit 
> env? And could you please also check if you have CCP or HPC pack 
> installed? The incorrect configuration of that might cause Open MPI hanging.

I haven't installed Microsoft's Compute Cluster Pack or High Performance
Computing Pack. At the moment I'm testing Open MPI on my Acer notebook
with Windows 7 Professional SP1. "ompi_info" hangs as well before it
prints any information about mca modules.


C:\Program Files\openmpi-1.6.1\bin>ompi_info
 Package: Open MPI Admin@HERMES Distribution
Open MPI: 1.6.1
   Open MPI SVN revision: r27106
   Open MPI release date: Aug 22, 2012
Open RTE: 1.6.1
   Open RTE SVN revision: r27106
   Open RTE release date: Aug 22, 2012
OPAL: 1.6.1
   OPAL SVN revision: r27106
   OPAL release date: Aug 22, 2012
 MPI API: 2.1
Ident string: 1.6.1
  Prefix: C:\Program Files\openmpi-1.6.1\bin/..
 Configured architecture: Windows-6.1 64 bit
  Configure host: HERMES
   Configured by: Admin
   Configured on: 14:40 27.08.2012
  Configure host: HERMES
Built by: Admin
Built on: 14:40 27.08.2012
  Built host: HERMES
  C bindings: yes
C++ bindings: yes
  Fortran77 bindings: no
  Fortran90 bindings: no
 Fortran90 bindings size: na
  C compiler: cl
 C compiler absolute: cl
  C compiler family name: MICROSOFT
  C compiler version: 1600
C++ compiler: cl
   C++ compiler absolute: cl
  Fortran77 compiler: none
  Fortran77 compiler abs: none
  Fortran90 compiler: none
  Fortran90 compiler abs: none
 C profiling: yes
   C++ profiling: yes
 Fortran77 profiling: no
 Fortran90 profiling: no
  C++ exceptions: no
  Thread support: no
   Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: no
 MPI parameter check: never
Memory profiling support: no
Memory debugging support: no
 libltdl support: no
   Heterogeneous support: yes
 mpirun default --prefix: yes
 MPI I/O support: yes
   MPI_WTIME support: gettimeofday
 Symbol vis. support: yes
   Host topology support: no
  MPI extensions: none
   FT Checkpoint support: yes (checkpoint thread: no)
 VampirTrace support: no
  MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
 MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
   MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128


Kind regards

Siegmar


> On 2012-08-29 12:33 PM, Siegmar Gross wrote:
> > Hi Shiqing,
> >
> >> It seems that the runtime environment is messed up with the different
> >> versions of Open MPI. I suggest you completely remove all the
> >> installations and install 1.6.1 again (just build the installation
> >> project again). It should work without any problem under Cygwin too.
> > I removed openmpi-1.6 und rebuilt openmpi-1.6.1. Now I can compile and
> > run a program in 32-bit mode but have still some problems.
> >
> > D:\...\prog\mpi\small_prog>mpicc init_finalize.c
> > Microsoft (R) 32-Bit C/C++-Optimierungscompiler Version
> >16.00.40219.01 für 80x86
> > Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.
> >
> > init_finalize.c
> > Microsoft (R) Incremental Linker Version 10.00.40219.01
> > Copyright (C) Microsoft Corporation.  All rights reserved.
> >
> > /out:init_finalize.exe
> > "/LIBPATH:C:\Program Files (x86)\openmpi-1.6.1\bin/../lib"
> > libmpi.lib
> > libopen-pal.lib
> > libopen-rte.lib
> > advapi32.lib
> > Ws2_32.lib
> > shlwapi.lib
> > init_finalize.obj
> >
> > D:\...\prog\mpi\small_prog>mpiexec init_finalize.exe
> > -
> > Sorry!  You were supposed to get help about:
> >  invalid if_inexclude
> > But I couldn't open the help file:
> >  D:\...\prog\mpi\small_prog\..\share\openmpi\help-mpi-btl-tcp.txt:
> >  No such file or directory.  Sorry!
> > -
> >
> > Hello!
> >
> >
> > Why does "mpiexec" look for the help file relativ to my current
> > program and not relative to itself? The file is part of the
> > package.
> >
> > dir "c:\Program Files 
(x86)\openmpi-1.6.1\share\openmpi\help-mpi-btl-tcp.txt"
> > ...
> > 03.04.2012  16:30   631 help-mpi-btl-tcp.txt
> >
> > Must I change something in the source code or can I set an environment
> > variable for help files? Why does "mpiexec" want to give some help at all?
> > I don't get this message for openmpi-1.5.1.
> >
> >
> > D:\...\prog\mpi\small_prog>mpicc init_finalize.c
> > Microsoft (R) 32-Bit C/C++-Optimierungscompiler Version 16.00.40219.01
> >für 80x86
> > Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.
> >
> > init_finalize.c
> >

[OMPI users] openmpi 1.6.1 and libnuma

[Tom Rosmond]

I just built Openmpi 1.6.1 with the '--with-libnuma=(dir)' and got a
'WARNING: unrecognized options' message.  I am running on a NUMA
architecture and have needed this feature with earlier Openmpi releases.
Is the support now native in the 1.6 versions?  If not, what should I
do?

T. Rosmond

Re: [OMPI users] MPI::Intracomm::Spawn and cluster configuration

[Brian Budge]

In the event that I need to get this up-and-running soon (I do need
something working within 2 weeks), can you recommend an older version
where this is expected to work?

Thanks,
  Brian

On Tue, Aug 28, 2012 at 4:58 PM, Brian Budge  wrote:
> Thanks!
>
> On Tue, Aug 28, 2012 at 4:57 PM, Ralph Castain  wrote:
>> Yeah, I'm seeing the hang as well when running across multiple machines. Let 
>> me dig a little and get this fixed.
>>
>> Thanks
>> Ralph
>>
>> On Aug 28, 2012, at 4:51 PM, Brian Budge  wrote:
>>
>>> Hmmm, I went to the build directories of openmpi for my two machines,
>>> went into the orte/test/mpi directory and made the executables on both
>>> machines.  I set the hostsfile in the env variable on the "master"
>>> machine.
>>>
>>> Here's the output:
>>>
>>> OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile
>>> ./simple_spawn
>>> Parent [pid 97504] starting up!
>>> 0 completed MPI_Init
>>> Parent [pid 97504] about to spawn!
>>> Parent [pid 97507] starting up!
>>> Parent [pid 97508] starting up!
>>> Parent [pid 30626] starting up!
>>> ^C
>>> zsh: interrupt  OMPI_MCA_orte_default_hostfile= ./simple_spawn
>>>
>>> I had to ^C to kill the hung process.
>>>
>>> When I run using mpirun:
>>>
>>> OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile
>>> mpirun -np 1 ./simple_spawn
>>> Parent [pid 97511] starting up!
>>> 0 completed MPI_Init
>>> Parent [pid 97511] about to spawn!
>>> Parent [pid 97513] starting up!
>>> Parent [pid 30762] starting up!
>>> Parent [pid 30764] starting up!
>>> Parent done with spawn
>>> Parent sending message to child
>>> 1 completed MPI_Init
>>> Hello from the child 1 of 3 on host budgeb-sandybridge pid 97513
>>> 0 completed MPI_Init
>>> Hello from the child 0 of 3 on host budgeb-interlagos pid 30762
>>> 2 completed MPI_Init
>>> Hello from the child 2 of 3 on host budgeb-interlagos pid 30764
>>> Child 1 disconnected
>>> Child 0 received msg: 38
>>> Child 0 disconnected
>>> Parent disconnected
>>> Child 2 disconnected
>>> 97511: exiting
>>> 97513: exiting
>>> 30762: exiting
>>> 30764: exiting
>>>
>>> As you can see, I'm using openmpi v 1.6.1.  I just barely freshly
>>> installed on both machines using the default configure options.
>>>
>>> Thanks for all your help.
>>>
>>>  Brian
>>>
>>> On Tue, Aug 28, 2012 at 4:39 PM, Ralph Castain  wrote:
 Looks to me like it didn't find your executable - could be a question of 
 where it exists relative to where you are running. If you look in your 
 OMPI source tree at the orte/test/mpi directory, you'll see an example 
 program "simple_spawn.c" there. Just "make simple_spawn" and execute that 
 with your default hostfile set - does it work okay?

 It works fine for me, hence the question.

 Also, what OMPI version are you using?

 On Aug 28, 2012, at 4:25 PM, Brian Budge  wrote:

> I see.  Okay.  So, I just tried removing the check for universe size,
> and set the universe size to 2.  Here's my output:
>
> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib
> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile ./master_exe
> [budgeb-interlagos:29965] [[4156,0],0] ORTE_ERROR_LOG: Fatal in file
> base/plm_base_receive.c at line 253
> [budgeb-interlagos:29963] [[4156,1],0] ORTE_ERROR_LOG: The specified
> application failed to start in file dpm_orte.c at line 785
>
> The corresponding run with mpirun still works.
>
> Thanks,
> Brian
>
> On Tue, Aug 28, 2012 at 2:46 PM, Ralph Castain  wrote:
>> I see the issue - it's here:
>>
>>> MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &puniverseSize, &flag);
>>>
>>> if(!flag) {
>>> std::cerr << "no universe size" << std::endl;
>>> return -1;
>>> }
>>> universeSize = *puniverseSize;
>>> if(universeSize == 1) {
>>> std::cerr << "cannot start slaves... not enough nodes" << std::endl;
>>> }
>>
>> The universe size is set to 1 on a singleton because the attribute gets 
>> set at the beginning of time - we haven't any way to go back and change 
>> it. The sequence of events explains why. The singleton starts up and 
>> sets its attributes, including universe_size. It also spins off an orte 
>> daemon to act as its own private "mpirun" in case you call comm_spawn. 
>> At this point, however, no hostfile has been read - the singleton is 
>> just an MPI proc doing its own thing, and the orte daemon is just 
>> sitting there on "stand-by".
>>
>> When your app calls comm_spawn, then the orte daemon gets called to 
>> launch the new procs. At that time, it (not the original singleton!) 
>> reads the hostfile to find out how many nodes are around, and then does 
>> the launch.
>>
>> You are trying to check the number of nodes from within the singleton, 
>> wh

Re: [OMPI users] fork in Fortran

[Dmitry N. Mikushin]

Hi,

Modern Fortran has a feature called ISO_C_BINDING. It essentially
allows to declare a binding of external function to be used from
Fortran program. You only need to provide a corresponding interface.
ISO_C_BINDING module contains C-like extensions in type system, but
you don't need them, as your function has no arguments :)

Example:

program fork_test

interface

function fork() bind(C)
use iso_c_binding
integer(c_int) :: fork
end function fork

end interface

print *, 'My PID = ', fork()

end program fork_test

$ make
gfortran fork.f90 -o fork
$ ./fork
 My PID = 4033
 My PID =0

For further info, please refer to language standard
http://gcc.gnu.org/wiki/GFortranStandards#Fortran_2003
If you have any questions - consider asking the gfortran community,
they are very friendly.

Best,
- D.

2012/8/30 sudhirs@ :
> Dear users,
>  How to use fork(), vfork() type functions in Fortran programming ??
>
> Thanking you in advance
>
> --
> Sudhir Kumar Sahoo
> Ph.D Scholar
> Dept. Of Chemistry
> IIT Kanpur-208016
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] valgrind/memory leaks from spawn

[Brian Budge]

Hi all -

I'm writing a program which will start in a single process.  This
program will call init (THREAD_MULTIPLE), and finalize.  In between,
it will call spawn an unknown number of times (think of the program as
a daemon that launches jobs over and over again).

I'm running a simple example right now, and I'm seeing that the more
times I spawn, the more memory I am leaking.  Since I would ideally
have my daemon run indefinitely, a true resource leak is a big deal
for me.

Using valgrind and gdb, I've been tracing into the code.  It appears
that some of the leaks are on purpose (comments say that the memory
should never be freed).  As long as those are one-offs, it's not
really a leak, so no problems there. On the other hand, I think I've
identified some real leaks.  Many of the allocations will increase by
a factor directly proportional to the number of spawns that I call.

One example that is easy enough for me to wrap my head around is that in
orte_grpcomm_base_update_modex_entries(), there is a call made to
modex_lookup_orte_proc that will call the OBJ_NEW() macro.  The object
is inserted into a hash table, and it appears that the entry is never
cleared from the hash table.

I'm not 100% certain that the leaks are not due to my (potential)
misuse of spawn.  Upon Finalize(), I get no reports of leaked handles,
but I'm unsure if that means I've released everything that I need to
release on my side.  Is there anything that typically needs to be
paired with a spawn to clean up resources?

In my simple example, the leak is just under 4K bytes per spawn.  This
isn't huge, and the chances that the program will run long enough at
any time to go into the gigabytes is not super high; however, I'd
still feel better if this could be eliminated.

Thanks,
  Brian

Re: [OMPI users] openmpi 1.6.1 and libnuma

[Jeff Squyres]

On Aug 30, 2012, at 11:26 AM, Tom Rosmond wrote:

> I just built Openmpi 1.6.1 with the '--with-libnuma=(dir)' and got a
> 'WARNING: unrecognized options' message.  I am running on a NUMA
> architecture and have needed this feature with earlier Openmpi releases.
> Is the support now native in the 1.6 versions?  If not, what should I
> do?


You shouldn't need this any more.  Open MPI has wholly replaced its use of 
libnuma with hwloc, which should handle NUMA architectures properly.

Try running a small test with something like what I cited in a post earlier 
today:

http://www.open-mpi.org/community/lists/users/2012/08/20066.php

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] what is a "node"?

[Gus Correa]

Hi  Zbigniew

Besides the OpenMPI processor affinity capability that Jeff mentioned.

If your Curie cluster has a resource manager [Torque, SGE, etc],
your job submission script to the resource manager/ queue system
should specifically request a single node, for the test that you have in 
mind.


For instance, on Torque/PBS, this would be done by adding this directive to
the top of the job script:

#PBS -l nodes=1:ppn=8
...
mpiexec -np 8 ...

meaning that you want the 8 processors [i.e. cores] to be in a single node.

On top of this, you need to add the appropriate process binding
keywords to the mpiexec command line, as Jeff suggested.
'man mpiexec' will tell you a lot about the OpenMPI process binding 
capability, specially in 1.6 and 1.4 series.


In the best of the worlds your resource manager has the ability to also 
assign a group of

cores exclusively to each of the jobs that may be sharing the node.
Say, job1 requests 4 cores and gets cores 0-3 and cannot use any other 
cores,
job2 requests 8 cores and gets cores 4-11 and cannot use any other 
cores, and so on.


However, not all resource managers/ queue systems are built this way 
[particularly the older versions],

and may let the various job processes to drift across all cores in the node.

If the resource manager is old and doesn't have that hardware locality 
capability,

and if you don't want your performance test to risk being polluted by
other jobs running on the same node, that perhaps share the same cores 
with your job,

then you can request all 32 cores in the node for your job,
but use only 8 of them to run your MPI program.
It is wasteful, but may be the only way to go.
For instance, on Torque:

#PBS -l nodes=1:ppn=32
...
mpiexec -np 8 ...

Again, add the OpenMPI process binding keywords to the mpiexec command line,
to ensure the use of a fixed group of 8 cores.

With SGE and Slurm the syntax is different than the above,
but I would guess that there is an equivalent setup.

I hope this helps,
Gus Correa

On 08/30/2012 08:07 AM, Jeff Squyres wrote:

In the OMPI v1.6 series, you can use the processor affinity options.  And you 
can use --report-bindings to show exactly where processes were bound.  For 
example:

-
% mpirun -np 4 --bind-to-core --report-bindings -bycore uptime
[svbu-mpi056:18904] MCW rank 0 bound to socket 0[core 0]: [B . . .][. . . .]
[svbu-mpi056:18904] MCW rank 1 bound to socket 0[core 1]: [. B . .][. . . .]
[svbu-mpi056:18904] MCW rank 2 bound to socket 0[core 2]: [. . B .][. . . .]
[svbu-mpi056:18904] MCW rank 3 bound to socket 0[core 3]: [. . . B][. . . .]
  05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
  05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
  05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
  05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
%
-

I bound each process to a single core, and mapped them on a round-robin basis 
by core.  Hence, all 4 processes ended up on their own cores on a single 
processor socket.

The --report-bindings output shows that this particular machine has 2 sockets, 
each with 4 cores.



On Aug 30, 2012, at 5:37 AM, Zbigniew Koza wrote:


Hi,

consider this specification:

"Curie fat consists in 360 nodes which contains 4 eight cores CPU Nehalem-EX clocked 
at 2.27 GHz, let 32 cores / node and 11520 cores for the full fat configuration"

Suppose I would like to run some performance tests just on a single processor 
rather than 4 of them.
Is there a way to do this?
I'm afraid specifying that I need 1 cluster node with 8 MPI prcesses
will result in OS distributing these 8 processes among 4
processors forming the node, and this is not what I'm after.

Z Koza
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] what is a "node"?

[Zbigniew Koza]

Thanks a lot!

Z Koza

2012/8/30 Gus Correa 

> Hi  Zbigniew
>
> Besides the OpenMPI processor affinity capability that Jeff mentioned.
>
> If your Curie cluster has a resource manager [Torque, SGE, etc],
> your job submission script to the resource manager/ queue system
> should specifically request a single node, for the test that you have in
> mind.
>
> For instance, on Torque/PBS, this would be done by adding this directive to
> the top of the job script:
>
> #PBS -l nodes=1:ppn=8
> ...
> mpiexec -np 8 ...
>
> meaning that you want the 8 processors [i.e. cores] to be in a single node.
>
> On top of this, you need to add the appropriate process binding
> keywords to the mpiexec command line, as Jeff suggested.
> 'man mpiexec' will tell you a lot about the OpenMPI process binding
> capability, specially in 1.6 and 1.4 series.
>
> In the best of the worlds your resource manager has the ability to also
> assign a group of
> cores exclusively to each of the jobs that may be sharing the node.
> Say, job1 requests 4 cores and gets cores 0-3 and cannot use any other
> cores,
> job2 requests 8 cores and gets cores 4-11 and cannot use any other cores,
> and so on.
>
> However, not all resource managers/ queue systems are built this way
> [particularly the older versions],
> and may let the various job processes to drift across all cores in the
> node.
>
> If the resource manager is old and doesn't have that hardware locality
> capability,
> and if you don't want your performance test to risk being polluted by
> other jobs running on the same node, that perhaps share the same cores
> with your job,
> then you can request all 32 cores in the node for your job,
> but use only 8 of them to run your MPI program.
> It is wasteful, but may be the only way to go.
> For instance, on Torque:
>
> #PBS -l nodes=1:ppn=32
> ...
> mpiexec -np 8 ...
>
> Again, add the OpenMPI process binding keywords to the mpiexec command
> line,
> to ensure the use of a fixed group of 8 cores.
>
> With SGE and Slurm the syntax is different than the above,
> but I would guess that there is an equivalent setup.
>
> I hope this helps,
> Gus Correa
>
>
> On 08/30/2012 08:07 AM, Jeff Squyres wrote:
>
>> In the OMPI v1.6 series, you can use the processor affinity options.  And
>> you can use --report-bindings to show exactly where processes were bound.
>>  For example:
>>
>> -
>> % mpirun -np 4 --bind-to-core --report-bindings -bycore uptime
>> [svbu-mpi056:18904] MCW rank 0 bound to socket 0[core 0]: [B . . .][. . .
>> .]
>> [svbu-mpi056:18904] MCW rank 1 bound to socket 0[core 1]: [. B . .][. . .
>> .]
>> [svbu-mpi056:18904] MCW rank 2 bound to socket 0[core 2]: [. . B .][. . .
>> .]
>> [svbu-mpi056:18904] MCW rank 3 bound to socket 0[core 3]: [. . . B][. . .
>> .]
>>   05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
>>   05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
>>   05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
>>   05:06:13 up 7 days,  6:57,  1 user,  load average: 0.29, 0.10, 0.03
>> %
>> -
>>
>> I bound each process to a single core, and mapped them on a round-robin
>> basis by core.  Hence, all 4 processes ended up on their own cores on a
>> single processor socket.
>>
>> The --report-bindings output shows that this particular machine has 2
>> sockets, each with 4 cores.
>>
>>
>>
>> On Aug 30, 2012, at 5:37 AM, Zbigniew Koza wrote:
>>
>>  Hi,
>>>
>>> consider this specification:
>>>
>>> "Curie fat consists in 360 nodes which contains 4 eight cores CPU
>>> Nehalem-EX clocked at 2.27 GHz, let 32 cores / node and 11520 cores for the
>>> full fat configuration"
>>>
>>> Suppose I would like to run some performance tests just on a single
>>> processor rather than 4 of them.
>>> Is there a way to do this?
>>> I'm afraid specifying that I need 1 cluster node with 8 MPI prcesses
>>> will result in OS distributing these 8 processes among 4
>>> processors forming the node, and this is not what I'm after.
>>>
>>> Z Koza
>>> __**_
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/**mailman/listinfo.cgi/users
>>>
>>
>>
> __**_
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/**mailman/listinfo.cgi/users
>

Re: [OMPI users] OMPI 1.6.x Hang on khugepaged 100% CPU time

[Yong Qin]

On Thu, Aug 30, 2012 at 5:12 AM, Jeff Squyres  wrote:
> On Aug 29, 2012, at 2:25 PM, Yong Qin wrote:
>
>> This issue has been observed on OMPI 1.6 and 1.6.1 with openib btl but
>> not on 1.4.5 (tcp btl is always fine). The application is VASP and
>> only one specific dataset is identified during the testing, and the OS
>> is SL 6.2 with kernel 2.6.32-220.23.1.el6.x86_64. The issue is that
>> when a certain type of load is put on OMPI 1.6.x, khugepaged thread
>> always runs with 100% CPU load, and it looks to me like that OMPI is
>> waiting for some memory to be available thus appears to be hung.
>> Reducing the per node processes would sometimes ease the problem a bit
>> but not always. So I did some further testing by playing around with
>> the kernel transparent hugepage support.
>>
>> 1. Disable transparent hugepage support completely (echo never
>>> /sys/kernel/mm/redhat_transparent_hugepage/enabled). This would allow
>> the program to progress as normal (as in 1.4.5). Total run time for an
>> iteration is 3036.03 s.
>
> I'll admit that we have not tested using transparent hugepages.  I wonder if 
> there's some kind of bad interaction going on here...

The transparent hugepage is "transparent", which means it is
automatically applied to all applications unless it is explicitly told
otherwise. I highly suspect that it is not working properly in this
case.

>
> What exactly does changing this setting do?

Here (http://lwn.net/Articles/423592/)  is a pretty good documentation
on what these settings would do to the behaviour of the THP. I don't
think I can explain it better than the article so I will leave it to
you to digest. :)

>
>> 2. Disable VM defrag effort (echo never
>>> /sys/kernel/mm/redhat_transparent_hugepage/defrag). This allows the
>> program to run as well, but the performance is horrible. The same
>> iteration takes 4967.40 s.
>>
>> 3. Disable defrag in khugepaged (echo no
>>> /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag). This
>> allows the program to run, and the performance is worse than #1 but
>> better than #2. The same iteration takes 3348.10 s.
>>
>> 4. Disable both VM defrag and khugepaged defrag (#2 + #3). Similar
>> performance as #3.
>>
>> So my question is, looks to me like this has to do with the memory
>> management in the openib btl, are we using huge pages in 1.6.x? If
>> that is true, is there a better way to resolve or workaround it within
>> OMPI itself without disabling transparent hugepage support? We'd like
>> to keep the hugepage support if possible.
>
> Mellanox -- can you comment on this?

THP is useful on large memory applications, which we have a lot here.
So having it working would definitely benefit us. But if there is no
work around from OMPI side, it is apparently more important to have
the application to run than just lose a few percent of performance, I
guess we will have to turn it off.

>
>> Also is this related to the
>> register memory imbalance issue that Jeff was mentioning recently
>> (http://blogs.cisco.com/performance/registered-memory-imbalances/)
>> because we definitely have this issue with this dataset from the
>> symptoms that I can tell, but I wouldn't expect it to hang on
>> khugepaged, or is this just a corner case?
>
> It *could* be... but I really have no idea (haven't thought about huge page 
> support w.r.t. registered memory exhaustion / imbalance).  Mellanox?
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


Thanks,

Yong Qin

Re: [OMPI users] MPI::Intracomm::Spawn and cluster configuration

[Ralph Castain]

not off the top of my head. However, as noted earlier, there is absolutely no 
advantage to a singleton vs mpirun start - all the singleton does is 
immediately fork/exec "mpirun" to support the rest of the job. In both cases, 
you have a daemon running the job - only difference is in the number of 
characters the user types to start it.


On Aug 30, 2012, at 8:44 AM, Brian Budge  wrote:

> In the event that I need to get this up-and-running soon (I do need
> something working within 2 weeks), can you recommend an older version
> where this is expected to work?
> 
> Thanks,
>  Brian
> 
> On Tue, Aug 28, 2012 at 4:58 PM, Brian Budge  wrote:
>> Thanks!
>> 
>> On Tue, Aug 28, 2012 at 4:57 PM, Ralph Castain  wrote:
>>> Yeah, I'm seeing the hang as well when running across multiple machines. 
>>> Let me dig a little and get this fixed.
>>> 
>>> Thanks
>>> Ralph
>>> 
>>> On Aug 28, 2012, at 4:51 PM, Brian Budge  wrote:
>>> 
 Hmmm, I went to the build directories of openmpi for my two machines,
 went into the orte/test/mpi directory and made the executables on both
 machines.  I set the hostsfile in the env variable on the "master"
 machine.
 
 Here's the output:
 
 OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile
 ./simple_spawn
 Parent [pid 97504] starting up!
 0 completed MPI_Init
 Parent [pid 97504] about to spawn!
 Parent [pid 97507] starting up!
 Parent [pid 97508] starting up!
 Parent [pid 30626] starting up!
 ^C
 zsh: interrupt  OMPI_MCA_orte_default_hostfile= ./simple_spawn
 
 I had to ^C to kill the hung process.
 
 When I run using mpirun:
 
 OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile
 mpirun -np 1 ./simple_spawn
 Parent [pid 97511] starting up!
 0 completed MPI_Init
 Parent [pid 97511] about to spawn!
 Parent [pid 97513] starting up!
 Parent [pid 30762] starting up!
 Parent [pid 30764] starting up!
 Parent done with spawn
 Parent sending message to child
 1 completed MPI_Init
 Hello from the child 1 of 3 on host budgeb-sandybridge pid 97513
 0 completed MPI_Init
 Hello from the child 0 of 3 on host budgeb-interlagos pid 30762
 2 completed MPI_Init
 Hello from the child 2 of 3 on host budgeb-interlagos pid 30764
 Child 1 disconnected
 Child 0 received msg: 38
 Child 0 disconnected
 Parent disconnected
 Child 2 disconnected
 97511: exiting
 97513: exiting
 30762: exiting
 30764: exiting
 
 As you can see, I'm using openmpi v 1.6.1.  I just barely freshly
 installed on both machines using the default configure options.
 
 Thanks for all your help.
 
 Brian
 
 On Tue, Aug 28, 2012 at 4:39 PM, Ralph Castain  wrote:
> Looks to me like it didn't find your executable - could be a question of 
> where it exists relative to where you are running. If you look in your 
> OMPI source tree at the orte/test/mpi directory, you'll see an example 
> program "simple_spawn.c" there. Just "make simple_spawn" and execute that 
> with your default hostfile set - does it work okay?
> 
> It works fine for me, hence the question.
> 
> Also, what OMPI version are you using?
> 
> On Aug 28, 2012, at 4:25 PM, Brian Budge  wrote:
> 
>> I see.  Okay.  So, I just tried removing the check for universe size,
>> and set the universe size to 2.  Here's my output:
>> 
>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib
>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile ./master_exe
>> [budgeb-interlagos:29965] [[4156,0],0] ORTE_ERROR_LOG: Fatal in file
>> base/plm_base_receive.c at line 253
>> [budgeb-interlagos:29963] [[4156,1],0] ORTE_ERROR_LOG: The specified
>> application failed to start in file dpm_orte.c at line 785
>> 
>> The corresponding run with mpirun still works.
>> 
>> Thanks,
>> Brian
>> 
>> On Tue, Aug 28, 2012 at 2:46 PM, Ralph Castain  wrote:
>>> I see the issue - it's here:
>>> 
 MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &puniverseSize, &flag);
 
 if(!flag) {
std::cerr << "no universe size" << std::endl;
return -1;
 }
 universeSize = *puniverseSize;
 if(universeSize == 1) {
std::cerr << "cannot start slaves... not enough nodes" << std::endl;
 }
>>> 
>>> The universe size is set to 1 on a singleton because the attribute gets 
>>> set at the beginning of time - we haven't any way to go back and change 
>>> it. The sequence of events explains why. The singleton starts up and 
>>> sets its attributes, including universe_size. It also spins off an orte 
>>> daemon to act as its own private "mpirun" in case you call comm_spawn. 
>

Re: [OMPI users] Accessing data member of MPI_File struct

[Ammar Ahmad Awan]

Thanks for your answer.

Yes, I mistakenly printed the return value of the function rather than
atomicity.

My real problem is that I want to access the fields from the MPI_File
structure other than the ones provided by the API e.g. the fd_sys.

Atomicity was just one example I used to explain my problem. If MPI_File is
an opaque structure, is there any other way or any other structure through
which I can reach the fields?

Thanks
-- ammar

On Thu, Aug 30, 2012 at 8:48 PM, Jeff Squyres  wrote:

> On Aug 30, 2012, at 5:05 AM, Ammar Ahmad Awan wrote:
>
> > int atomicity;
> >
> > // method 1
> > printf("atomicity : %d", MPI_File_get_atomicity(fh,&atomicity));
>
> I think you want:
>
> int atomicity;
> MPI_File_get_atomicity(fh, &atomicity);
> printf("atomicity: %d\n", atomicity);
>
> MPI_File is an opaque structure; you won't be able to access any of the
> fields inside of it directly.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>