[OMPI users] Choices on how to implement a python module in MPI.
Dear list, I have a python module written in C++ to help users manipulate a huge amount of genetics data. Using this module, users can write a script to create/load/manipulate data easily. For efficiency and memory management reasons, I would like to write a MPI version of the module so that I can spread the data to other machines. I have some experience with MPI-1 so I started with the conventional design. That is to say, a fixed number of nodes are started and execute the same script. The data is split across nodes but all nodes can read/write data as if the data is local. That is to say, write operation is done on one of the nodes that has that piece of data, and results of read operation are broadcasted so that they appear to be local to all the nodes. The broadcast is needed to ensure identical execution logic of the script on all nodes. Although a test module is up and running, making sure all scripts *see* the same data and execute the same script has proven to be very inefficient and difficult. For example, if a script perform some action based on a local random number, different nodes would probably be out of sync. I am thinking of an implementation in which only the head node executes the script. It creates the slave nodes and asks them to act on their local data if needed. RMA can be used so that the head node can access data from slave nodes directly. This looks like an efficient solution but I am not sure how to instruct the slave nodes on what they should do. I mean, it is difficult to tell a slave node to execute a certain function with such and such parameters. Treating slave nodes as memory storage and use RMA for all the operations does not sound like a good idea either. I have been evaluating different approaches and have not decided which way to do. I would highly appreciate any advise on how to design and implement such a module. Many thanks in advance. Bo
Re: [OMPI users] Can't run simple job with openmpi using the Intel compiler
On 2/2/07, Dennis McRitchie wrote: When I submit a simple job (described below) using PBS, I always get one of the following two errors: 1) [adroit-28:03945] [0,0,1]-[0,0,0] mca_oob_tcp_peer_recv_blocking: recv() failed with errno=104 2) [adroit-30:03770] [0,0,3]-[0,0,0] mca_oob_tcp_peer_complete_connect: connection failed (errno=111) - retrying (pid=3770) Hi Dennis, Looks like you could be blocked by a firewall. Can you make sure that you disable firewalls on both nodes and try ? gurhan The program does a uname and prints out results to standard out. The only MPI calls it makes are MPI_Init, MPI_Comm_size, MPI_Comm_rank, and MPI_Finalize. I have tried it with both openmpi v 1.1.2 and 1.1.4, built with Intel C compiler 9.1.045, and get the same results. But if I build the same versions of openmpi using gcc, the test program always works fine. The app itself is built with mpicc. It runs successfully if run from the command line with "mpiexec -n X ", where X is 1 to 8, but if I wrap it in the following qsub command file: --- #PBS -l pmem=512mb,nodes=1:ppn=1,walltime=0:10:00 #PBS -m abe # #PBS -o /home0/dmcr/my_mpi/curt/uname_test.gcc.stdout # #PBS -e /home0/dmcr/my_mpi/curt/uname_test.gcc.stderr cd /home/dmcr/my_mpi/openmpi echo "About to call mpiexec" module list mpiexec -n 1 uname_test.intel echo "After call to mpiexec" it fails on any number of processors from 1 to 8, and the application segfaults. The complete standard error of an 8-processsor job follows (note that mpiexec ran on adroit-31, but usually there is no info about adroit-31 in standard error): - Currently Loaded Modulefiles: 1) intel/9.1/32/C/9.1.045 4) intel/9.1/32/default 2) intel/9.1/32/Fortran/9.1.040 5) openmpi/intel/1.1.2/32 3) intel/9.1/32/Iidb/9.1.045 Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at addr:0x5 [0] func:/usr/local/openmpi/1.1.4/intel/i386/lib/libopal.so.0 [0xb72c5b] *** End of error message *** ^@[adroit-29:03934] [0,0,2]-[0,0,0] mca_oob_tcp_peer_recv_blocking: recv() failed with errno=104 [adroit-28:03945] [0,0,1]-[0,0,0] mca_oob_tcp_peer_recv_blocking: recv() failed with errno=104 [adroit-30:03770] [0,0,3]-[0,0,0] mca_oob_tcp_peer_complete_connect: connection failed (errno=111) - retrying (pid=3770) -- The complete standard error of an 1-processsor job follows: -- Currently Loaded Modulefiles: 1) intel/9.1/32/C/9.1.045 4) intel/9.1/32/default 2) intel/9.1/32/Fortran/9.1.040 5) openmpi/intel/1.1.2/32 3) intel/9.1/32/Iidb/9.1.045 Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at addr:0x2 [0] func:/usr/local/openmpi/1.1.2/intel/i386/lib/libopal.so.0 [0x27d847] *** End of error message *** ^@[adroit-31:08840] [0,0,1]-[0,0,0] mca_oob_tcp_peer_complete_connect: connection failed (errno=111) - retrying (pid=8840) --- Any thoughts as to why this might be failing? Thanks, Dennis Dennis McRitchie Computational Science and Engineering Support (CSES) Academic Services Department Office of Information Technology Princeton University ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Choices on how to implement a python module in MPI.
Have you looked at the self-scheduling algorithm described in "USING MPI" by Gropp, Lusk, and Skjellum. I have seen efficient implementations of it for large satellite data assimilation problems in numerical weather prediction, where load distribution across processors cannot be predicted in advance. It is somewhat analogous to similar algorithms in OPENMP, where the number of 'tasks' is significantly larger than the number of processors. T. Rosmond On Sat, 2007-02-03 at 23:48 -0600, Bo Peng wrote: > Dear list, > > I have a python module written in C++ to help users manipulate a huge > amount of genetics data. Using this module, users can write a script > to create/load/manipulate data easily. For efficiency and memory > management reasons, I would like to write a MPI version of the module > so that I can spread the data to other machines. > > I have some experience with MPI-1 so I started with the conventional > design. That is to say, a fixed number of nodes are started and > execute the same script. The data is split across nodes but all nodes > can read/write data as if the data is local. That is to say, write > operation is done on one of the nodes that has that piece of data, and > results of read operation are broadcasted so that they appear to be > local to all the nodes. The broadcast is needed to ensure identical > execution logic of the script on all nodes. > > Although a test module is up and running, making sure all scripts > *see* the same data and execute the same script has proven to be very > inefficient and difficult. For example, if a script perform some > action based on a local random number, different nodes would probably > be out of sync. > > I am thinking of an implementation in which only the head node > executes the script. It creates the slave nodes and asks them to act > on their local data if needed. RMA can be used so that the head node > can access data from slave nodes directly. This looks like an > efficient solution but I am not sure how to instruct the slave nodes > on what they should do. I mean, it is difficult to tell a slave node > to execute a certain function with such and such parameters. Treating > slave nodes as memory storage and use RMA for all the operations does > not sound like a good idea either. > > I have been evaluating different approaches and have not decided which > way to do. I would highly appreciate any advise on how to design and > implement such a module. > > Many thanks in advance. > Bo > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
