could you please check that ofed_info -s indeed prints mofed 2.4-1.0.0? why LD_PRELOAD needed in your command line? Can you try
module load hpcx mpirun -np $np test.exe ? On Sat, Apr 18, 2015 at 8:39 AM, Subhra Mazumdar <subhramazumd...@gmail.com> wrote: > I followed the instructions as in the README, now getting a different > error: > > [root@JARVICE hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5]# > ../openmpi-1.8.4/openmpinstall/bin/mpirun --allow-run-as-root --mca mtl mxm > -x LD_PRELOAD="../openmpi-1.8.4/openmpinstall/lib/libmpi.so.1 > ./mxm/lib/libmxm.so.2" -n 1 ../backend localhost : -x > LD_PRELOAD="../openmpi-1.8.4/openmpinstall/lib/libmpi.so.1 > ./mxm/lib/libmxm.so.2 ../libci.so" -n 1 ../app2 > > -------------------------------------------------------------------------- > > WARNING: a request was made to bind a process. While the system > > supports binding the process itself, at least one node does NOT > > support binding memory to the process location. > > Node: JARVICE > > This usually is due to not having the required NUMA support installed > > on the node. In some Linux distributions, the required support is > > contained in the libnumactl and libnumactl-devel packages. > > This is a warning only; your job will continue, though performance may be > degraded. > > -------------------------------------------------------------------------- > > i am backend > > [1429334876.139452] [JARVICE:449 :0] ib_dev.c:445 MXM WARN failed > call to ibv_exp_use_priv_env(): Function not implemented > > [1429334876.139464] [JARVICE:449 :0] ib_dev.c:456 MXM ERROR > ibv_query_device() returned 38: Function not implemented > > [1429334876.139982] [JARVICE:449 :0] ib_dev.c:445 MXM WARN failed > call to ibv_exp_use_priv_env(): Function not implemented > > [1429334876.139990] [JARVICE:449 :0] ib_dev.c:456 MXM ERROR > ibv_query_device() returned 38: Function not implemented > > [1429334876.142649] [JARVICE:450 :0] ib_dev.c:445 MXM WARN failed > call to ibv_exp_use_priv_env(): Function not implemented > > [1429334876.142666] [JARVICE:450 :0] ib_dev.c:456 MXM ERROR > ibv_query_device() returned 38: Function not implemented > > [1429334876.143235] [JARVICE:450 :0] ib_dev.c:445 MXM WARN failed > call to ibv_exp_use_priv_env(): Function not implemented > > [1429334876.143243] [JARVICE:450 :0] ib_dev.c:456 MXM ERROR > ibv_query_device() returned 38: Function not implemented > > -------------------------------------------------------------------------- > > Initialization of MXM library failed. > > Error: Input/output error > > -------------------------------------------------------------------------- > > [JARVICE:449 :0] Caught signal 11 (Segmentation fault) > > [JARVICE:450 :0] Caught signal 11 (Segmentation fault) > > ==== backtrace ==== > > 2 0x000000000005640c mxm_handle_error() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm-v3.2/src/mxm/util/debug/debug.c:641 > > 3 0x000000000005657c mxm_error_signal_handler() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm-v3.2/src/mxm/util/debug/debug.c:616 > > 4 0x00000000000329a0 killpg() ??:0 > > 5 0x000000000004812c _IO_vfprintf() ??:0 > > 6 0x000000000006f6da vasprintf() ??:0 > > 7 0x0000000000059b3b opal_show_help_vstring() ??:0 > > 8 0x0000000000026630 orte_show_help() ??:0 > > 9 0x0000000000001a3f mca_bml_r2_add_procs() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/ompi-mellanox-v1.8/ompi/mca/bml/r2/bml_r2.c:409 > > 10 0x0000000000004475 mca_pml_ob1_add_procs() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/ompi-mellanox-v1.8/ompi/mca/pml/ob1/pml_ob1.c:332 > > 11 0x00000000000442f3 ompi_mpi_init() ??:0 > > 12 0x0000000000067cb0 PMPI_Init_thread() ??:0 > > 13 0x000000000000d0ca l_getLocalFromConfig() > /root/rain_ib/interposer/libciutils.c:83 > > 14 0x000000000000c7b4 __cudaRegisterFatBinary() > /root/rain_ib/interposer/libci.c:4055 > > 15 0x0000000000402b59 > _ZL70__sti____cudaRegisterAll_39_tmpxft_00000703_00000000_6_app2_cpp1_ii_hwv() > tmpxft_00000703_00000000-3_app2.cudafe1.cpp:0 > > 16 0x0000000000402dd6 __do_global_ctors_aux() crtstuff.c:0 > > =================== > > ==== backtrace ==== > > 2 0x000000000005640c mxm_handle_error() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm-v3.2/src/mxm/util/debug/debug.c:641 > > 3 0x000000000005657c mxm_error_signal_handler() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm-v3.2/src/mxm/util/debug/debug.c:616 > > 4 0x00000000000329a0 killpg() ??:0 > > 5 0x000000000004812c _IO_vfprintf() ??:0 > > 6 0x000000000006f6da vasprintf() ??:0 > > 7 0x0000000000059b3b opal_show_help_vstring() ??:0 > > 8 0x0000000000026630 orte_show_help() ??:0 > > 9 0x0000000000001a3f mca_bml_r2_add_procs() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/ompi-mellanox-v1.8/ompi/mca/bml/r2/bml_r2.c:409 > > 10 0x0000000000004475 mca_pml_ob1_add_procs() > > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/ompi-mellanox-v1.8/ompi/mca/pml/ob1/pml_ob1.c:332 > > 11 0x00000000000442f3 ompi_mpi_init() ??:0 > > 12 0x0000000000067cb0 PMPI_Init_thread() ??:0 > > 13 0x0000000000404fdf main() /root/rain_ib/backend/backend.c:1237 > > 14 0x000000000001ed1d __libc_start_main() ??:0 > > 15 0x0000000000402db9 _start() ??:0 > > =================== > > -------------------------------------------------------------------------- > > mpirun noticed that process rank 1 with PID 450 on node JARVICE exited on > signal 11 (Segmentation fault). > > -------------------------------------------------------------------------- > > [JARVICE:00447] 1 more process has sent help message help-mtl-mxm.txt / > mxm init > > [JARVICE:00447] Set MCA parameter "orte_base_help_aggregate" to 0 to see > all help / error messages > > [root@JARVICE hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5]# > > > Subhra. > > > On Mon, Apr 13, 2015 at 10:58 PM, Mike Dubman <mi...@dev.mellanox.co.il> > wrote: > >> Have you followed installation steps from README (Also here for reference >> http://bgate.mellanox.com/products/hpcx/README.txt) >> >> ... >> >> * Load OpenMPI/OpenSHMEM v1.8 based package: >> >> % source $HPCX_HOME/hpcx-init.sh >> % hpcx_load >> % env | grep HPCX >> % mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_usempi >> % oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem >> % hpcx_unload >> >> 3. Load HPCX environment from modules >> >> * Load OpenMPI/OpenSHMEM based package: >> >> % module use $HPCX_HOME/modulefiles >> % module load hpcx >> % mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c >> % oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem >> % module unload hpcx >> >> ... >> >> On Tue, Apr 14, 2015 at 5:42 AM, Subhra Mazumdar < >> subhramazumd...@gmail.com> wrote: >> >>> I am using 2.4-1.0.0 mellanox ofed. >>> >>> I downloaded mofed tarball >>> hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5.tar and extracted >>> it. It has mxm directory. >>> >>> hpcx-v1.2.0-325-[root@JARVICE ~]# ls >>> hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5 >>> archive fca hpcx-init-ompi-mellanox-v1.8.sh ibprof >>> modulefiles ompi-mellanox-v1.8 sources VERSION >>> bupc-master hcoll hpcx-init.sh knem >>> mxm README.txt utils >>> >>> I tried using LD_PRELOAD for libmxm, but getting a different error stack >>> now as following >>> >>> [root@JARVICE ~]# ./openmpi-1.8.4/openmpinstall/bin/mpirun >>> --allow-run-as-root --mca mtl mxm -x >>> LD_PRELOAD="./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1 >>> ./hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm/lib/libmxm.so.2" >>> -n 1 ./backend localhost : -x >>> LD_PRELOAD="./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1 >>> ./hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm/lib/libmxm.so.2 >>> ./libci.so" -n 1 ./app2 >>> i am backend >>> [JARVICE:00564] mca: base: components_open: component pml / cm open >>> function failed >>> [JARVICE:564 :0] Caught signal 11 (Segmentation fault) >>> [JARVICE:00565] mca: base: components_open: component pml / cm open >>> function failed >>> [JARVICE:565 :0] Caught signal 11 (Segmentation fault) >>> ==== backtrace ==== >>> 2 0x000000000005640c mxm_handle_error() >>> /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm-v3.2/src/mxm/util/debug/debug.c:641 >>> 3 0x000000000005657c mxm_error_signal_handler() >>> /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u5-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v1.2.0-325-gcc-MLNX_OFED_LINUX-2.4-1.0.0-redhat6.5/mxm-v3.2/src/mxm/util/debug/debug.c:616 >>> 4 0x00000000000329a0 killpg() ??:0 >>> 5 0x0000000000045491 mca_base_components_close() ??:0 >>> 6 0x000000000004e99a mca_base_framework_close() ??:0 >>> 7 0x0000000000045431 mca_base_component_close() ??:0 >>> 8 0x000000000004515c mca_base_framework_components_open() ??:0 >>> 9 0x00000000000a0de9 mca_pml_base_open() pml_base_frame.c:0 >>> 10 0x000000000004eb1c mca_base_framework_open() ??:0 >>> 11 0x0000000000043eb3 ompi_mpi_init() ??:0 >>> 12 0x0000000000067cb0 PMPI_Init_thread() ??:0 >>> 13 0x0000000000404fdf main() /root/rain_ib/backend/backend.c:1237 >>> 14 0x000000000001ed1d __libc_start_main() ??:0 >>> 15 0x0000000000402db9 _start() ??:0 >>> =================== >>> >>> -------------------------------------------------------------------------- >>> A requested component was not found, or was unable to be opened. This >>> means that this component is either not installed or is unable to be >>> used on your system (e.g., sometimes this means that shared libraries >>> that the component requires are unable to be found/loaded). Note that >>> Open MPI stopped checking at the first component that it did not find. >>> >>> Host: JARVICE >>> Framework: mtl >>> Component: mxm >>> >>> -------------------------------------------------------------------------- >>> >>> -------------------------------------------------------------------------- >>> mpirun noticed that process rank 0 with PID 564 on node JARVICE exited >>> on signal 11 (Segmentation fault). >>> >>> -------------------------------------------------------------------------- >>> [JARVICE:00562] 1 more process has sent help message help-mca-base.txt / >>> find-available:not-valid >>> [JARVICE:00562] Set MCA parameter "orte_base_help_aggregate" to 0 to see >>> all help / error messages >>> >>> >>> Subhra >>> >>> >>> On Sun, Apr 12, 2015 at 10:48 PM, Mike Dubman <mi...@dev.mellanox.co.il> >>> wrote: >>> >>>> seems like mxm was not found in your ld_library_path. >>>> >>>> what mofed version do you use? >>>> does it have /opt/mellanox/mxm in it? >>>> You could just run mpirun from HPCX package which looks for mxm >>>> internally and recompile ompi as mentioned in README. >>>> >>>> On Mon, Apr 13, 2015 at 3:24 AM, Subhra Mazumdar < >>>> subhramazumd...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I used mxm mtl as follows but getting segfault. It says mxm component >>>>> not found but I have compiled openmpi with mxm. Any idea what I might be >>>>> missing? >>>>> >>>>> [root@JARVICE ~]# ./openmpi-1.8.4/openmpinstall/bin/mpirun >>>>> --allow-run-as-root --mca pml cm --mca mtl mxm -n 1 -x >>>>> LD_PRELOAD=./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1 ./backend >>>>> localhosst : -n 1 -x LD_PRELOAD="./libci.so >>>>> ./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1" ./app2 >>>>> i am backend >>>>> [JARVICE:08398] *** Process received signal *** >>>>> [JARVICE:08398] Signal: Segmentation fault (11) >>>>> [JARVICE:08398] Signal code: Address not mapped (1) >>>>> [JARVICE:08398] Failing at address: 0x10 >>>>> [JARVICE:08398] [ 0] /lib64/libpthread.so.0(+0xf710)[0x7ff8d0ddb710] >>>>> [JARVICE:08398] [ 1] >>>>> /root/openmpi-1.8.4/openmpinstall/lib/libopen-pal.so.6(mca_base_components_close+0x21)[0x7ff8cf9ae491] >>>>> [JARVICE:08398] [ 2] >>>>> /root/openmpi-1.8.4/openmpinstall/lib/libopen-pal.so.6(mca_base_framework_close+0x6a)[0x7ff8cf9b799a] >>>>> [JARVICE:08398] [ 3] >>>>> /root/openmpi-1.8.4/openmpinstall/lib/libopen-pal.so.6(mca_base_component_close+0x21)[0x7ff8cf9ae431] >>>>> [JARVICE:08398] [ 4] >>>>> /root/openmpi-1.8.4/openmpinstall/lib/libopen-pal.so.6(mca_base_framework_components_open+0x11c)[0x7ff8cf9ae15c] >>>>> [JARVICE:08398] [ 5] >>>>> ./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1(+0xa0de9)[0x7ff8d1089de9] >>>>> [JARVICE:08398] [ 6] >>>>> /root/openmpi-1.8.4/openmpinstall/lib/libopen-pal.so.6(mca_base_framework_open+0x7c)[0x7ff8cf9b7b1c] >>>>> [JARVICE:08398] [ 7] [JARVICE:08398] mca: base: components_open: >>>>> component pml / cm open function failed >>>>> >>>>> ./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1(ompi_mpi_init+0x4b3)[0x7ff8d102ceb3] >>>>> [JARVICE:08398] [ 8] >>>>> ./openmpi-1.8.4/openmpinstall/lib/libmpi.so.1(PMPI_Init_thread+0x100)[0x7ff8d1050cb0] >>>>> [JARVICE:08398] [ 9] ./backend[0x404fdf] >>>>> [JARVICE:08398] [10] >>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7ff8cfeded1d] >>>>> [JARVICE:08398] [11] ./backend[0x402db9] >>>>> [JARVICE:08398] *** End of error message *** >>>>> >>>>> -------------------------------------------------------------------------- >>>>> A requested component was not found, or was unable to be opened. This >>>>> means that this component is either not installed or is unable to be >>>>> used on your system (e.g., sometimes this means that shared libraries >>>>> that the component requires are unable to be found/loaded). Note that >>>>> Open MPI stopped checking at the first component that it did not find. >>>>> >>>>> Host: JARVICE >>>>> Framework: mtl >>>>> Component: mxm >>>>> >>>>> -------------------------------------------------------------------------- >>>>> >>>>> -------------------------------------------------------------------------- >>>>> mpirun noticed that process rank 0 with PID 8398 on node JARVICE >>>>> exited on signal 11 (Segmentation fault). >>>>> >>>>> -------------------------------------------------------------------------- >>>>> >>>>> >>>>> Subhra. >>>>> >>>>> >>>>> On Fri, Apr 10, 2015 at 12:12 AM, Mike Dubman < >>>>> mi...@dev.mellanox.co.il> wrote: >>>>> >>>>>> no need IPoIB, mxm uses native IB. >>>>>> >>>>>> Please see HPCX (pre-compiled ompi, integrated with MXM and FCA) >>>>>> README file for details how to compile/select. >>>>>> >>>>>> The default transport is UD for internode communication and >>>>>> shared-memory for intra-node. >>>>>> >>>>>> http://bgate,mellanox.com/products/hpcx/ >>>>>> >>>>>> Also, mxm included in the Mellanox OFED. >>>>>> >>>>>> On Fri, Apr 10, 2015 at 5:26 AM, Subhra Mazumdar < >>>>>> subhramazumd...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Does ipoib need to be configured on the ib cards for mxm (I have a >>>>>>> separate ethernet connection too)? Also are there special flags in >>>>>>> mpirun >>>>>>> to select from UD/RC/DC? What is the default? >>>>>>> >>>>>>> Thanks, >>>>>>> Subhra. >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 31, 2015 at 9:46 AM, Mike Dubman < >>>>>>> mi...@dev.mellanox.co.il> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> mxm uses IB rdma/roce technologies. Once can select UD/RC/DC >>>>>>>> transports to be used in mxm. >>>>>>>> >>>>>>>> By selecting mxm, all MPI p2p routines will be mapped to >>>>>>>> appropriate mxm functions. >>>>>>>> >>>>>>>> M >>>>>>>> >>>>>>>> On Mon, Mar 30, 2015 at 7:32 PM, Subhra Mazumdar < >>>>>>>> subhramazumd...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi MIke, >>>>>>>>> >>>>>>>>> Does the mxm mtl use infiniband rdma? Also from programming >>>>>>>>> perspective, do I need to use anything else other than >>>>>>>>> MPI_Send/MPI_Recv? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Subhra. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Mar 29, 2015 at 11:14 PM, Mike Dubman < >>>>>>>>> mi...@dev.mellanox.co.il> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> openib btl does not support this thread model. >>>>>>>>>> You can use OMPI w/ mxm (-mca mtl mxm) and multiple thread mode >>>>>>>>>> lin 1.8 x series or (-mca pml yalla) in the master branch. >>>>>>>>>> >>>>>>>>>> M >>>>>>>>>> >>>>>>>>>> On Mon, Mar 30, 2015 at 9:09 AM, Subhra Mazumdar < >>>>>>>>>> subhramazumd...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Can MPI_THREAD_MULTIPLE and openib btl work together in open >>>>>>>>>>> mpi 1.8.4? If so are there any command line options needed during >>>>>>>>>>> run time? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Subhra. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2015/03/26574.php >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Kind Regards, >>>>>>>>>> >>>>>>>>>> M. >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> Link to this post: >>>>>>>>>> http://www.open-mpi.org/community/lists/users/2015/03/26575.php >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org >>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> Link to this post: >>>>>>>>> http://www.open-mpi.org/community/lists/users/2015/03/26580.php >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Kind Regards, >>>>>>>> >>>>>>>> M. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/users/2015/03/26584.php >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26663.php >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Kind Regards, >>>>>> >>>>>> M. >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26665.php >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2015/04/26686.php >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Kind Regards, >>>> >>>> M. >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2015/04/26688.php >>>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2015/04/26711.php >>> >> >> >> >> -- >> >> Kind Regards, >> >> M. >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/04/26712.php >> > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/04/26752.php > -- Kind Regards, M.
