A few things:
0. Rather than go a few more rounds of "how was Open MPI configured", can you
send all the information listed here:
https://www.open-mpi.org/community/help/
That will tell us a lot about exactly how your Open MPI was configured,
installed, etc.
1. Your mpirun error is because you were trying to execute the C source code,
not a binary:
mpirun /opt/openmpi/openmpi-3.0.0_src/examples/hello_c.c
In general, you first have to compile hello_c.c into an executable, and then
you can run that (yes, I know that mpicc is currently failing for you because
of the missing CUDA symbols; I'm just making the point that you can't proceed
to the mpirun step until the mpicc step completes successfully).
2. I would not add/modify /etc/ld.so.conf.d files unless you really, really
have to. LD_LIBRARY_PATH should be sufficient (that's a personal preference of
mine -- modifying /etc/ld.so.conf.d should be an option of last resort and/or
for fairly special circumstances)
3. Your CUDA problems may be indirectly due to one of Open MPI's plugins not
properly disabling CUDA support when the user passes --without-cuda (per
https://github.com/open-mpi/ompi/issues/4219). Specifically: it should be very
possible to solve the CUDA unresolved symbols issue, but avoiding the issue
altogether by disabling CUDA in Open MPI seems to have a bug at the moment.
:-( A few possible workarounds:
3a. Make sure that Open MPI can't find the CUDA headers/libraries when it is
configuring hwloc. This may be a little tricky if your CUDA libraries are in
the default compiler/linker search paths. But it may be possible. For
example, if you're not using CUDA on your systems, uninstall the CUDA
headers/libraries (and then re-configure/build/install Open MPI).
3b. Add the appropriate -L and -l flags to your mpicc command line to find the
CUDA libraries. This is a little cumbersome and exactly the type of thing that
mpicc is supposed to hide from you, but due to #3, Open MPI is inadvertently
still activating CUDA without realizing it (and therefore the wrapper isn't
aware of the CUDA support in Open MPI / didn't add the appropriate -L / -l
flags for CUDA for you). That will resolve the mpicc can't-find-symbols issue,
and then you should be able to mpirun (presuming the CUDA libraries are
findable either by linker default search paths, or you have them in a directory
in your LD_LIBRARY_PATH).
> On Sep 21, 2017, at 3:32 AM, Tim Jim <timothy.m....@gmail.com> wrote:
>
> Hi,
>
> I tried as you suggested: export nvml_enable=no, then reconfigured and ran
> make all install again, but mpicc is still producing the same error. What
> should I try next?
>
> Many thanks,
> Tim
>
> On 21 September 2017 at 16:12, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> Tim,
>
>
> do that in your shell, right before invoking configure.
>
> export nvml_enable=no
>
> ./configure ...
>
> make && make install
>
>
> you can keep the --without-cuda flag (i think this is unrelated though)
>
>
> Cheers,
>
> Gilles
>
> On 9/21/2017 3:54 PM, Tim Jim wrote:
> Dear Gilles,
>
> Thanks for the mail - where should I set export nvml_enable=no? Should I
> reconfigure with default cuda support or keep the --without-cuda flag?
>
> Kind regards,
> Tim
>
> On 21 September 2017 at 15:22, Gilles Gouaillardet <gil...@rist.or.jp
> <mailto:gil...@rist.or.jp>> wrote:
>
> Tim,
>
>
> i am not familiar with CUDA, but that might help
>
> can you please
>
> export nvml_enable=no
>
> and then re-configure and rebuild Open MPI ?
>
>
> i hope this will help you
>
>
> Cheers,
>
>
> Gilles
>
>
>
> On 9/21/2017 3:04 PM, Tim Jim wrote:
>
> Hello,
>
> Apologies to bring up this old thread - I finally had a chance
> to try again with openmpi but I am still have trouble getting
> it to run. I downloaded version 3.0.0 hoping it would solve
> some of the problems but on running mpicc for the previous
> test case, I am still getting an undefined reference error. I
> did as you suggested and also configured it to install without
> cuda using
>
> ./configure --without-cuda --prefix=/opt/openmpi/openmpi-3.0.0
>
> and at the end of the summary, CUDA support shows 'no'.
> Unfortunately, the error is still the same, and for some
> reason, mpicc still seems to have referenced my cuda targets.
>
> tjim@DESKTOP-TA3P0PS:~/Documents$ mpicc
> /opt/mpi4py/mpi4py_src/demo/helloworld.c -o hello.bin
> mpicc:
> /usr/local/cuda-8.0/targets/x86_64-linux/lib/libOpenCL.so.1:
> no version information available (required by
> /opt/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40)
> /opt/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40: undefined
> reference to `clGetPlatformInfo@OPENCL_1.0'
> /opt/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40: undefined
> reference to `clGetPlatformIDs@OPENCL_1.0'
> /opt/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40: undefined
> reference to `clGetDeviceInfo@OPENCL_1.0'
> /opt/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40: undefined
> reference to `clGetDeviceIDs@OPENCL_1.0'
> collect2: error: ld returned 1 exit status
>
> I also attempted to test mpirun, as suggested in the readme,
> however I get the following problem:
>
> tjim@DESKTOP-TA3P0PS:~/Documents$ mpirun
> /opt/openmpi/openmpi-3.0.0_src/examples/hello_c.c
> mpirun:
> /usr/local/cuda-8.0/targets/x86_64-linux/lib/libOpenCL.so.1:
> no version information available (required by
> /opt/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40)
>
> --------------------------------------------------------------------------
> Open MPI tried to fork a new process via the "execve" system
> call but
> failed. Open MPI checks many things before attempting to launch a
> child process, but nothing is perfect. This error may be
> indicative
> of another problem on the target host, or even something as
> silly as
> having specified a directory for your application. Your job
> will now
> abort.
>
> Local host: DESKTOP-TA3P0PS
> Working dir: /home/tjim/Documents
> Application name:
> /opt/openmpi/openmpi-3.0.0_src/examples/hello_c.c
> Error: Exec format error
>
> --------------------------------------------------------------------------
>
> --------------------------------------------------------------------------
> mpirun was unable to start the specified application as it
> encountered an
> error:
>
> Error code: 1
> Error name: (null)
> Node: DESKTOP-TA3P0PS
>
> when attempting to start process rank 0.
>
> --------------------------------------------------------------------------
> 4 total processes failed to start
> [DESKTOP-TA3P0PS:15231] 3 more processes have sent help
> message help-orte-odls-default.txt / execve error
> [DESKTOP-TA3P0PS:15231] Set MCA parameter
> "orte_base_help_aggregate" to 0 to see all help / error messages
>
>
> Do you have any suggestions to what might have gone wrong on
> this install? I'm not sure if this thread is still alive, so
> if you need a refresh on the situation/any more info, please
> let me know.
> Kind regards,
> Tim
>
> On 24 May 2017 at 09:12, Tim Jim <timothy.m....@gmail.com
> <mailto:timothy.m....@gmail.com>
> <mailto:timothy.m....@gmail.com
> <mailto:timothy.m....@gmail.com>>> wrote:
>
> Thanks for the thoughts, I'll give it a go. For reference,
> I have
> installed it in the opt directory, as that is where I have
> kept my
> installs currently. Will this be a problem when calling
> mpi from
> other packages?
>
> Thanks,
> Tim
>
> On 24 May 2017 06:30, "Reuti" <re...@staff.uni-marburg.de
> <mailto:re...@staff.uni-marburg.de>
> <mailto:re...@staff.uni-marburg.de
> <mailto:re...@staff.uni-marburg.de>>> wrote:
>
> Hi,
>
> Am 23.05.2017 um 05:03 schrieb Tim Jim:
>
> > Dear Reuti,
> >
> > Thanks for the reply. What options do I have to test
> whether
> it has successfully built?
>
> LIke before: can you compile and run mpihello.c this
> time –
> all as ordinary user in case you installed the Open
> MPI into
> something like $HOME/local/openmpi-2.1.1 and set paths
> accordingly. There is no need to be root to install a
> personal
> Open MPI version in your home directory.
>
> -- Reuti
>
>
> >
> > Thanks and kind regards.
> > Tim
> >
> > On 22 May 2017 at 19:39, Reuti
> <re...@staff.uni-marburg.de <mailto:re...@staff.uni-marburg.de>
> <mailto:re...@staff.uni-marburg.de
> <mailto:re...@staff.uni-marburg.de>>> wrote:
> > Hi,
> >
> > > Am 22.05.2017 um 07:22 schrieb Tim Jim
> <timothy.m....@gmail.com
> <mailto:timothy.m....@gmail.com>
> <mailto:timothy.m....@gmail.com
>
> <mailto:timothy.m....@gmail.com>>>:
>
> > >
> > > Hello,
> > >
> > > Thanks for your message. I'm trying to get this to
> work on
> a single
> > > machine.
> >
> > Ok.
> >
> >
> > > How might you suggest getting OpenMPIworking without
> python and
> > > CUDA?
> >
> > It looks like it's detected automatically. It should be
> possible to disable it with the command line option:
> >
> > $ ./configure --without-cuda …
> >
> > At the end of the configure step out should liste
> some lines
> like:
> >
> > Miscellaneous
> > -----------------------
> > CUDA support: no
> >
> > The mpi4py seems unrelated to the compilation of
> Open MPI
> itself AFAICS.
> >
> >
> > > I don't recall setting anything for either, as the
> only
> command I had
> > > run was "./configure
> --prefix=/opt/openmpi/openmpi-2.1.0"
> - did it possibly
> > > pick up the paths by accident?
> > >
> > > Regarding the lib directory, I checked that the path
> physically exists.
> > > Regarding the final part of the email, is it a problem
> that 'undefined
> > > reference' is appearing?
> >
> > Yes, it tries to resolve missing symbols and didn't
> succeed.
> >
> > -- Reuti
> >
> >
> > >
> > > Thanks and regards,
> > > Tim
> > >
> > > On 22 May 2017 at 06:54, Reuti
> <re...@staff.uni-marburg.de <mailto:re...@staff.uni-marburg.de>
> <mailto:re...@staff.uni-marburg.de
>
> <mailto:re...@staff.uni-marburg.de>>> wrote:
> > >
> > >> -----BEGIN PGP SIGNED MESSAGE-----
> > >> Hash: SHA1
> > >>
> > >> Hi,
> > >>
> > >> Am 18.05.2017 um 07:44 schrieb Tim Jim:
> > >>
> > >>> Hello,
> > >>>
> > >>> I have been having some issues with trying to get
> OpenMPI working with
> > >> mpi4py. I've tried to break down my
> troubleshooting into
> a few chunks
> > >> below, and I believe that there are a few, distinct
> issues that need
> > >> solving.
> > >>
> > >> Are you speaking here of a single machine or a
> cluster?
> > >>
> > >>
> > >>> Following some troubleshooting in the following
> link:
> > >>>
> https://bitbucket.org/mpi4py/mpi4py/issues/69/building-
> <https://bitbucket.org/mpi4py/mpi4py/issues/69/building->
>
> <https://bitbucket.org/mpi4py/mpi4py/issues/69/building-
> <https://bitbucket.org/mpi4py/mpi4py/issues/69/building->>
> > >> mpi4py-with-openmpi-gives-error
> > >>> -the mpi4py folks have suggested it an issue
> that might
> be better
> > >> answered here.
> > >>
> > >> First approach would be to get Open MPI working,
> without
> CUDA and Python
> > >> being involved.
> > >>
> > >>
> > >>> In summary, I have attempted to install OpenMPI on
> Ubuntu 16.04 to the
> > >> following prefix: /opt/openmpi-openmpi-2.1.0. I
> have also
> manually added
> > >> the following to my .bashrc:
> > >>> export PATH="/opt/openmpi/openmpi-2.1.0/bin:$PATH"
> > >>> MPI_DIR=/opt/openmpi/openmpi-2.1.0
> > >>> export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH
> > >>
> > >> This looks fine, although I don't recall setting
> MPI_DIR
> for Open MPI
> > >> itself. It might be a necessity for mpi4py though.
> > >>
> > >> One pitfall might be that "lib" is sometimes being
> created as "lib64" by
> > >> `libtool`. I forgot the details when this is
> happening,
> but it depends on
> > >> the version of `libtool` being used.
> > >>
> > >>
> > >>> I later became aware that Ubuntu may handle the
> LD_LIBRARY_PATH
> > >> differently
> > >>
> > >> I don't think that Ubuntu will do anything
> different than
> any other Linux.
> > >>
> > >> Did you compile Open MPI on your own, or did you
> install
> any repository?
> > >>
> > >> Are the CUDA application written by yourself or any
> freely available
> > >> applications?
> > >>
> > >> - -- Reuti
> > >>
> > >>
> > >>> and instead added a new file containing the
> library path
> > >> /opt/openmpi/openmpi-2.1.0/lib to
> /etc/ld.so.conf.d/openmpi-2-1-0.conf,
> > >> in the style of everything else in that directory.
> > >>>
> > >>> I tried to run "mpicc helloworld.c -o hello.bin"
> as a
> test on a demo
> > >> file (as instructed in the link) to check the
> installation but I had
> > >> permission issues, since it was installed win opt.
> However, when I
> > >> attempted to run the previous with sudo, or sudo
> -E, in
> both cases, mpicc
> > >> could not be found. (Perhaps this is a separate issue
> with my sudo env)
> > >>>
> > >>> To check that mpicc actually works, I have copied
> helloworld.c to a
> > >> directory where I could execute mpicc without
> sudo. On
> running the above
> > >> command, I receive the following error:
> > >>>
> > >>> mpicc:
>
> /usr/local/cuda-8.0/targets/x86_64-linux/lib/libOpenCL.so.1: no
> > >> version information available (required by
> /opt/openmpi/openmpi-2.1.0/
> > >> lib/libopen-pal.so.20)
> > >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20:
> undefined reference
> > >> to `clGetPlatformInfo@OPENCL_1.0'
> > >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20:
> undefined reference
> > >> to `clGetPlatformIDs@OPENCL_1.0'
> > >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20:
> undefined reference
> > >> to `clGetDeviceInfo@OPENCL_1.0'
> > >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20:
> undefined reference
> > >> to `clGetDeviceIDs@OPENCL_1.0'
> > >>> collect2: error: ld returned 1 exit status
> > >>>
> > >>> I am unsure if I have an installation or permission
> issues, and I'd be
> > >> grateful if anyone can shed some light based on the
> trials I've done so
> > >> far. (I should add I also have a CUDA
> installation, which
> I'd like to
> > >> leverage too, if possible). I'm still fairly new
> to the
> ins and outs of
> > >> this, so I may have missed something obvious.
> Please let
> me know if any
> > >> other info is required.
> > >>>
> > >>> Many thanks and kind regards,
> > >>> Tim
> > >>>
> > >>> --
> > >>>
> > >>> Timothy Jim
> > >>> PhD Researcher in Aerospace
> > >>> Creative Flow Research Division,
> > >>> Institute of Fluid Science, Tohoku University
> > >>> www.linkedin.com/in/timjim/
> <http://www.linkedin.com/in/timjim/>
> <http://www.linkedin.com/in/timjim/
> <http://www.linkedin.com/in/timjim/>>
> > >>> _______________________________________________
> > >>> users mailing list
> > >>> users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>
> <mailto:users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>>
> > >>>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
>
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
> > >>
> > >> -----BEGIN PGP SIGNATURE-----
> > >> Comment: GPGTools - https://gpgtools.org
> > >>
> > >>
>
> iEYEARECAAYFAlkiDKIACgkQo/GbGkBRnRou3wCeNztodNKHRxXBAp5ylz041OMP
> > >> RIwAoLWkjzK1Qr8eWle/PdZGVUsvdN1M
> > >> =R6up
> > >> -----END PGP SIGNATURE-----
> > >> _______________________________________________
> > >> users mailing list
> > >> users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>
> <mailto:users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>>
> > >>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
>
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
> > >>
> > >
> > >
> > >
> > > --
> > >
> > >
> > > *Timothy Jim**PhD Researcher in Aerospace*
> > >
> > > Creative Flow Research Division,
> > > Institute of Fluid Science, Tohoku University
> > >
> > > www.linkedin.com/in/timjim/
> <http://www.linkedin.com/in/timjim/>
> <http://www.linkedin.com/in/timjim/
> <http://www.linkedin.com/in/timjim/>>
> > > _______________________________________________
> > > users mailing list
> > > users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>
> <mailto:users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>>
> > >
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
>
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
> >
> > _______________________________________________
> > users mailing list
> > users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>
> <mailto:users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>>
> >
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
>
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
> >
> >
> >
> > --
> >
> > Timothy Jim
> > PhD Researcher in Aerospace
> > Creative Flow Research Division,
> > Institute of Fluid Science, Tohoku University
> > www.linkedin.com/in/timjim/
> <http://www.linkedin.com/in/timjim/>
> <http://www.linkedin.com/in/timjim/
> <http://www.linkedin.com/in/timjim/>>
> > _______________________________________________
> > users mailing list
> > users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>
> <mailto:users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>>
> >
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
>
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> <mailto:users@lists.open-mpi.org
> <mailto:users@lists.open-mpi.org>>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
>
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
>
>
>
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/users
> <https://lists.open-mpi.org/mailman/listinfo/users>
>
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/users
> <https://lists.open-mpi.org/mailman/listinfo/users>
>
>
>
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
>
>
> --
>
> Timothy Jim
> PhD Researcher in Aerospace
> Creative Flow Research Division,
> Institute of Fluid Science, Tohoku University
> www.linkedin.com/in/timjim/
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
--
Jeff Squyres
jsquy...@cisco.com
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
