Hi Timo,
Thank you for sharing the link for cross-compilation of dealii. In the
same webpage there is a link
<https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html> to
information about cmake toolchain information and in that webpage I found
the following :
Cross Compiling for the Cray Linux Environment
<https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#id14>
<https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#cross-compiling-for-the-cray-linux-environment>
Cross compiling for compute nodes in the Cray Linux Environment can be done
without needing a separate toolchain file.
Specifying -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment on the CMake command
line will ensure that the appropriate build settings and search paths are
configured. The platform will pull its configuration from the current
environment variables and will configure a project to use the compiler
wrappers from the Cray Programming Environment’s PrgEnv-* modules if
present and loaded.
The default configuration of the Cray Programming Environment is to only
support static libraries. This can be overridden and shared libraries
enabled by setting the CRAYPE_LINK_TYPE environment variable to dynamic.
Running CMake without specifying CMAKE_SYSTEM_NAME
<https://cmake.org/cmake/help/latest/variable/CMAKE_SYSTEM_NAME.html#variable:CMAKE_SYSTEM_NAME>
will
run the configure step in host mode assuming a standard Linux environment.
If not overridden, the PrgEnv-* compiler wrappers will end up getting used,
which if targeting the either the login node or compute node, is likely not
the desired behavior. The exception to this would be if you are building
directly on a NID instead of cross-compiling from a login node. If trying
to build software for a login node, you will need to either first unload
the currently loaded PrgEnv-* module or explicitly tell CMake to use the
system compilers in /usr/bin instead of the Cray wrappers. If instead
targeting a compute node is desired, just specify the CMAKE_SYSTEM_NAME
<https://cmake.org/cmake/help/latest/variable/CMAKE_SYSTEM_NAME.html#variable:CMAKE_SYSTEM_NAME>
as
mentioned above.
In my case I checked that PrgEnv-intel is one of the modules loaded by
default. I did specify CMAKE_SYSTEM_NAME earlier while running cmake, when
attempting the installation process on the login node and on the
interactive node. I will now try the following :
1) Compiling with PrgEnv-intel on the compute node with
CMAKE_SYSTEM_NAME=CrayLinuxEnvironment.
2) Unloading PrgEnv-intel and compiling on the login node.
Please let me know if you have any other suggestions. Thank you.
Best,
Aaditya
On Saturday, August 22, 2020 at 3:23:49 PM UTC-4 Timo Heister wrote:
> Hi Aaditya,
>
> it looks like you are cross compiling due to the cray system you are
> on? I don't have first-hand experience, but I know some users have had
> success with it in the past.
>
> The error message stems from the fact that we build a small tool
> "expand_instantiations" during the build process. If you are cross
> compiling, this executable obviously won't work, so the rest of the
> build fails.
>
> Take a look at https://dealii.org/9.2.0/developers/porting.html
> (search for expand_instantiations) and let us know if that helps.
>
> Best,
> Timo
>
>
> On Fri, Aug 21, 2020 at 2:52 PM Aaditya Lakshmanan
> <aadit...@gmail.com> wrote:
> >
> > Hi Jean-Paul,
> > I cleaned the previous installation, deleted all the files generated in
> the build directory and attempted a fresh build process as follows :
> >
> > cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment -DCMAKE_C_COMPILER=cc
> -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DDEAL_II_WITH_MPI=ON
> -DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_P4EST=ON -DDEAL_II_WITH_LAPACK=ON
> -DCMAKE_INSTALL_PREFIX=/global/project/projectdirs/m2360/packagesCPFE/dealii_trial
>
> ../dealii
> >
> > which ran successfully after which I tried installing deal.ii via
> >
> > make -j 4 install
> >
> > which exited with the following error(just as before) :
> >
> > [ 40%] Building CXX object
> bundled/muparser_v2_2_4/CMakeFiles/obj_muparser_release.dir/src/muParserTest.cpp.o
> > [ 40%] Generating data_out_dof_data.inst
> > /bin/sh: expand_instantiations: command not found
> > make[2]: ***
> [source/numerics/CMakeFiles/obj_numerics_inst.dir/build.make:91:
> source/numerics/data_out_dof_data.inst] Error 127
> > make[1]: *** [CMakeFiles/Makefile2:3074:
> source/numerics/CMakeFiles/obj_numerics_inst.dir/all] Error 2
> > make[1]: *** Waiting for unfinished jobs....
> > [ 41%] Built target obj_muparser_debug
> > [ 41%] Generating fe_abf.inst
> >
> > In the build/bin directory there is a file expand_instantiations of the
> following type :
> >
> > expand_instantiations: ELF 64-bit LSB executable, x86-64, version 1
> (SYSV), dynamically linked, interpreter /lib64/l,
> BuildID[sha1]=3ee25c761ad827a88f618b972735b201bc6a2bbe, for GNU/Linux
> 3.2.0, with debug_info, not stripped
> >
> > Is that the file you were referring to earlier? I have attached the
> detailed.log file generated in the build directory. Let me know if you need
> any more information from me. Thank you for your help.
> >
> > Best,
> > Aaditya
> >
> > On Friday, August 21, 2020 at 2:03:59 PM UTC-4 Jean-Paul Pelteret wrote:
> >>
> >> Hi Aaditya,
> >>
> >> Great, let us know if that works for you. You should see the
> expand_instantiations binary appear in the “bin” sub-directory with your
> build folder. If it still doesn’t appear there, then perhaps you could
> create a new build directory and then post the configuration and full build
> logs that come from a fresh build. It could also be that this utility
> program is failing to build.
> >>
> >> Best,
> >> Jean-Paul
> >>
> >> On 21 Aug 2020, at 19:37, Aaditya Lakshmanan <aadit...@gmail.com>
> wrote:
> >>
> >> Hi Jean-Paul,
> >> Thank you for your suggestion. I shall try the installation process
> with fewer processors(perhaps 4) and if even that doesn't work, then I will
> try it in serial.
> >>
> >> Best,
> >> Aaditya
> >>
> >> On Friday, August 21, 2020 at 1:07:54 PM UTC-4 Jean-Paul Pelteret wrote:
> >>>
> >>> Hi Aaditaya,
> >>>
> >>> I don’t think that I’ve ever seen anyone have this problem before. We
> build the expand_instantiations binary, and its supposed to be used when
> expanding files in which we define various combinations of template
> parameters to instantiate templates within the library. If you’re
> interested, you can read more about this here
> >>> https://dealii.org/current/doxygen/deal.II/CodingConventions.html
> >>> under the "Instantiation of templated functions" section.
> >>>
> >>> I’m not at all sure what’s gone wrong, but what might be happening is
> that by building in parallel with so many processes you somehow manage to
> get to an object that requires some template parameter expansions before
> this expand_instantiations binary has finished building. Perhaps you can
> try building in serial (or with fewer process) for a little bit, and then
> cancel once you see that you’re definitely at a point where the main
> library object modules are being compiled.
> >>>
> >>> Perhaps there’s something more that we can do from our side if this is
> indeed the cause of the problem, but that would be a question for Matthias
> (who manages our CMake configuration) to answer. I’m not sure if we can
> somehow delay all other tasks until expand_instantiations binary has been
> built.
> >>>
> >>> I hope that this helps you to resolve the issue.
> >>>
> >>> Best,
> >>> Jean-Paul
> >>>
> >>>
> >>> On 20 Aug 2020, at 16:16, Aaditya Lakshmanan <aadit...@gmail.com>
> wrote:
> >>>
> >>> Hi Everyone,
> >>>
> >>> I have been trying to install deal.ii on NERSC Cori cluster for use on
> the KNL compute nodes using the following procedure :
> >>>
> >>>
> >>>
> >>> Clone and setup :
> >>>
> >>> git clone --single-branch --branch dealii-9.1
> https://github.com/dealii/dealii.git
> >>>
> >>> mkdir build
> >>>
> >>> mkdir dealii_trial
> >>>
> >>> cd build
> >>>
> >>>
> >>>
> >>> Build and install on an interactive node :
> >>>
> >>> cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment -DCMAKE_C_COMPILER=cc
> -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DDEAL_II_WITH_MPI=ON
> -DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_P4EST=ON -DDEAL_II_WITH_LAPACK=ON
> -DCMAKE_INSTALL_PREFIX=/global/project/projectdirs/m2360/packagesCPFE/dealii_trial/
>
> ../dealii
> >>>
> >>> which completes without any issues after which
> >>>
> >>>
> >>>
> >>> make -j 32 install
> >>>
> >>> eventually yields the following error :
> >>>
> >>>
> --------------------------------------------------------------------------
> >>>
> >>> [ 27%] Building CXX object
> bundled/umfpack/AMD/Source/CMakeFiles/obj_amd_long_debug.dir/amd_defaults.cc.o
> >>>
> >>> /bin/sh: expand_instantiations: command not found
> >>>
> >>> make[2]: ***
> [source/numerics/CMakeFiles/obj_numerics_inst.dir/build.make:91:
> source/numerics/data_out_dof_data.inst] Error 127
> >>>
> >>> make[2]: *** Waiting for unfinished jobs....
> >>>
> >>>
> ---------------------------------------------------------------------------
> >>>
> >>> Details on procedure before cloning deal.ii are included below :
> >>>
> >>> (i) I first loaded the necessary modules so that module list displayed
> >>>
> >>> Currently Loaded Modulefiles:
> >>>
> >>> 1) modules/3.2.11.4 16) rca/2.2.20-7.0.1.1_4.46__g8e3fb5b.ari
> >>>
> >>> 2) altd/2.0 17) atp/2.1.3
> >>>
> >>> 3) darshan/3.1.7 18) PrgEnv-intel/6.0.5
> >>>
> >>> 4) intel/19.0.3.199 19) craype-mic-knl
> >>>
> >>> 5) craype-network-aries 20) cray-mpich/7.7.14
> >>>
> >>> 6) craype/2.6.2 21) craype-hugepages2M
> >>>
> >>> 7) udreg/2.3.2-7.0.1.1_3.31__g8175d3d.ari 22) valgrind/3.15.0
> >>>
> >>> 8) ugni/6.0.14.0-7.0.1.1_7.33__ge78e5b0.ari 23) zlib/1.2.11
> >>>
> >>> 9) pmi/5.0.14 24) metis/5.1.0
> >>>
> >>> 10) dmapp/7.1.1-7.0.1.1_4.48__g38cf134.ari 25) parmetis/4.0.3
> >>>
> >>> 11) gni-headers/5.0.12.0-7.0.1.1_6.28__g3b1768f.ari 26)
> superlu-dist/6.1.1
> >>>
> >>> 12) xpmem/2.2.20-7.0.1.1_4.10__g0475745.ari 27) hypre/2.15.1
> >>>
> >>> 13) job/2.2.4-7.0.1.1_3.36__g36b56f4.ari 28) cmake/3.14.4
> >>>
> >>> 14) dvs/2.12_2.2.156-7.0.1.1_8.9__g5aab709e 29) cray-hdf5/1.10.2.0
> >>>
> >>> 15) alps/6.6.58-7.0.1.1_6.4__g437d88db.ari 30) petsc-real-64/3.11.2
> >>>
> >>>
> >>>
> >>> (ii) Set environment variables for dynamic linking
> >>>
> >>> export XTPE_LINK_TYPE=dynamic
> >>>
> >>> export CRAYPE_LINK_TYPE=dynamic
> >>>
> >>>
> >>>
> >>> (iii) p4est installation
> >>>
> >>> ./configure
> --prefix=/global/project/projectdirs/m2360/packagesCPFE/p4est --host=x86_64
> --enable-vtk-zlib=yes --enable-mpi=yes F77=ftn FC=ftn CC=cc CXX=CC
> >>>
> >>> make -j 32
> >>>
> >>> make -j 32 install
> >>>
> >>> export P4EST_DIR=/global/project/projectdirs/m2360/packagesCPFE/p4est
> >>>
> >>> I did initially submit a ticket on NERSC support for assistance with
> the installation process and after a few iterations, the person(
> https://github.com/JBlaschke) helping me out asked me to contact the
> developers. I would really appreciate your help on this.
> >>>
> >>> Thank you.
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> The deal.II project is located at http://www.dealii.org/
> >>> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> >>> ---
> >>> You received this message because you are subscribed to the Google
> Groups "deal.II User Group" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> an email to dealii+un...@googlegroups.com.
> >>> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dealii/aaff8931-185f-46d0-8f2c-eb535ae9428en%40googlegroups.com
> .
> >>>
> >>>
> >>
> >> --
> >> The deal.II project is located at http://www.dealii.org/
> >> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> >> ---
> >> You received this message because you are subscribed to the Google
> Groups "deal.II User Group" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to dealii+un...@googlegroups.com.
> >>
> >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dealii/0b6d6ab9-f630-4517-a2a0-7106604aa391n%40googlegroups.com
> .
> >>
> >>
> > --
> > The deal.II project is located at http://www.dealii.org/
> > For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> > ---
> > You received this message because you are subscribed to the Google
> Groups "deal.II User Group" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to dealii+un...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/dealii/a2e65ec8-9691-4fbc-9722-182304e229d8n%40googlegroups.com
> .
>
>
>
> --
> Timo Heister
> http://www.math.clemson.edu/~heister/
>
--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see
https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dealii/2cf2edfb-f073-436c-a894-f2e924c54881n%40googlegroups.com.
