Carl, I opened https://github.com/open-mpi/ompi/issues/9444 to specifically track the issue related to the op/avx component
TL;DR nvhpc compilers can compile AVX512 intrinsics (so far so good), but do not define at least one of these macros __AVX512BW__ __AVX512F__ __AVX512VL__ and Open MPI is not happy about it. If you can have all these 3 macros defined by the nvhpc compilers, that would be great! Otherwise, I will let George decide if and how Open MPI addresses this issue Cheers, Gilles On Thu, Sep 30, 2021 at 11:33 PM Carl Ponder via users < users@lists.open-mpi.org> wrote: > > Are you able to fix the problem by adding extern qualifiers? > We could push this change back to the OpenMPI developers... > > ------------------------------ > Subject: Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, > build hints? > Date: Thu, 30 Sep 2021 08:46:26 -0400 > From: Bennet Fauber <ben...@umich.edu> <ben...@umich.edu> > To: Carl Ponder <cpon...@nvidia.com> <cpon...@nvidia.com>, Open MPI Users > <users@lists.open-mpi.org> <users@lists.open-mpi.org> > CC: Ray Muno <m...@aem.umn.edu> <m...@aem.umn.edu> > > *External email: Use caution opening links or attachments* > You may be seeing this? > > C language issues Default to -fno-common > > A common mistake in C is omitting extern when declaring a global variable > in a header file. If the header is included by several files it results in > multiple definitions of the same variable. In previous GCC versions this > error is ignored. GCC 10 defaults to -fno-common, which means a linker > error will now be reported. To fix this, use extern in header files when > declaring global variables, and ensure each global is defined in exactly > one C file. If tentative definitions of particular variables need to be > placed in a common block, __attribute__((__common__)) can be used to > force that behavior even in code compiled without -fcommon. As a > workaround, legacy C code where all tentative definitions should be placed > into a common block can be compiled with -fcommon. > > https://gcc.gnu.org/gcc-10/porting_to.html > <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fgcc-10%2Fporting_to.html&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029906571%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FlRK041uTdnp63G5e4wf%2FzHDGvG57d7eM8tcf%2BuKf%2FA%3D&reserved=0> > > > On Thu, Sep 30, 2021 at 6:56 AM Carl Ponder via users < > users@lists.open-mpi.org> wrote: > >> >> For now, you can suppress this error building OpenMPI 4.1.1 >> >> ./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0): >> multiple definition of `ompi_op_avx_functions_avx2' >> ./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0): >> first defined here >> ./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o): >> In function`ompi_op_avx_2buff_min_uint16_t_avx2': >> /project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651: >> multiple definition of `ompi_op_avx_3buff_functions_avx2' >> >> ./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651: >> first defined here >> >> with the NVHPC/PGI 21.9 compiler by using the setting >> >> configure -*-enable-mca-no-build=op-avx* ... >> >> We're still looking at the cause here. I don't have any advice about the >> problem with 21.7. >> >> ------------------------------ >> Subject: Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, >> build hints? >> Date: Wed, 29 Sep 2021 12:25:43 -0500 >> From: Ray Muno via users <users@lists.open-mpi.org> >> <users@lists.open-mpi.org> >> Reply-To: Open MPI Users <users@lists.open-mpi.org> >> <users@lists.open-mpi.org> >> To: users@lists.open-mpi.org >> CC: Ray Muno <m...@aem.umn.edu> <m...@aem.umn.edu> >> >> External email: Use caution opening links or attachments >> >> >> Tried this >> >> configure CC='nvc -fPIC' CXX='nvc++ -fPIC' FC='nvfortran -fPIC' >> >> Configure completes. Compiles quite a way through. Dies in a different >> place. It does get past the >> first error, however with libmpi_usempif08.la >> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flibmpi_usempif08.la%2F&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029916567%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CfXXt78F6Skl7XZ%2F6tQ9xb%2Bmw0oBfJZRE7N4e2sFVZw%3D&reserved=0> >> >> >> FCLD libmpi_usempif08.la >> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flibmpi_usempif08.la%2F&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029926557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Y0cIKD5ysVINSz3n3DCowM%2F%2Fwu3aKvB%2F38kIOy6tCss%3D&reserved=0> >> make[2]: Leaving directory >> >> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/use-mpi-f08' >> Making all in mpi/fortran/mpiext-use-mpi-f08 >> make[2]: Entering directory >> >> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08' >> PPFC mpi-f08-ext-module.lo >> FCLD libforce_usempif08_module_to_be_built.la >> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flibforce_usempif08_module_to_be_built.la%2F&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029936555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=beQtBZm5vofCdhzP9Bcpv6I5Jk6C%2BvuEwZg0u3O1D28%3D&reserved=0> >> make[2]: Leaving directory >> >> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08' >> >> Dies here now. >> >> CCLD liblocal_ops_avx512.la >> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fliblocal_ops_avx512.la%2F&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029936555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fmeUtItN8b7smy9qCL1bpQjZ1sIDG0hVvw0edT%2BYIBY%3D&reserved=0> >> CCLD mca_op_avx.la >> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmca_op_avx.la%2F&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029946549%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qBp1CsG71YhFazdRnGGknmJOD7L71cJ0URATUXkHjo0%3D&reserved=0> >> ./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0): >> multiple >> definition of `ompi_op_avx_functions_avx2' >> ./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0): >> first defined here >> ./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o): >> In function >> `ompi_op_avx_2buff_min_uint16_t_avx2': >> /project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651: >> multiple >> definition of `ompi_op_avx_3buff_functions_avx2' >> >> ./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651: >> first defined here >> make[2]: *** [mca_op_avx.la >> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmca_op_avx.la%2F&data=04%7C01%7Ccponder%40nvidia.com%7Cf88b7801f927431655ba08d98410580c%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637686028029956544%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SL7eNSZSeb0hZWoRHjkoFWCG0tJzWNx14HKe5X5hZVk%3D&reserved=0>] >> Error 2 >> make[2]: Leaving directory >> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mca/op/avx' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory >> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi' >> make: *** [all-recursive] Error 1 >> >> >> On 9/29/21 11:42 AM, Bennet Fauber via users wrote: >> >> Ray, >> >> If all the errors about not being compiled with -fPIC are still appearing, >> there may be a bug that >> is preventing the option from getting through to the compiler(s). It might >> be worth looking through >> the logs to see the full compile command for one or more of them to see >> whether that is true? Say, >> libs/comm_spawn_multiple_f08.o for example? >> >> If -fPIC is missing, you may be able to recompile that manually with the >> -fPIC in place, then remake >> and see if that also causes the link error to go away, that would be a good >> start. >> >> Hope this helps, -- bennet >> >> >> >> On Wed, Sep 29, 2021 at 12:29 PM Ray Muno via users >> <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> >> <users@lists.open-mpi.org>> wrote: >> >> I did try that and it fails at the same place. >> >> Which version of the nVidia HPC-SDK are you using? I a m using 21.7. I >> see there is an upgrade to >> 21.9, which came out since I installed. I have that installed and will >> try to see if they changed >> anything. Not much in the releases notes to indicate any major changes. >> >> -Ray Muno >> >> >> On 9/29/21 10:54 AM, Jing Gong wrote: >> > Hi, >> > >> > >> > Before Nvidia persons look into details,pProbably you can try to add >> the flag "-fPIC" to the >> > nvhpc compiler likes cc="nvc -fPIC", which at least worked with me. >> > >> > >> > >> > /Jing >> > >> > >> >> ---------------------------------------------------------------------------------------------------- >> > *From:* users <users-boun...@lists.open-mpi.org >> <mailto:users-boun...@lists.open-mpi.org> >> <users-boun...@lists.open-mpi.org>> on >> behalf of Ray Muno via users >> > <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >> <users@lists.open-mpi.org>> >> > *Sent:* Wednesday, September 29, 2021 17:22 >> > *To:* Open MPI User's List >> > *Cc:* Ray Muno >> > *Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia >> HPC-SDk, build hints? >> > Thanks, I looked through previous emails here in the user list. >> Iguess I need to subscribe >> to the >> > Developers list. >> > >> > -Ray Muno >> > >> > On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote: >> >> Ray -- >> >> >> >> Looks like this is a dup of >> https://github.com/open-mpi/ompi/issues/8919 >> <https://github.com/open-mpi/ompi/issues/8919> >> <https://github.com/open-mpi/ompi/issues/8919> >> <https://github.com/open-mpi/ompi/issues/8919 >> <https://github.com/open-mpi/ompi/issues/8919> >> <https://github.com/open-mpi/ompi/issues/8919>> >> >> <https://github.com/open-mpi/ompi/issues/8919 >> <https://github.com/open-mpi/ompi/issues/8919> >> <https://github.com/open-mpi/ompi/issues/8919> >> <https://github.com/open-mpi/ompi/issues/8919 >> <https://github.com/open-mpi/ompi/issues/8919> >> <https://github.com/open-mpi/ompi/issues/8919>>>. >> >> >> >> >> >> > >> -- >> >> Ray Muno >> IT Systems Administrator >> e-mail: m...@umn.edu <mailto:m...@umn.edu> <m...@umn.edu> >> >> University of Minnesota >> Aerospace Engineering and Mechanics >> >> >> >> >> -- >> >> Ray Muno >> IT Systems Administrator >> e-mail: m...@umn.edu >> Phone: (612) 625-9531 >> >> University of Minnesota >> Aerospace Engineering and Mechanics >> 110 Union St. S.E. >> Minneapolis, MN 55455 >> >
