Dear Gilles, Thank you! Indeed, this appears to address the issue in my test.
I have yet to run a full regression test on our actual code to ensure that there are no other side effects. I don’t expect any, though. ** Interestingly, removing '-lmpi_usempif08 -lmpi_usempi_ignore_tkr’ actually fixes a second regression that I had found only last night and not yet reported. Specifically, I have a very well tested mpi_alltoallv call (part of our standard regression tests). Versions of this exact call live in multiple codes, including libraries. With the unpatched OpenMPI version on the same macbook and in a specific case, this suddenly resulted in: [egr-vwb3-mbp2:3573] *** An error occurred in MPI_Alltoallv [egr-vwb3-mbp2:3573] *** reported by process [1710358529,0] [egr-vwb3-mbp2:3573] *** on communicator MPI COMMUNICATOR 7 SPLIT FROM 0 [egr-vwb3-mbp2:3573] *** MPI_ERR_TRUNCATE: message truncated [egr-vwb3-mbp2:3573] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [egr-vwb3-mbp2:3573] *** and potentially your MPI job) [egr-vwb3-mbp2.local:03572] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal [egr-vwb3-mbp2.local:03572] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages This regression is also fixed. I do not have a simple test case for this behavior yet since I was still investigating it. My plan was to create a test case, but extracting such a case from the larger code we have is more difficult. I can still try to create a test case for this as well but I’ll be watching with great interest if the earlier MPI_IN_PLACE regression can be nailed down and fixed in a more general way. Perhaps the two are related. Thanks again, Best wishes Volker > On Jul 27, 2017, at 10:46 AM, Gilles Gouaillardet <gil...@rist.or.jp> wrote: > > Volker, > > > since you are only using > > include 'mpif.h' > > > a workaround is you edit your /.../share/openmpi/mpifort-wrapper-data.txt > and simply remove '-lmpi_usempif08 -lmpi_usempi_ignore_tkr' > > > Cheers, > > Gilles > > On 7/27/2017 3:28 PM, Volker Blum wrote: >> Thanks! >> >> If you wish, please also keep me posted. >> >> Best wishes >> Volker >> >>> On Jul 27, 2017, at 7:50 AM, Gilles Gouaillardet >>> <gilles.gouaillar...@gmail.com> wrote: >>> >>> Thanks Jeff for your offer, i will contact you off-list later >>> >>> >>> i tried a gcc+gfortran and gcc+ifort on both linux and OS X >>> so far, only gcc+ifort on OS X is failing >>> i will try icc+ifort on OS X from now >>> >>> short story, MPI_IN_PLACE is not recognized as such by the ompi >>> fortran wrapper, and i do not know why. >>> >>> the attached program can be used to evidence the issue. >>> >>> >>> Cheers, >>> >>> Gilles >>> >>> On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum <volker.b...@duke.edu> wrote: >>>> Thanks! That’s great. Sounds like the exact combination I have here. >>>> >>>> Thanks also to George. Sorry that the test did not trigger on a more >>>> standard platform - that would have simplified things. >>>> >>>> Best wishes >>>> Volker >>>> >>>>> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet <gil...@rist.or.jp> >>>>> wrote: >>>>> >>>>> Folks, >>>>> >>>>> >>>>> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka >>>>> clang) and ifort 17.0.4 >>>>> >>>>> >>>>> i will investigate this from now >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Gilles >>>>> >>>>> On 7/27/2017 9:28 AM, George Bosilca wrote: >>>>>> Volker, >>>>>> >>>>>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with >>>>>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the >>>>>> 3.0.0-rc1 tarball and the current master, and you test completes without >>>>>> errors on all cases. >>>>>> >>>>>> Once you figure out an environment where you can consistently replicate >>>>>> the issue, I would suggest to attach to the processes and: >>>>>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches >>>>>> what the C layer expects >>>>>> - what is the collective algorithm used by Open MPI >>>>>> >>>>>> I have a "Fortran 101" level question. When you pass an array a(:) as >>>>>> argument, what exactly gets passed via the Fortran interface to the >>>>>> corresponding C function ? >>>>>> >>>>>> George. >>>>>> >>>>>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum <volker.b...@duke.edu >>>>>> <mailto:volker.b...@duke.edu>> wrote: >>>>>> >>>>>> Thanks! Yes, trying with Intel 2017 would be very nice. >>>>>> >>>>>>> On Jul 26, 2017, at 6:12 PM, George Bosilca <bosi...@icl.utk.edu >>>>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>>>>> No, I don't have (or used where they were available) the Intel >>>>>> compiler. I used clang and gfortran. I can try on a Linux box with >>>>>> the Intel 2017 compilers. >>>>>>> George. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum >>>>>> <volker.b...@duke.edu <mailto:volker.b...@duke.edu>> wrote: >>>>>>> Did you use Intel Fortran 2017 as well? >>>>>>> >>>>>>> (I’m asking because I did see the same issue with a combination >>>>>> of an earlier Intel Fortran 2017 version and OpenMPI on an >>>>>> Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on >>>>>> the same machine. Perhaps I can revive my access to that >>>>>> combination somehow.) >>>>>>> Best wishes >>>>>>> Volker >>>>>>> >>>>>>>> On Jul 26, 2017, at 5:55 PM, George Bosilca >>>>>> <bosi...@icl.utk.edu <mailto:bosi...@icl.utk.edu>> wrote: >>>>>>>> I thought that maybe the underlying allreduce algorithm fails >>>>>> to support MPI_IN_PLACE correctly, but I can't replicate on any >>>>>> machine (including OSX) with any number of processes. >>>>>>>> George. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum >>>>>> <volker.b...@duke.edu <mailto:volker.b...@duke.edu>> wrote: >>>>>>>> Thanks! >>>>>>>> >>>>>>>> I tried ‘use mpi’, which compiles fine. >>>>>>>> >>>>>>>> Same result as with ‘include mpif.h', in that the output is >>>>>>>> >>>>>>>> * MPI_IN_PLACE does not appear to work as intended. >>>>>>>> * Checking whether MPI_ALLREDUCE works at all. >>>>>>>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work. >>>>>>>> >>>>>>>> Hm. Any other thoughts? >>>>>>>> >>>>>>>> Thanks again! >>>>>>>> Best wishes >>>>>>>> Volker >>>>>>>> >>>>>>>>> On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet >>>>>> <gilles.gouaillar...@gmail.com >>>>>> <mailto:gilles.gouaillar...@gmail.com>> wrote: >>>>>>>>> Volker, >>>>>>>>> >>>>>>>>> With mpi_f08, you have to declare >>>>>>>>> >>>>>>>>> Type(MPI_Comm) :: mpi_comm_global >>>>>>>>> >>>>>>>>> (I am afk and not 100% sure of the syntax) >>>>>>>>> >>>>>>>>> A simpler option is to >>>>>>>>> >>>>>>>>> use mpi >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> Gilles >>>>>>>>> >>>>>>>>> Volker Blum <volker.b...@duke.edu >>>>>> <mailto:volker.b...@duke.edu>> wrote: >>>>>>>>>> Hi Gilles, >>>>>>>>>> >>>>>>>>>> Thank you very much for the response! >>>>>>>>>> >>>>>>>>>> Unfortunately, I don’t have access to a different system >>>>>> with the issue right now. As I said, it’s not new; it just keeps >>>>>> creeping up unexpectedly again on different platforms. What >>>>>> puzzles me is that I’ve encountered the same problem with low but >>>>>> reasonable frequency over a period of now over five years. >>>>>>>>>> We can’t require F’08 in our application, unfortunately, >>>>>> since this standard is too new. Since we maintain a large >>>>>> application that has to run on a broad range of platforms, Fortran >>>>>> 2008 would not work for many of our users. In a few years, this >>>>>> will be different, but not yet. >>>>>>>>>> On gfortran: In our own tests, unfortunately, Intel Fortran >>>>>> consistently produced much faster executable code in the past. The >>>>>> latter observation may also change someday, but for us, the >>>>>> performance difference was an important constraint. >>>>>>>>>> I did suspect mpif.h, too. Not sure how to best test this >>>>>> hypothesis, however. >>>>>>>>>> Just replacing >>>>>>>>>> >>>>>>>>>>> include 'mpif.h' >>>>>>>>>>> with >>>>>>>>>>> use mpi_f08 >>>>>>>>>> did not work, for me. >>>>>>>>>> >>>>>>>>>> This produces a number of compilation errors: >>>>>>>>>> >>>>>>>>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 >>>>>> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x >>>>>>>>>> check_mpi_in_place_08.f90(55): error #6303: The assignment >>>>>> operation or the binary expression operation is invalid for the >>>>>> data types of the two operands. [MPI_COMM_WORLD] >>>>>>>>>> mpi_comm_global = MPI_COMM_WORLD >>>>>>>>>> ----------------------^ >>>>>>>>>> check_mpi_in_place_08.f90(57): error #6285: There is no >>>>>> matching specific subroutine for this generic subroutine call. >>>>>> [MPI_COMM_SIZE] >>>>>>>>>> call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr) >>>>>>>>>> ---------^ >>>>>>>>>> check_mpi_in_place_08.f90(58): error #6285: There is no >>>>>> matching specific subroutine for this generic subroutine call. >>>>>> [MPI_COMM_RANK] >>>>>>>>>> call MPI_COMM_RANK(mpi_comm_global, myid, mpierr) >>>>>>>>>> ---------^ >>>>>>>>>> check_mpi_in_place_08.f90(75): error #6285: There is no >>>>>> matching specific subroutine for this generic subroutine call. >>>>>> [MPI_ALLREDUCE] >>>>>>>>>> call MPI_ALLREDUCE(MPI_IN_PLACE, & >>>>>>>>>> ---------^ >>>>>>>>>> check_mpi_in_place_08.f90(94): error #6285: There is no >>>>>> matching specific subroutine for this generic subroutine call. >>>>>> [MPI_ALLREDUCE] >>>>>>>>>> call MPI_ALLREDUCE(check_success, aux_check_success, 1, >>>>>> MPI_LOGICAL, & >>>>>>>>>> ---------^ >>>>>>>>>> check_mpi_in_place_08.f90(119): error #6285: There is no >>>>>> matching specific subroutine for this generic subroutine call. >>>>>> [MPI_ALLREDUCE] >>>>>>>>>> call MPI_ALLREDUCE(test_data(:), & >>>>>>>>>> ------------^ >>>>>>>>>> check_mpi_in_place_08.f90(140): error #6285: There is no >>>>>> matching specific subroutine for this generic subroutine call. >>>>>> [MPI_ALLREDUCE] >>>>>>>>>> call MPI_ALLREDUCE(check_conventional_mpi, >>>>>> aux_check_success, 1, MPI_LOGICAL, & >>>>>>>>>> ------------^ >>>>>>>>>> compilation aborted for check_mpi_in_place_08.f90 (code 1) >>>>>>>>>> >>>>>>>>>> This is an interesting result, however … what might I be >>>>>> missing? Another use statement? >>>>>>>>>> Best wishes >>>>>>>>>> Volker >>>>>>>>>> >>>>>>>>>>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet >>>>>> <gilles.gouaillar...@gmail.com >>>>>> <mailto:gilles.gouaillar...@gmail.com>> wrote: >>>>>>>>>>> Volker, >>>>>>>>>>> >>>>>>>>>>> thanks, i will have a look at it >>>>>>>>>>> >>>>>>>>>>> meanwhile, if you can reproduce this issue on a more >>>>>> mainstream >>>>>>>>>>> platform (e.g. linux + gfortran) please let me know. >>>>>>>>>>> >>>>>>>>>>> since you are using ifort, Open MPI was built with Fortran >>>>>> 2008 >>>>>>>>>>> bindings, so you can replace >>>>>>>>>>> include 'mpif.h' >>>>>>>>>>> with >>>>>>>>>>> use mpi_f08 >>>>>>>>>>> and who knows, that might solve your issue >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> >>>>>>>>>>> Gilles >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum >>>>>> <volker.b...@duke.edu <mailto:volker.b...@duke.edu>> wrote: >>>>>>>>>>>> Dear Gilles, >>>>>>>>>>>> >>>>>>>>>>>> Thank you very much for the fast answer. >>>>>>>>>>>> >>>>>>>>>>>> Darn. I feared it might not occur on all platforms, since >>>>>> my former Macbook >>>>>>>>>>>> (with an older OpenMPI version) no longer exhibited the >>>>>> problem, a different >>>>>>>>>>>> Linux/Intel Machine did last December, etc. >>>>>>>>>>>> >>>>>>>>>>>> On this specific machine, the configure line is >>>>>>>>>>>> >>>>>>>>>>>> ./configure CC=gcc FC=ifort F77=ifort >>>>>>>>>>>> >>>>>>>>>>>> ifort version 17.0.4 >>>>>>>>>>>> >>>>>>>>>>>> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v >>>>>>>>>>>> Configured with: >>>>>> --prefix=/Applications/Xcode.app/Contents/Developer/usr >>>>>>>>>>>> --with-gxx-include-dir=/usr/include/c++/4.2.1 >>>>>>>>>>>> Apple LLVM version 8.1.0 (clang-802.0.42) >>>>>>>>>>>> Target: x86_64-apple-darwin16.6.0 >>>>>>>>>>>> Thread model: posix >>>>>>>>>>>> InstalledDir: >>>>>>>>>>>> >>>>>> >>>>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin >>>>>>>>>>>> The full test program is appended. >>>>>>>>>>>> >>>>>>>>>>>> Compilation: >>>>>>>>>>>> >>>>>>>>>>>> mpif90 check_mpi_in_place.f90 >>>>>>>>>>>> >>>>>>>>>>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90 >>>>>>>>>>>> /usr/local/openmpi-3.0.0rc1/bin/mpif90 >>>>>>>>>>>> >>>>>>>>>>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun >>>>>>>>>>>> /usr/local/openmpi-3.0.0rc1/bin/mpirun >>>>>>>>>>>> >>>>>>>>>>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np >>>>>> 2 a.out >>>>>>>>>>>> * MPI_IN_PLACE does not appear to work as intended. >>>>>>>>>>>> * Checking whether MPI_ALLREDUCE works at all. >>>>>>>>>>>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work. >>>>>>>>>>>> >>>>>>>>>>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np >>>>>> 1 a.out >>>>>>>>>>>> * MPI_IN_PLACE does not appear to work as intended. >>>>>>>>>>>> * Checking whether MPI_ALLREDUCE works at all. >>>>>>>>>>>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work. >>>>>>>>>>>> >>>>>>>>>>>> Hopefully, no trivial mistakes in the testcase. I just >>>>>> spent a few days >>>>>>>>>>>> tracing this issue through a fairly large code, which is >>>>>> where the issue >>>>>>>>>>>> originally arose (and leads to wrong numbers). >>>>>>>>>>>> >>>>>>>>>>>> Best wishes >>>>>>>>>>>> Volker >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Jul 26, 2017, at 9:46 AM, Gilles Gouaillardet >>>>>>>>>>>>> <gilles.gouaillar...@gmail.com >>>>>> <mailto:gilles.gouaillar...@gmail.com>> wrote: >>>>>>>>>>>>> Volker, >>>>>>>>>>>>> >>>>>>>>>>>>> i was unable to reproduce this issue on linux >>>>>>>>>>>>> >>>>>>>>>>>>> can you please post your full configure command line, >>>>>> your gnu >>>>>>>>>>>>> compiler version and the full test program ? >>>>>>>>>>>>> >>>>>>>>>>>>> also, how many mpi tasks are you running ? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> >>>>>>>>>>>>> Gilles >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum >>>>>> <volker.b...@duke.edu <mailto:volker.b...@duke.edu>> wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran >>>>>> 2017 and gcc on a >>>>>>>>>>>>>> current MacOS system. For this version, it seems to me >>>>>> that MPI_IN_PLACE >>>>>>>>>>>>>> returns incorrect results (while other MPI >>>>>> implementations, including some >>>>>>>>>>>>>> past OpenMPI versions, work fine). >>>>>>>>>>>>>> >>>>>>>>>>>>>> This can be seen with a simple Fortran example code, >>>>>> shown below. In the >>>>>>>>>>>>>> test, the values of all entries of an array “test_data” >>>>>> should be 1.0d0 if >>>>>>>>>>>>>> the behavior were as intended. However, the version of >>>>>> OpenMPI I have >>>>>>>>>>>>>> returns 0.d0 instead. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I’ve seen this behavior on some other compute platforms >>>>>> too, in the past, >>>>>>>>>>>>>> so it wasn’t new to me. Still, I thought that this >>>>>> time, I’d ask. Any >>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> Best wishes >>>>>>>>>>>>>> Volker >>>>>>>>>>>>>> >>>>>>>>>>>>>> ! size of test data array >>>>>>>>>>>>>> integer :: n_data >>>>>>>>>>>>>> >>>>>>>>>>>>>> ! array that contains test data for MPI_IN_PLACE >>>>>>>>>>>>>> real*8, allocatable :: test_data(:) >>>>>>>>>>>>>> >>>>>>>>>>>>>> integer :: mpierr >>>>>>>>>>>>>> >>>>>>>>>>>>>> n_data = 10 >>>>>>>>>>>>>> >>>>>>>>>>>>>> allocate(test_data(n_data),stat=mpierr) >>>>>>>>>>>>>> >>>>>>>>>>>>>> ! seed test data array for allreduce call below >>>>>>>>>>>>>> if (myid.eq.0) then >>>>>>>>>>>>>> test_data(:) = 1.d0 >>>>>>>>>>>>>> else >>>>>>>>>>>>>> test_data(:) = 0.d0 >>>>>>>>>>>>>> end if >>>>>>>>>>>>>> >>>>>>>>>>>>>> ! Sum the test_data array over all MPI tasks >>>>>>>>>>>>>> call MPI_ALLREDUCE(MPI_IN_PLACE, & >>>>>>>>>>>>>> test_data(:), & >>>>>>>>>>>>>> n_data, & >>>>>>>>>>>>>> MPI_DOUBLE_PRECISION, & >>>>>>>>>>>>>> MPI_SUM, & >>>>>>>>>>>>>> mpi_comm_global, & >>>>>>>>>>>>>> mpierr ) >>>>>>>>>>>>>> >>>>>>>>>>>>>> ! The value of all entries of test_data should now be >>>>>> 1.d0 on all MPI >>>>>>>>>>>>>> tasks. >>>>>>>>>>>>>> ! If that is not the case, then the MPI_IN_PLACE flag >>>>>> may be broken. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Volker Blum >>>>>>>>>>>>>> Associate Professor >>>>>>>>>>>>>> Ab Initio Materials Simulations >>>>>>>>>>>>>> Duke University, MEMS Department >>>>>>>>>>>>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC >>>>>> 27708, USA >>>>>>>>>>>>>> volker.b...@duke.edu <mailto:volker.b...@duke.edu> >>>>>>>>>>>>>> https://aims.pratt.duke.edu >>>>>>>>>>>>>> +1 (919) 660 5279 <tel:%2B1%20%28919%29%20660%205279> >>>>>>>>>>>>>> Twitter: Aimsduke >>>>>>>>>>>>>> >>>>>>>>>>>>>> Office: 1111 Hudson Hall >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e=> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> users mailing list >>>>>>>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e=> >>>>>>>>>>>> Volker Blum >>>>>>>>>>>> Associate Professor >>>>>>>>>>>> Ab Initio Materials Simulations >>>>>>>>>>>> Duke University, MEMS Department >>>>>>>>>>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC >>>>>> 27708, USA >>>>>>>>>>>> volker.b...@duke.edu <mailto:volker.b...@duke.edu> >>>>>>>>>>>> https://aims.pratt.duke.edu >>>>>>>>>>>> +1 (919) 660 5279 <tel:%2B1%20%28919%29%20660%205279> >>>>>>>>>>>> Twitter: Aimsduke >>>>>>>>>>>> >>>>>>>>>>>> Office: 1111 Hudson Hall >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> users mailing list >>>>>>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=e9pjil1vV3SDa40dQJww0p-d0LhgyQzX_kPNhmz-oUE&s=Y4hrMiRzNuObkpm0vPojCqr6Cx6uS_wLxNyAfUaBz70&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=e9pjil1vV3SDa40dQJww0p-d0LhgyQzX_kPNhmz-oUE&s=Y4hrMiRzNuObkpm0vPojCqr6Cx6uS_wLxNyAfUaBz70&e=> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=e9pjil1vV3SDa40dQJww0p-d0LhgyQzX_kPNhmz-oUE&s=Y4hrMiRzNuObkpm0vPojCqr6Cx6uS_wLxNyAfUaBz70&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=e9pjil1vV3SDa40dQJww0p-d0LhgyQzX_kPNhmz-oUE&s=Y4hrMiRzNuObkpm0vPojCqr6Cx6uS_wLxNyAfUaBz70&e=> >>>>>>>>>> Volker Blum >>>>>>>>>> Associate Professor >>>>>>>>>> Ab Initio Materials Simulations >>>>>>>>>> Duke University, MEMS Department >>>>>>>>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC >>>>>> 27708, USA >>>>>>>>>> volker.b...@duke.edu <mailto:volker.b...@duke.edu> >>>>>>>>>> https://aims.pratt.duke.edu >>>>>>>>>> +1 (919) 660 5279 <tel:%2B1%20%28919%29%20660%205279> >>>>>>>>>> Twitter: Aimsduke >>>>>>>>>> >>>>>>>>>> Office: 1111 Hudson Hall >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=HDubxzHgm3hz7-NQfgobz7rkGf0LWBTlGGqdgSoPCC4&s=2D1Arirt92pKR6i2-4KQKZ8YhSnZ2TPkouQQePHvNf0&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=HDubxzHgm3hz7-NQfgobz7rkGf0LWBTlGGqdgSoPCC4&s=2D1Arirt92pKR6i2-4KQKZ8YhSnZ2TPkouQQePHvNf0&e=> >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=HDubxzHgm3hz7-NQfgobz7rkGf0LWBTlGGqdgSoPCC4&s=2D1Arirt92pKR6i2-4KQKZ8YhSnZ2TPkouQQePHvNf0&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=HDubxzHgm3hz7-NQfgobz7rkGf0LWBTlGGqdgSoPCC4&s=2D1Arirt92pKR6i2-4KQKZ8YhSnZ2TPkouQQePHvNf0&e=> >>>>>>>> Volker Blum >>>>>>>> Associate Professor >>>>>>>> Ab Initio Materials Simulations >>>>>>>> Duke University, MEMS Department >>>>>>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA >>>>>>>> >>>>>>>> volker.b...@duke.edu <mailto:volker.b...@duke.edu> >>>>>>>> https://aims.pratt.duke.edu >>>>>>>> +1 (919) 660 5279 <tel:%2B1%20%28919%29%20660%205279> >>>>>>>> Twitter: Aimsduke >>>>>>>> >>>>>>>> Office: 1111 Hudson Hall >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e= >>>>>>>> >>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e=> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwICAg&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=4W_1_DI3QU9PRT2fhTXyUeXiu7HvTSNzX48E-9ifoTc&s=i3f7Olcbyor4Pu0hv6YlgO10cJ_XvOR13zZn7PPIGto&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwICAg&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=4W_1_DI3QU9PRT2fhTXyUeXiu7HvTSNzX48E-9ifoTc&s=i3f7Olcbyor4Pu0hv6YlgO10cJ_XvOR13zZn7PPIGto&e=> >>>>>>> Volker Blum >>>>>>> Associate Professor >>>>>>> Ab Initio Materials Simulations >>>>>>> Duke University, MEMS Department >>>>>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA >>>>>>> >>>>>>> volker.b...@duke.edu <mailto:volker.b...@duke.edu> >>>>>>> https://aims.pratt.duke.edu >>>>>>> +1 (919) 660 5279 <tel:%2B1%20%28919%29%20660%205279> >>>>>>> Twitter: Aimsduke >>>>>>> >>>>>>> Office: 1111 Hudson Hall >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e= >>>>>>> >>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e=> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>>> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwICAg&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=h6zBsOI45o8fovfy43A2FyCt-fL_yVpNbVSf1OA8CrQ&s=YNADyKbvRnxPmnDVmVlYmsYgZEr8m-etPBXLHPRkflw&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwICAg&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=h6zBsOI45o8fovfy43A2FyCt-fL_yVpNbVSf1OA8CrQ&s=YNADyKbvRnxPmnDVmVlYmsYgZEr8m-etPBXLHPRkflw&e=> >>>>>> >>>>>> Volker Blum >>>>>> Associate Professor >>>>>> Ab Initio Materials Simulations >>>>>> Duke University, MEMS Department >>>>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA >>>>>> >>>>>> volker.b...@duke.edu <mailto:volker.b...@duke.edu> >>>>>> https://aims.pratt.duke.edu >>>>>> +1 (919) 660 5279 <tel:%2B1%20%28919%29%20660%205279> >>>>>> Twitter: Aimsduke >>>>>> >>>>>> Office: 1111 Hudson Hall >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e= >>>>>> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e=> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> users@lists.open-mpi.org >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e= >>>>> _______________________________________________ >>>>> users mailing list >>>>> users@lists.open-mpi.org >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=0X7DUjFfbjVVpI_r3F9MFV4Cs6DpbwqBCobJBfWB-DI&s=i7T7XLEvUGLBfkNjzFjixk1JmT_QB0nGMM2j_uhDWyQ&e= >>>> Volker Blum >>>> Associate Professor >>>> Ab Initio Materials Simulations >>>> Duke University, MEMS Department >>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA >>>> >>>> volker.b...@duke.edu >>>> https://aims.pratt.duke.edu >>>> +1 (919) 660 5279 >>>> Twitter: Aimsduke >>>> >>>> Office: 1111 Hudson Hall >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> users@lists.open-mpi.org >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIFaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=h3HNno_BCKRMuxrZ26zNcyS_JTV7oDefvQtaN7NF3Kk&s=ENCO2vndR26kHOuem57zq1jvk1Tl1ftPBQKcQ2xIpwQ&e= >>> <test_constants.f90>_______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwICAg&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=h3HNno_BCKRMuxrZ26zNcyS_JTV7oDefvQtaN7NF3Kk&s=ENCO2vndR26kHOuem57zq1jvk1Tl1ftPBQKcQ2xIpwQ&e= >> Volker Blum >> Associate Professor >> Ab Initio Materials Simulations >> Duke University, MEMS Department >> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA >> >> volker.b...@duke.edu >> https://aims.pratt.duke.edu >> +1 (919) 660 5279 >> Twitter: Aimsduke >> >> Office: 1111 Hudson Hall >> >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=_NZvZhsoGaDbJEC8nwG5oDMiMtSi6cOdtsEboB-uXKQ&s=gVgnzXkfNMEN3UwBtvjPBhrTJPNJs2ty-7pEpHKvImI&e= >> > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=_NZvZhsoGaDbJEC8nwG5oDMiMtSi6cOdtsEboB-uXKQ&s=gVgnzXkfNMEN3UwBtvjPBhrTJPNJs2ty-7pEpHKvImI&e= Volker Blum Associate Professor Ab Initio Materials Simulations Duke University, MEMS Department 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA volker.b...@duke.edu https://aims.pratt.duke.edu +1 (919) 660 5279 Twitter: Aimsduke Office: 1111 Hudson Hall _______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
