NO, NO, NO, Any try or suggestions are meaningful. I appreciate help from all of you. Have a nice day.
Qiyue Lu On Wed, Oct 25, 2023 at 11:57 AM Barry Smith <[email protected]> wrote: > > I apologize, my suggestion of adding the --with-openmp fails in your > environment. You should follow Satish's recommendation of adding the flags > in your makefile exactly as needed. > > If you are looking at building only your sources with openmp - using petsc > formatted makefile [using petsc build rules], > you can specify it via CFLAGS - either in makefile - or on command line. > > > For ex: [this example is using src/ksp/ksp/tutorials/makefile - with the > corresponding make fules] > > [balay@pj01 tutorials]$ make ex2 > mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas > -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector > -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include > -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic > ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib > -L/home/balay/petsc/arch-linux-c-debug/lib > -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 > -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11 > -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ > -lquadmath -o ex2 > [balay@pj01 tutorials]$ make clean > [balay@pj01 tutorials]$ make ex2 CFLAGS=-fopenmp > mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas > -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector > -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include > -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic ex2.c > -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib > -L/home/balay/petsc/arch-linux-c-debug/lib > -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 > -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm -lX11 > -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ > -lquadmath -o ex2 > [balay@pj01 tutorials]$ > > > On Oct 25, 2023, at 12:47 PM, Qiyue Lu <[email protected]> wrote: > > Thanks, however, CUDAFLAGS doesn't work. > Even I removed all -fopenmp string from the make file, it still complains > nvcc doesn't know -fopenmp. It seems by enabling --with-openmp, there is a > background -fopenmp option added without -Xcompiler to any compiler. > > On Wed, Oct 25, 2023 at 11:44 AM Satish Balay <[email protected]> wrote: > >> I guess the flag you are looking for is CUDAFLAGS >> >> >>> >> balay@petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100 >> CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp >> /usr/local/cuda/bin/nvcc -o ex100.o -c >> -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include -ccbin mpicxx >> -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo >> -gencode arch=compute_86,code=sm_86 -Xcompiler -fopenmp >> -I/scratch/balay/petsc/include >> -I/scratch/balay/petsc/arch-linux-c-debug/include >> -I/usr/local/cuda/include `pwd`/ex100.cu >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector >> -fvisibility=hidden -g3 -O0 -fopenmp -Wl,-export-dynamic ex100.o >> -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib >> -L/scratch/balay/petsc/arch-linux-c-debug/lib >> -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 >> -L/usr/local/cuda/lib64/stubs >> -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib >> -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 >> -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart >> -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ >> -lquadmath -o ex100 >> rm ex100.o >> balay@petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ >> <<< >> >> Satish >> >> On Wed, 25 Oct 2023, Qiyue Lu wrote: >> >> > Even with >> > CXXFLAGS=-Xcompiler -fopenmp -std=c++17 >> > LDFLAGS= -Xcompiler -fopenmp >> > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common >> > include ${PETSC_DIR}/lib/petsc/conf/variables >> > include ${PETSC_DIR}/lib/petsc/conf/rules >> > >> > won't work. >> > >> > On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <[email protected]> wrote: >> > >> > > Thanks for your reply, using this configurations: >> > > >> > > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >> > > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1 >> --with-openmp=1 >> > > --with-threadsafety=1* >> > > However, I got an error like: >> > > *nvcc fatal : Unknown option '-fopenmp'* >> > > Previously, when I don't have --with-openmp for the configuration, the >> > > PETSc make system can build my *.cu code using nvcc and g++, of >> course, >> > > OpenMP doesn't work. Now with this --with-openmp option, it cannot >> even >> > > build. The interesting thing is, I got this error even after removing >> the >> > > *-fopenmp* from *CXXFLAGS* contents: >> > > CXXFLAGS=-std=c++17 >> > > LDFLAGS= >> > > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common >> > > include ${PETSC_DIR}/lib/petsc/conf/variables >> > > include ${PETSC_DIR}/lib/petsc/conf/rules >> > > >> > > >> > > >> > > Thanks, >> > > Qiyue Lu >> > > >> > > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <[email protected]> >> wrote: >> > > >> > >> >> > >> On Wed, 25 Oct 2023, Qiyue Lu wrote: >> > >> >> > >> > Hello, >> > >> > I have an in-house code enabled OpenMP and it works. Now I am >> trying to >> > >> > incorporate PETSc as the linear solver and build together using the >> > >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I >> found the >> > >> > OpenMP part doesn't work anymore. >> > >> >> > >> If you are looking at building only your sources with openmp - using >> > >> petsc formatted makefile [using petsc build rules], >> > >> you can specify it via CFLAGS - either in makefile - or on command >> line. >> > >> >> > >> >>>>>>> >> > >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with >> the >> > >> corresponding make fules] >> > >> >> > >> [balay@pj01 tutorials]$ make ex2 >> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas >> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector >> > >> -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include >> > >> -I/home/balay/petsc/arch-linux-c-debug/include >> -Wl,-export-dynamic >> > >> ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib >> > >> -L/home/balay/petsc/arch-linux-c-debug/lib >> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib >> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 >> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm >> -lX11 >> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath >> -lstdc++ >> > >> -lquadmath -o ex2 >> > >> [balay@pj01 tutorials]$ make clean >> > >> [balay@pj01 tutorials]$ make ex2 CFLAGS=-fopenmp >> > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas >> > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector >> > >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include >> > >> -I/home/balay/petsc/arch-linux-c-debug/include >> -Wl,-export-dynamic >> > >> ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib >> > >> -L/home/balay/petsc/arch-linux-c-debug/lib >> > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib >> > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 >> > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm >> -lX11 >> > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath >> -lstdc++ >> > >> -lquadmath -o ex2 >> > >> [balay@pj01 tutorials]$ >> > >> <<<<< >> > >> >> > >> Satish >> > >> >> > >> >> > >> > Should I re-configure the petsc installation with --with-openmp=1 >> > >> option? I >> > >> > wonder are the building rules affected by this missing option? >> > >> > >> > >> > Thanks, >> > >> > Qiyue Lu >> > >> > >> > >> >> > >> >> > >> > >
