Which version of hypre do you use? I used petsc's --download-hypre, which automatically installed hypre-2.29.0. I did not see the error.
$ ./heat_diff_cu -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -use_gpu_aware_mpi 0 -mat_type aijcusparse -vec_type cuda The start and end indices of mat for each rank: 0 100 Total time taken for KSP solve: rank: 0 0.0112464 --Junchao Zhang On Tue, Sep 12, 2023 at 10:15 AM Maruthi NH <[email protected]> wrote: > Hi Junchao Zhang, > > Thanks for the help. Updating PETSc fixed the problem. However, if I use > boomeramg from hypre as follows, I get a similar error. > > ./heat_diff_cu --ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg > -use_gpu_aware_mpi 0 -mat_type aijcusparse -vec_type cuda > > ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) had > an illegal value: NULL pointer > > CUSPARSE ERROR (code = 3, invalid value) at csr_matrix_cuda_utils.c:57 > > Regards, > Maruthi > > > > > > On Tue, Sep 12, 2023 at 1:09 AM Junchao Zhang <[email protected]> > wrote: > >> Hi, Maruthi, >> I could run your example on my machine. BTW, I added these at the end >> of main() to free petsc objects. >> VecDestroy(&vout); >> VecDestroy(&x); >> VecDestroy(&b); >> VecDestroy(&u); >> MatDestroy(&A); >> VecScatterDestroy(&ctx); >> KSPDestroy(&ksp); >> >> If you use cuda-12.2, maybe the problem is already fixed by MR >> https://gitlab.com/petsc/petsc/-/merge_requests/6828 >> You can use petsc/main branch to try. Note your petsc version is from >> Date: 2023-08-13 >> >> Thanks. >> --Junchao Zhang >> >> >> On Mon, Sep 11, 2023 at 12:10 PM Maruthi NH <[email protected]> wrote: >> >>> Hi Barry Smith, >>> >>> Thanks for the quick response. >>> >>> Here is the code I used to test PETSc on GPU. >>> This is the command I used to run >>> mpiexec.hydra -n 1 ./heat_diff_cu -Nx 10000000 -ksp_type gmres -mat_type >>> aijcusparse -vec_type cuda -use_gpu_aware_mpi 0 -pc_type gamg >>> -ksp_converged_reason >>> >>> Regards, >>> Maruthi >>> >>> >>> >>> On Sun, Sep 10, 2023 at 11:37 PM Barry Smith <[email protected]> wrote: >>> >>>> >>>> >>>> On Sep 10, 2023, at 5:54 AM, Maruthi NH <[email protected]> wrote: >>>> >>>> Hi all, >>>> >>>> I am trying to accelerate the linear solver with PETSc GPU backend. For >>>> testing I have a simple 1D heat diffusion solver, here are some >>>> observations. >>>> 1. If I use -pc_type gamg it throws the following error >>>> ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) >>>> had an illegal value: NULL pointer >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: GPU error >>>> [0]PETSC ERROR: cuSPARSE errorcode 3 (CUSPARSE_STATUS_INVALID_VALUE) : >>>> invalid value >>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.19.4-959-g92f1e92e88 >>>> GIT Date: 2023-08-13 19:43:04 +0000 >>>> >>>> Can you share the code that triggers this? >>>> >>>> 2. Default pc ilu takes about 1.2 seconds on a single CPU and it takes >>>> about 105.9 seconds on a GPU. Similar observations with pc_type asm >>>> I have NVIDIA RTX A2000 8GB Laptop GPU >>>> >>>> >>>> This is expected. The triangular solves sequentialize on the GPU so >>>> naturally are extremely slow since they cannot take advantage of the >>>> massive parallelism of the GPU. >>>> >>>> >>>> 3. What I could be missing? Also, are there any general guidelines for >>>> better GPU performance using PETSc? >>>> >>>> Regards, >>>> Maruthi >>>> >>>> >>>>
