Re: [petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

2018-09-25 Thread Mark Adams
FYI, I've built Barry's updates on SUMMIT and tested it on SUMMITDEV. I can't run on SUMMIT now. It has been merged in the master branch. THis is how you run the cuda tests (in the PETSc root directory): make -f gmakefile.test test globsearch="snes*tutorials*ex19*cuda*" Mark On Thu, Sep 20, 20

Re: [petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

2018-09-20 Thread Smith, Barry F.
Brian, I have finished making the (relatively few) changes needed to get PETSc's GAMG to run on a combination of the CPU and GPU. Any of the AMG kernels that has a CUDA backed is run automatically on the GPU while the kernels without a CUDA backend are run on the CPU. In particular t

Re: [petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

2018-09-19 Thread Mills, Richard Tran
Hi Brian, Your message to petsc-dev has prompted some ongoing discussion among the core PETSc developers, and we'll hopefully be able to give you an outline of a coherent plan to help you meet your ECP milestones soon. We have had adding GPU support within PETSc's GAMG preconditioner on our lis

Re: [petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

2018-09-19 Thread Brian Van Straalen
Hi Jed, Good to hear from you! For the first cases where the geometry is not moving the matrix stays fixed for many many solves and shipping it once to the GPU makes perfect sense. Mark has been helping us with our AMG parameter space. plain GAMG aggregation has worked well for us (hence w

Re: [petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

2018-09-19 Thread Jed Brown
Brian, how frequently do you need to update the matrix (thus rebuild the preconditioner)? If it is infrequent, we could (in the near term) provide AMG setup on CPU with solves on GPU. What is your typical problem size per node to be run on Summit? What is your MPI/OpenMP(?) decomposition? Are t

[petsc-dev] Subsurface application and Algebraic Multigrid on GPUs

2018-09-19 Thread Brian Van Straalen
So Baky and I have been at the Brookhaven GPU Hackathon now for three days, talking to everyone. We have also been emailing with people who will respond to us from the hypre team and the PETSc team, as well as reading every blog post and mail archive and message board and from what we can tell, a