Are you using
https://urldefense.us/v3/__https://develop.openfoam.com/modules/external-solver__;!!G_uCfscf7eWS!YgXDdUKnfJdPEbKAQzR8TYlQzciWTcMmU_6jrKexdBB6UajcOHUvJhfaBjBtMN2cX6EUFDrcF4LcFItaSLxbAb0$
If so you might be able to add it like
petsc
{
options
{
log_view;
ksp_type cg;
pc_type bjacobi;
sub_pc_type icc;
}
as discussed on the webpage for setting PETSc solver options
> On Jun 14, 2024, at 3:44 PM, Sircar, Arpan <[email protected]> wrote:
>
> Hi Junchao and Barry,
>
> I tried adding -log_view to my OpenFOAM command since that is what I am
> running the entire package through. However, it does not recognize that as a
> valid option. I am not sure where to put that tag in this setup.
> I am however using a Kokkos profiling tool, the output of which (for the GPU
> run) is attached with this email. Do let me know if this is useful or if you
> have ideas of where to put the -log_view tag.
>
> Junchao - Great nice running into you here.
>
> Thanks,
> Arpan
> From: Junchao Zhang <[email protected] <mailto:[email protected]>>
> Sent: Friday, June 14, 2024 3:16 PM
> To: Sircar, Arpan <[email protected] <mailto:[email protected]>>
> Cc: Barry Smith <[email protected] <mailto:[email protected]>>;
> [email protected] <mailto:[email protected]>
> <[email protected] <mailto:[email protected]>>; Gottiparthi,
> Kalyan <[email protected] <mailto:[email protected]>>
> Subject: Re: [petsc-users] [EXTERNAL] Re: Running PETSc with a Kokkos backend
> on OLCF Frontier
>
> Arpan,
> Did you add -log_view ?
>
> --Junchao Zhang
>
>
> On Fri, Jun 14, 2024 at 2:00 PM Sircar, Arpan via petsc-users
> <[email protected] <mailto:[email protected]>> wrote:
> This Message Is From an External Sender
> This message came from outside your organization.
>
> Hi Barry,
>
> Thanks for your prompt response. These are run with with the same PETSc
> solvers but the one on GPUs (log_pKok) uses mataijkokkos while the other one
> does not.
>
> Please let me know if you need any other information.
>
> Thanks,
> Arpan
> From: Barry Smith <[email protected] <mailto:[email protected]>>
> Sent: Friday, June 14, 2024 1:47 PM
> To: Sircar, Arpan <[email protected] <mailto:[email protected]>>
> Cc: [email protected] <mailto:[email protected]>
> <[email protected] <mailto:[email protected]>>; Gottiparthi,
> Kalyan <[email protected] <mailto:[email protected]>>
> Subject: [EXTERNAL] Re: [petsc-users] Running PETSc with a Kokkos backend on
> OLCF Frontier
>
>
> Please run both the CPU solvers and GPU solvers cases with -log_view and
> send the two outputs.
>
> Barry
>
>
>> On Jun 14, 2024, at 1:35 PM, Sircar, Arpan via petsc-users
>> <[email protected] <mailto:[email protected]>> wrote:
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Hi,
>>
>> We have been working with OpenFOAM (an open-source CFD software) which can
>> transfer its matrices to PETSc to use its linear solvers. This has been
>> tested and is working well on OCLF's Frontier machine. Next we are trying to
>> use the Kokkos backend to run it on Frontier GPUs. While the
>> OpenFOAM+PETSc+Kokkos environment built correctly on Frontier using the
>> modules sourced (attached file bash_petsc4foam_gpu) and configuring PETSc
>> correctly (attached file config_gpu), the GPU solve seems to take more time
>> than the CPU solve.
>>
>> The PETSc run-time options we are using are attached to this email (file
>> fvSolution_petsc_pKok_Uof). Could you please take a look and let us know if
>> this combination of options is fine? In this approach we are trying to solve
>> the pressure equation only on the GPUs.
>>
>> Thanks,
>> Arpan
>>
>> Arpan Sircar
>> R&D Associate Staff
>> Thermal Hydraulics Group
>> Nuclear Energy and Fuel Cycle Division
>> Oak Ridge National Laboratory
>>
>> <fvSolution_petsc_pKok_Uof><bash_petsc4foam_gpu><config_gpu>
>
> <log.kok>
