On Tue 17. May 2022 at 11:56, Loic Gouarin <[email protected]> wrote:
> Hi Dave, > > could you explain what you mean by state ? > Ah - by state Matt and I mean any auxiliary data stored in the user context passed to KSPSetComputeOperators which is required to define your operator and is distributed. For example, a Vec which is used store coefficients of your PDE. Thanks, > > Loic > Le 17/05/2022 à 11:50, Dave May a écrit : > > Hi Loic, > > Can you confirm if your problem has stored state which needs to be > repartitioned? > > Thanks, > Dave > > On Tue 17. May 2022 at 09:07, Loic Gouarin <[email protected]> > wrote: > >> >> Le 17/05/2022 à 01:43, Dave May a écrit : >> >> >> >> On Tue, 17 May 2022 at 00:12, Matthew Knepley <[email protected]> wrote: >> >>> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin < >>> [email protected]> wrote: >>> >>>> Le 16/05/2022 à 21:54, Barry Smith a écrit : >>>> >>>> >>>> >>>> On May 16, 2022, at 3:50 PM, Loic Gouarin < >>>> [email protected]> wrote: >>>> >>>> Thanks Barry for your quick reply. >>>> Le 16/05/2022 à 21:41, Barry Smith a écrit : >>>> >>>> >>>> Loic, >>>> >>>> From your code it looks like you are using a DM. Is it a DMDA or a >>>> shell DM? If it is a DMDA then the process is intended to be pretty >>>> straightforward. PCTELESCOPE should create a new DMDA that has the same >>>> properties as the coarse grid DMDA but lives on a smaller number of ranks. >>>> From this you can just provide multiple levels of coarsening of the DMDA to >>>> produce the smaller multigrid problems. >>>> >>>> It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get >>>> the DM which is the same of the coarse grid and build all the levels as >>>> previously for the levels of PCTELESCOPE ? >>>> >>>> >>>> Yes, I think you should be able to pull out of the coarse level PC >>>> (on the subset of ranks) a DM that and then setup coarse levels from that. >>>> But you need to look at an example that that uses telescope and probably >>>> even the telescope code itself to see all the details. I don't know them >>>> anymore. >>>> >>>> I looked at the source code but It didn't help. I will try to explain >>>> more clearly my experiments to understand what I make wrong and you can >>>> help more easily. >>>> >>> Hi Loic! >>> >>> Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it >>> creates a new DMDA and copies over all the information it can see. However, >>> I think you may have state that is attached to the old DMDA that is not >>> getting redistributed. If this is true, I can help you write a hook that >>> redistributes that state when TELESCOPE maps between distributions. >>> >> >> If you have state you need to propagate between different communicators, >> you will have to use the telescope type which is internally is referred to >> as >> >> TELESCOPE_COARSEDM >> >> >> To activate this type you need the option >> >> -pc_telescope_use_coarse_dm >> >> Or to call PCTelescopeSetUseCoarseDM() >> The assumptions of usng this options are described here >> https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM >> >> and source associated with the COARSEDM >> >> https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html >> >> Using TELESCOPE_COARSEDM you will have access to the hooks Matt is >> referring to. >> >> Thanks Matthew and Dave. >> >> I used TELESCOPE_COARSEDM in my previous tests but it tells me that "Zero >> instances of a coarse DM were found". And indeed, my coarse solver only has >> the DM on the coarse mesh and not on the fine level with a coarse DM >> atttached. I think it's because I use DMKSPSetComputeOperators to create >> the KSP on each level. >> >> So for now, I just added >> KSP coarse; >> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); >> PC pc_coarse; >> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); >> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); >> ierr = PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr); >> >> but it's not enough. >> >> Cheers, >> >> Loic >> >> >> Cheers, >> Dave >> >> >> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> >>>> Loic >>>> >>>> >>>> Barry >>>> >>>> >>>> Can you let us know more details of what exactly goes wrong with >>>> attempts? It is likely straightforward to get things working, perhaps due >>>> to our unclear documentation. >>>> >>>> I essentially have wrong state or NULL matrices into the PCTELESCOPE. >>>> >>>> Loic >>>> >>>> >>>> Barry >>>> >>>> >>>> On May 16, 2022, at 3:28 PM, Loic Gouarin < >>>> [email protected]> wrote: >>>> >>>> Hello, >>>> >>>> I could have posted my message on the user list but it seems to me that >>>> it's more in the petsc development part. Don't hesitate to tell me if I'm >>>> wrong. >>>> I am developing a code called cafes that studies fluid-particle >>>> interactions in a Stokes fluid. For that, we implemented the whole solver >>>> in matrix free on a cartesian grid. The preconditioner is naturally a >>>> geometrical multigrid for the velocity. The set of matrix free for each >>>> level is set by hand (I worked with Matthew on this a long time ago now!). >>>> I would have liked to implement telescope on this part. I started it >>>> but I didn't get out of it. So I would like to have your help. >>>> What I tried to do was to describe my classical multigrid up to a level >>>> N1 and then take the coarse solver and apply the telescope preconditioner >>>> to it up to a level N2. I don't know if this is the best way but it seemed >>>> the most intuitive. The problem is that I could not get the coarse matrix >>>> because the setup is not done yet (I use a DMKSPSetComputeOperators for the >>>> matrices). >>>> >>>> The construction of the matrix free for each level is here: >>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88 >>>> >>>> The description of the Stokes preconditioner is described here : >>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162 >>>> >>>> I have tried multiple implementations without success. For me, the >>>> start is to add at line 105 something like >>>> KSP coarse; >>>> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); >>>> PC pc_coarse; >>>> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); >>>> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); >>>> >>>> Then, to create the several matrices of the levels of TELESCOPE, I >>>> thought using the same process as before with DMKSPSetComputeOperators. But >>>> it doesn't work... >>>> Could you tell me how I can make it work ? >>>> >>>> Thanks, >>>> Loic >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>
