Le 17/05/2022 à 12:03, Dave May a écrit :
On Tue 17. May 2022 at 11:56, Loic Gouarin
<[email protected]> wrote:
Hi Dave,
could you explain what you mean by state ?
Ah - by state Matt and I mean any auxiliary data stored in the user
context passed to KSPSetComputeOperators which is required to define
your operator and is distributed. For example, a Vec which is used
store coefficients of your PDE.
Yes, I have a context which gives the operators (mult and diagonal) and
some other informations to build the matrix free on each multigrid level.
You can find the idea here:
https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91
It's the definition of the operator used inside DMKSPComputeOperators.
Loic
Thanks,
Loic
Le 17/05/2022 à 11:50, Dave May a écrit :
Hi Loic,
Can you confirm if your problem has stored state which needs to
be repartitioned?
Thanks,
Dave
On Tue 17. May 2022 at 09:07, Loic Gouarin
<[email protected]> wrote:
Le 17/05/2022 à 01:43, Dave May a écrit :
On Tue, 17 May 2022 at 00:12, Matthew Knepley
<[email protected]> wrote:
On Mon, May 16, 2022 at 9:59 AM Loic Gouarin
<[email protected]> wrote:
Le 16/05/2022 à 21:54, Barry Smith a écrit :
On May 16, 2022, at 3:50 PM, Loic Gouarin
<[email protected]> wrote:
Thanks Barry for your quick reply.
Le 16/05/2022 à 21:41, Barry Smith a écrit :
Loic,
From your code it looks like you are using a
DM. Is it a DMDA or a shell DM? If it is a DMDA
then the process is intended to be pretty
straightforward. PCTELESCOPE should create a new
DMDA that has the same properties as the coarse
grid DMDA but lives on a smaller number of ranks.
From this you can just provide multiple levels of
coarsening of the DMDA to produce the smaller
multigrid problems.
It's a DMDA. So what you mean is to take the KSP
of PCTELESCOPE, get the DM which is the same of
the coarse grid and build all the levels as
previously for the levels of PCTELESCOPE ?
Yes, I think you should be able to pull out of
the coarse level PC (on the subset of ranks) a DM
that and then setup coarse levels from that. But
you need to look at an example that that uses
telescope and probably even the telescope code
itself to see all the details. I don't know them
anymore.
I looked at the source code but It didn't help. I
will try to explain more clearly my experiments to
understand what I make wrong and you can help more
easily.
Hi Loic!
Here is my guess. When TELESCOPE redistributes the
coarse grid DMDA, it creates a new DMDA and copies over
all the information it can see. However, I think you may
have state that is attached to the old DMDA that is not
getting redistributed. If this is true, I can help you
write a hook that redistributes that state when
TELESCOPE maps between distributions.
If you have state you need to propagate between different
communicators, you will have to use the telescope type which
is internally is referred to as
TELESCOPE_COARSEDM
To activate this type you need the option
-pc_telescope_use_coarse_dm
Or to call PCTelescopeSetUseCoarseDM()
The assumptions of usng this options are described here
https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM
and source associated with the COARSEDM
https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html
Using TELESCOPE_COARSEDM you will have access to the hooks
Matt is referring to.
Thanks Matthew and Dave.
I used TELESCOPE_COARSEDM in my previous tests but it tells
me that "Zero instances of a coarse DM were found". And
indeed, my coarse solver only has the DM on the coarse mesh
and not on the fine level with a coarse DM atttached. I think
it's because I use DMKSPSetComputeOperators to create the KSP
on each level.
So for now, I just added
KSP coarse;
ierr=PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
PC pc_coarse;
ierr=KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
ierr=PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
ierr=PCTelescopeSetUseCoarseDM(pc_coarse,
PETSC_TRUE);CHKERRQ(ierr);
but it's not enough.
Cheers,
Loic
Cheers,
Dave
Thanks,
Matt
Thanks,
Loic
Barry
Can you let us know more details of what
exactly goes wrong with attempts? It is likely
straightforward to get things working, perhaps
due to our unclear documentation.
I essentially have wrong state or NULL matrices
into the PCTELESCOPE.
Loic
Barry
On May 16, 2022, at 3:28 PM, Loic Gouarin
<[email protected]> wrote:
Hello,
I could have posted my message on the user list
but it seems to me that it's more in the petsc
development part. Don't hesitate to tell me if
I'm wrong.
I am developing a code called cafes that studies
fluid-particle interactions in a Stokes fluid.
For that, we implemented the whole solver in
matrix free on a cartesian grid. The
preconditioner is naturally a geometrical
multigrid for the velocity. The set of matrix
free for each level is set by hand (I worked
with Matthew on this a long time ago now!).
I would have liked to implement telescope on
this part. I started it but I didn't get out of
it. So I would like to have your help.
What I tried to do was to describe my classical
multigrid up to a level N1 and then take the
coarse solver and apply the telescope
preconditioner to it up to a level N2. I don't
know if this is the best way but it seemed the
most intuitive. The problem is that I could not
get the coarse matrix because the setup is not
done yet (I use a DMKSPSetComputeOperators for
the matrices).
The construction of the matrix free for each
level is here:
https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88
The description of the Stokes preconditioner is
described here :
https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162
I have tried multiple implementations without
success. For me, the start is to add at line 105
something like
KSP coarse;
ierr = PCMGGetCoarseSolve(pc_i,
&coarse);CHKERRQ(ierr);
PC pc_coarse;
ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
ierr = PCSetType(pc_coarse,
PCTELESCOPE);CHKERRQ(ierr);
Then, to create the several matrices of the
levels of TELESCOPE, I thought using the same
process as before with DMKSPSetComputeOperators.
But it doesn't work...
Could you tell me how I can make it work ?
Thanks,
Loic
--
What most experimenters take for granted before they
begin their experiments is infinitely more interesting
than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
<http://www.cse.buffalo.edu/~knepley/>
