On Thu, Oct 28, 2021 at 10:37 AM Barry Smith <[email protected]> wrote:
> > > On Oct 28, 2021, at 10:31 AM, Matthew Knepley <[email protected]> wrote: > > On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <[email protected]> wrote: > >> >> Matt, >> >> How difficult would it be to rework DMPLEX to allow the use of >> VecGhost? We have performance problems with GPUs with simple DMNETWORK >> models because the code spends more time uselessly copying the local part >> of the vector to another vector in global to local and local to global; >> more than 1/2 the time of the total simulation. >> > > Firedrake already does this because they "vec ghost" their vectors by > default. Here is what you need: > > When you create the PetscSection, by default it orders the unknowns > according to the default point numbering. This > is what causes the ghost unknowns to be mixed in with the > local unknowns. However, PetscSection allows you to set > a point permutation > > > https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html > > This determines the order of dogs by iterating through points in this > permutation, and you can put all shared points at the end. > > > How do I know what are shared points to put at the end? Couldn't DMPLEX > do this automatically with an option? Where is the Firedrake code that does > this with DMPLEX so I can see it? > Yes, it could. I was thinking this as I wrote you, and also thinking how tired I am :) Here is how to do it. Suppose you only care about ghost points (Firedrake wants three layers: totalling internal, roots, and ghosts), then there is a simple test DMGetLocalSection(dm, &s) PetscSectionGetChart(s, &pStart, &pEnd); DMGetPointSF(dm, &sf); PetscSFGetGraph(sf, NULL, &nleaves, &leaves, NULL) for (p = pStart; p < pEnd; ++p) { PetscFindInt(p, nleaves, leaves, &loc); if (loc >= 0) {I am a ghost point} } If you also care about roots (shared points that you own), you also need PetscSFComputeDegreeBegin(sf, &rootdegree); PetscSFComputeDegreeEnd(sf, &rootdegree); and then also check rootdegree[p] > 0 If you get it working, I will put it inside Plex and make an option to turn it on. Thanks, Matt > Does this make sense? > > Thanks, > > Matt > > >> Barry >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
