> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <[email protected]> wrote: > > On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <[email protected] > <mailto:[email protected]>> wrote: > > Matt, > > How difficult would it be to rework DMPLEX to allow the use of VecGhost? > We have performance problems with GPUs with simple DMNETWORK models because > the code spends more time uselessly copying the local part of the vector to > another vector in global to local and local to global; more than 1/2 the > time of the total simulation. > > Firedrake already does this because they "vec ghost" their vectors by > default. Here is what you need: > > When you create the PetscSection, by default it orders the unknowns > according to the default point numbering. This > is what causes the ghost unknowns to be mixed in with the local unknowns. > However, PetscSection allows you to set > a point permutation > > > https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html > > <https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html> > > This determines the order of dogs by iterating through points in this > permutation, and you can put all shared points at the end.
How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? > > Does this make sense? > > Thanks, > > Matt > > Barry > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
