Here is a specific example. I have 10 global edges and 10 global vertices. The
vertices and edges represent neurons and synapses in the brain, respectively.
Vertices 0-7 and 7-9 have excitation and depression effect on the target
vertices. The first rank owns 5 edges (0-4) and 6 vertices ( 0-4 local and 8
ghost). The second rank owns 5 edges (5-9) and 6 vertices (5-9 local and 4
ghost). Note that edges 4 (neuron 8 depresses neuron 4) and 8 (neuron 4 excite
neuron 8) that share the same vertices on different ranks. The generated Plex
is correct based on the cone and support outputs. However, the SF object is
completely wrong as shown below. The first rank supposed to have 10 roots and
one leaf (vertex 8), and the second rank suppose to have the same number of
roots and leaf (vertex 4) as the first rank. My guess is something went wrong
in sfPoint when determining the shared local vertices. I can provide more
outputs from DMPlex if it helps, please let me know.
DM Object: 2 MPI processes
type: network
NSubnets: 1; NEdges: 10; NVertices: 10; NSharedVertices: 0.
[0] nEdges: 5; nVertices: 6; nSharedVertices: 0
Subnet 0: nEdges 5, nVertices(include shared vertices) 6
edge 0: 4 ----> 0
edge 1: 4 ----> 1
edge 2: 4 ----> 2
edge 3: 4 ----> 3
edge 4: 8 ----> 4
[1] nEdges: 5; nVertices: 6; nSharedVertices: 0
Subnet 0: nEdges 5, nVertices(include shared vertices) 6
edge 5: 4 ----> 5
edge 6: 4 ----> 6
edge 7: 4 ----> 7
edge 8: 4 ----> 8
edge 9: 4 ----> 9
Cones:
[0] Max cone size: 2
[0]: 0 <---- 9 (0)
[0]: 0 <---- 5 (0)
[0]: 1 <---- 9 (0)
[0]: 1 <---- 6 (0)
[0]: 2 <---- 9 (0)
[0]: 2 <---- 7 (0)
[0]: 3 <---- 9 (0)
[0]: 3 <---- 8 (0)
[0]: 4 <---- 10 (0)
[0]: 4 <---- 9 (0)
[1] Max cone size: 2
[1]: 0 <---- 5 (0)
[1]: 0 <---- 6 (0)
[1]: 1 <---- 5 (0)
[1]: 1 <---- 7 (0)
[1]: 2 <---- 5 (0)
[1]: 2 <---- 8 (0)
[1]: 3 <---- 5 (0)
[1]: 3 <---- 9 (0)
[1]: 4 <---- 5 (0)
[1]: 4 <---- 10 (0)
PetscSF Object: 2 MPI processes
type: basic
[0] Number of roots=11, leaves=2, remote ranks=1
[0] 9 <- (1,5)
[0] 10 <- (1,9)
[1] Number of roots=11, leaves=0, remote ranks=0
[0] Roots referenced by my leaves, by rank
[0] 1: 2 edges
[0] 9 <- 5
[0] 10 <- 9
[1] Roots referenced by my leaves, by rank
MultiSF sort=rank-order
________________________________
From: Abhyankar, Shrirang G <[email protected]>
Sent: Wednesday, December 1, 2021 4:17 PM
To: Knepley, Matthew G (VISIT) <[email protected]>; Barry Smith
<[email protected]>
Cc: petsc-dev <[email protected]>; Zhang, Hong <[email protected]>;
Betrie, Getnet <[email protected]>
Subject: Re: DMPLEX cannot support two different edges for the same two
vertices, hence DMPLEX cannot?
“You can certainly have many fields on a given edge, but I don't know what it
would mean to have two edges since no topological query could tell the
difference.”
The two edges in a power grid represent two parallel power lines that are
connected between two locations (vertices). There are line ids (stored in the
component data) to distinguish the two lines.
Thanks,
Shri
From: Matthew Knepley <[email protected]>
Date: Wednesday, December 1, 2021 at 4:07 PM
To: Barry Smith <[email protected]>
Cc: "Abhyankar, Shrirang G" <[email protected]>, PETSc Development
<[email protected]>, "Zhang, Hong" <[email protected]>, Getnet Betrie
<[email protected]>
Subject: Re: DMPLEX cannot support two different edges for the same two
vertices, hence DMPLEX cannot?
On Wed, Dec 1, 2021 at 4:51 PM Barry Smith
<[email protected]<mailto:[email protected]>> wrote:
It may not happen with one rank. Or it may.
With more than one rank one may run a code and think it is working correctly
but that does not mean the PetscSF is correct. It is generating "extra" degrees
of freedom for these vertices each associated with a different edge and not
properly connecting them. You can tell by doing a full DMView on the dmplex and
PetscSF and seeing more roots than there should be listed. Hence the numerics
will be wrong even though it does not crash or complain.
Okay, I think we need specific tests to see what is going on.
Theoretically, having multiple edges between vertices does not make sense since
the edge _means_ (v_i, v_j) topologically. You can certainly have many fields
on a given edge, but I don't know what it would mean to have two edges since no
topological query could tell the difference.
However, you can certainly tell Plex anything you want to tell it.
DMPlexBuildFromCellListParallel() gets only cells and vertices, so you are
saying that there is a duplicate cell. That is allowed and should work
properly, EXCEPT if you wanted to identify a cell only by its cone (then you
could not tell them apart). I do this for faces, but I did not think I ever did
it for cells.
Thanks,
Matt
On Dec 1, 2021, at 4:37 PM, Abhyankar, Shrirang G
<[email protected]<mailto:[email protected]>> wrote:
Barry,
“Is there anything we can do to support having multiple edges between the same
two vertices”
Some of my power grid datasets have multiple edges between the same two
vertices and I’ve not faced an issue with DMNetwork. However, all the data was
read on rank 0 only (and then distributed).
Maybe the issue is with the edges being passed on different ranks?
Thanks,
Shri
From: Barry Smith <[email protected]<mailto:[email protected]>>
Date: Wednesday, December 1, 2021 at 3:19 PM
To: "Knepley, Matthew G (VISIT)" <[email protected]<mailto:[email protected]>>,
PETSc Development <[email protected]<mailto:[email protected]>>,
"Abhyankar, Shrirang G"
<[email protected]<mailto:[email protected]>>, "Zhang,
Hong" <[email protected]<mailto:[email protected]>>, Getnet Betrie
<[email protected]<mailto:[email protected]>>
Subject: DMPLEX cannot support two different edges for the same two vertices,
hence DMPLEX cannot?
Check twice before you click! This email originated from outside PNNL.
Matt,
If DMPlexBuildFromCellListParallel() is called with two edges that have
the same two vertices what will happen? It looks like it ends up with an
incorrect PetscSF if the two edges are passed on different ranks. Hence the
DMPLEX is not valid and produces garbage.
Neurons can be connected to themselves which seems to be breaking DMPLEX
and hence DMNETWORK.
Is there anything we can do to support having multiple edges between the
same two vertices? If not is there a way we can have
DMPlexBuildFromCellListParallel() generate an error automatically if there are
such extra edges in the input data.
Thanks
Barry
In this work, the neurons are represented by vertices in the network and each
synapse is a graph edge.
--
What most experimenters take for granted before they begin their experiments is
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=04%7C01%7Cshrirang.abhyankar%40pnnl.gov%7Ce56f4d6ffa10474f3c7508d9b516feb5%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637739932634554446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WwSfLYl2hPyHwrfkJ1SuBmp6l88d6xJulwBPBvu%2BbK0%3D&reserved=0>
