On Fri, Apr 14, 2023 at 8:54 AM Jeremy Theler <[email protected]> wrote:
> Hi Mark. So glad you answered. > > > 0) what is your test problem? eg, 3D Lapacian with Q1 finite > > elements. > > I said in my first email it was linear elasticty (and I gave a link > where you can see the geometry, BCs, etc.) but I did not specifty > further details. > OK, this is fine. Nice 3D tet mesh with P2 (P1 will lock). I assume you have a benign Poisson ratio. > It is linear elasticity with displacement-based FEM formulation using > unstructured curved 10-noded tetrahedra. > > The matrix is marked as SPD with MatSetOption() and the solver is > indeed CG and not the default GMRES. > > good > > First, you can get GAMG diagnostics by running with '-info :pc' and > > grep on GAMG. > > Great advice. Now I have a lot more of information but I'm not sure how > to analyze it. Find attached for each combination of threshold and > PETSc version the output of -info :pc -ksp_monitor -ksp_view > > (new_py-env) 07:43 ~/Downloads/log 2$ grep "grid complexity" * .... infopc-0.02-17.log:[0] PCSetUp_GAMG(): (null): 7 levels, grid complexity = 1.28914 infopc-0.02-17.log: Complexity: grid = 1.04361 operator = 1.28914 infopc-0.02-18.log: Complexity: grid = 1.05658 operator = 1.64555 infopc-0.02-19.log: Complexity: grid = 1.05658 operator = 1.64555 I was using "grid complexity " and changed to the accepted term "operator". You can see that the new coarening method is pretty different on this problem. For an isiotropoic problem like this a zero threshould is a good place to start, but you can use it to tune. > In general it looks like 3.18 and 3.19 have less KSP iterations than > 3.17 but the overall time is larger. > We need to see the solve times. Run with -log_view and grep for KSPSolve We can look at the setup time separately In practice the setup time is amortized unless you use a full Newton nonlinear solver. Your iteration counts are reasonable. They go up a little with Jacobi. Here are your grid sizes: (new_py-env) 07:55 ~/Downloads/log 2$ grep N= infopc-0.0001-19.log [0] <pc> PCSetUp_GAMG(): (null): level 0) N=568386, n data rows=3, n data cols=6, nnz/row (ave)=82, np=1 [0] <pc> PCSetUp_GAMG(): (null): 1) N=22206, n data cols=6, nnz/row (ave)=445, 1 active pes [0] <pc> PCSetUp_GAMG(): (null): 2) N=2628, n data cols=6, nnz/row (ave)=1082, 1 active pes [0] <pc> PCSetUp_GAMG(): (null): 3) N=180, n data cols=6, nnz/row (ave)=180, 1 active pes [0] <pc> PCSetUp_GAMG(): (null): 4) N=18, n data cols=6, nnz/row (ave)=18, 1 active pes [0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 3 (N=180) with emax = 2.0785 emin = 0.0468896 [0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 2 (N=2628) with emax = 1.78977 emin = 5.91431e-08 [0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 1 (N=22206) with emax = 2.12541 emin = 4.27581e-09 [0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 0 (N=568386) with emax = 3.42185 emin = 0.0914406 (new_py-env) 07:56 ~/Downloads/log 2$ grep N= infopc-0.0001-15.log [0] PCSetUp_GAMG(): level 0) N=568386, n data rows=3, n data cols=6, nnz/row (ave)=82, np=1 [0] PCSetUp_GAMG(): 1) N=15642, n data cols=6, nnz/row (ave)=368, 1 active pes [0] PCSetUp_GAMG(): 2) N=1266, n data cols=6, nnz/row (ave)=468, 1 active pes [0] PCSetUp_GAMG(): 3) N=108, n data cols=6, nnz/row (ave)=108, 1 active pes [0] PCSetUp_GAMG(): 4) N=12, n data cols=6, nnz/row (ave)=12, 1 active pes The new version coarsens slower. BTW. Use something like -pc_gamg_coarse_eq_limit 1000 Your coarse grids are too small. You can grep on MatLUFactor to check that the coarse grid solve/factor is under control but 1000 in 3D is pretty conservative. I am sure you are are going to want more aggresive coarsening (newer versions): -pc_gamg_aggressive_coarsening <1> Just try 10 (ie, all levels) to start. Mark > > Anyway, sorry for the changes. > > I hate changing GAMG for this reason and I hate AMG for this reason! > > No need to apologize, I just want to better understand how to better > exploit your code! > > Thanks > -- > jeremy > > > > > Thanks, > > Mark > > > > > > > > On Thu, Apr 13, 2023 at 8:17 AM Jeremy Theler <[email protected]> > > wrote: > > > When using GAMG+cg for linear elasticity and providing the near > > > nullspace computed by MatNullSpaceCreateRigidBody(), I used to find > > > "experimentally" that a small value of -pc_gamg_threshold in the > > > order > > > of 0.0001 would slightly decrease the solve time. > > > > > > Starting with 3.18, I started seeing that any positive value for > > > the > > > treshold would increase the solve time. I did a quick parametric > > > (serial) run solving an elastic problem with a matrix size of > > > approx > > > 570k x 570k for different values of GAMG threshold and different > > > PETSc > > > versions (compiled with the same compiler, options and flags). > > > > > > I noted that > > > > > > 1. starting from 3.18, a threshold of 0.0001 that used to improve > > > the > > > speed now worsens it. > > > 2. PETSc 3.17 looks like a "sweet spot" of speed > > > > > > I would like to hear any comments you might have. > > > > > > The wall time shown includes the time needed to read the mesh and > > > assemble the stiffness matrix. It is a refined version of the > > > NAFEMS > > > LE10 benchmark described here: > > > https://seamplex.com/feenox/examples/mechanical.html#nafems-le10- > > > thick-plate-pressure-benchmark > > > > > > If you want, I could dump the matrix, rhs and near nullspace > > > vectors > > > and share them. > > > > > > -- > > > jeremy theler > > > > >
