ASM is just the sub PC with one proc but gets weaker with more procs unless you use jacobi. (maybe I am missing something).
On Thu, May 4, 2023 at 8:31 AM Mark Lohry <[email protected]> wrote: > Please send the output of -snes_view. >> > pasted below. anything stand out? > > > SNES Object: 1 MPI process > type: newtonls > maximum iterations=1, maximum function evaluations=-1 > tolerances: relative=0.1, absolute=1e-15, solution=1e-15 > total number of linear solver iterations=20 > total number of function evaluations=22 > norm schedule ALWAYS > Jacobian is never rebuilt > Jacobian is applied matrix-free with differencing > Preconditioning Jacobian is built using finite differences with coloring > SNESLineSearch Object: 1 MPI process > type: basic > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI process > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=20, initial guess is zero > tolerances: relative=0.1, absolute=1e-15, divergence=10. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI process > type: asm > total subdomain blocks = 1, amount of overlap = 0 > restriction/interpolation type - RESTRICT > Local solver information for first block is in the following KSP and > PC objects on rank 0: > Use -ksp_view ::ascii_info_detail to display information for all > blocks > KSP Object: (sub_) 1 MPI process > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (sub_) 1 MPI process > type: ilu > out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1., needed 1. > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqbaij > rows=16384, cols=16384, bs=16 > package used to perform factorization: petsc > total: nonzeros=1277952, allocated nonzeros=1277952 > block size is 16 > linear system matrix = precond matrix: > Mat Object: (sub_) 1 MPI process > type: seqbaij > rows=16384, cols=16384, bs=16 > total: nonzeros=1277952, allocated nonzeros=1277952 > total number of mallocs used during MatSetValues calls=0 > block size is 16 > linear system matrix followed by preconditioner matrix: > Mat Object: 1 MPI process > type: mffd > rows=16384, cols=16384 > Matrix-free approximation: > err=1.49012e-08 (relative error in function evaluation) > Using wp compute h routine > Does not compute normU > Mat Object: 1 MPI process > type: seqbaij > rows=16384, cols=16384, bs=16 > total: nonzeros=1277952, allocated nonzeros=1277952 > total number of mallocs used during MatSetValues calls=0 > block size is 16 > > On Thu, May 4, 2023 at 8:30 AM Mark Adams <[email protected]> wrote: > >> If you are using MG what is the coarse grid solver? >> -snes_view might give you that. >> >> On Thu, May 4, 2023 at 8:25 AM Matthew Knepley <[email protected]> wrote: >> >>> On Thu, May 4, 2023 at 8:21 AM Mark Lohry <[email protected]> wrote: >>> >>>> Do they start very similarly and then slowly drift further apart? >>>> >>>> >>>> Yes, this. I take it this sounds familiar? >>>> >>>> See these two examples with 20 fixed iterations pasted at the end. The >>>> difference for one solve is slight (final SNES norm is identical to 5 >>>> digits), but in the context I'm using it in (repeated applications to solve >>>> a steady state multigrid problem, though here just one level) the >>>> differences add up such that I might reach global convergence in 35 >>>> iterations or 38. It's not the end of the world, but I was expecting that >>>> with -np 1 these would be identical and I'm not sure where the root cause >>>> would be. >>>> >>> >>> The initial KSP residual is different, so its the PC. Please send the >>> output of -snes_view. If your ASM is using direct factorization, then it >>> could be randomness in whatever LU you are using. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> 0 SNES Function norm 2.801842107848e+04 >>>> 0 KSP Residual norm 4.045639499595e+01 >>>> 1 KSP Residual norm 1.917999809040e+01 >>>> 2 KSP Residual norm 1.616048521958e+01 >>>> [...] >>>> 19 KSP Residual norm 8.788043518111e-01 >>>> 20 KSP Residual norm 6.570851270214e-01 >>>> Linear solve converged due to CONVERGED_ITS iterations 20 >>>> 1 SNES Function norm 1.801309983345e+03 >>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1 >>>> >>>> >>>> Same system, identical initial 0 SNES norm, 0 KSP is slightly different >>>> >>>> 0 SNES Function norm 2.801842107848e+04 >>>> 0 KSP Residual norm 4.045639473002e+01 >>>> 1 KSP Residual norm 1.917999883034e+01 >>>> 2 KSP Residual norm 1.616048572016e+01 >>>> [...] >>>> 19 KSP Residual norm 8.788046348957e-01 >>>> 20 KSP Residual norm 6.570859588610e-01 >>>> Linear solve converged due to CONVERGED_ITS iterations 20 >>>> 1 SNES Function norm 1.801311320322e+03 >>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1 >>>> >>>> On Wed, May 3, 2023 at 11:05 PM Barry Smith <[email protected]> wrote: >>>> >>>>> >>>>> Do they start very similarly and then slowly drift further apart? >>>>> That is the first couple of KSP iterations they are almost identical but >>>>> then for each iteration get a bit further. Similar for the SNES >>>>> iterations, >>>>> starting close and then for more iterations and more solves they start >>>>> moving apart. Or do they suddenly jump to be very different? You can run >>>>> with -snes_monitor -ksp_monitor >>>>> >>>>> On May 3, 2023, at 9:07 PM, Mark Lohry <[email protected]> wrote: >>>>> >>>>> This is on a single MPI rank. I haven't checked the coloring, was just >>>>> guessing there. But the solutions/residuals are slightly different from >>>>> run >>>>> to run. >>>>> >>>>> Fair to say that for serial JFNK/asm ilu0/gmres we should expect >>>>> bitwise identical results? >>>>> >>>>> >>>>> On Wed, May 3, 2023, 8:50 PM Barry Smith <[email protected]> wrote: >>>>> >>>>>> >>>>>> No, the coloring should be identical every time. Do you see >>>>>> differences with 1 MPI rank? (Or much smaller ones?). >>>>>> >>>>>> >>>>>> >>>>>> > On May 3, 2023, at 8:42 PM, Mark Lohry <[email protected]> wrote: >>>>>> > >>>>>> > I'm running multiple iterations of newtonls with an MFFD/JFNK >>>>>> nonlinear solver where I give it the sparsity. PC asm, KSP gmres, with >>>>>> SNESSetLagJacobian -2 (compute once and then frozen jacobian). >>>>>> > >>>>>> > I'm seeing slight (<1%) but nonzero differences in residuals from >>>>>> run to run. I'm wondering where randomness might enter here -- does the >>>>>> jacobian coloring use a random seed? >>>>>> >>>>>> >>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>
