> On Aug 21, 2023, at 4:18 PM, Jonas Lundgren <[email protected]> wrote:
>
> Thanks, Barry!
>
> What solution do you have in mind? I tried a bit myself using SCATTER_FORWARD
> of the input vector x in PCApply_Redistribute, together with allowing for
> nonzero initial guess in KSPPREONLY, but that might not be the best solution
> in a public branch?
Yes, that is basically what I would do.
1) Change
PetscCheck(ksp->guess_zero, PetscObjectComm((PetscObject)ksp), PETSC_ERR_USER,
"Running KSP of preonly doesn't make sense with nonzero initial guess\n\
you probably want a KSP type of Richardson");
to be allowed it if the PC is redistribute
2) if the inner ksp has nonzero initial guess flag then scatter forward the x
When you run, you would need both -ksp_initial_guess_nonzero
-redistribute_initial_guess_nonzero
Does this sound right? If you already have the code, or mostly, you could make
a MR and we could finish it and merge it into main, or I can just do it.
Barry
>
> I guess the big gain is due to the fact that the subsequent solvings of
> state/adjoint problems are done with similar (but not exactly the same)
> linear operator, so that they become almost the same problem. On the other
> hand, the state and adjoint problems are not similar to each other, making
> the solution to one a very bad initial guess to the other.
>
> Again, thank you for your support.
>
> Best regards,
> Jonas Lundgren
>
> Från: Barry Smith <[email protected] <mailto:[email protected]>>
> Skickat: den 21 augusti 2023 22:04
> Till: Jonas Lundgren <[email protected] <mailto:[email protected]>>
> Kopia: [email protected] <mailto:[email protected]>
> Ämne: Re: [petsc-users] (Sub) KSP initial guess with PCREDISTRIBUTE
>
>
> Ok, thanks. Definitely more than I expected.
>
> It is easy to add the support you requested. I'll push a branch latter
> today.
>
> Barry
>
>
>
> On Aug 21, 2023, at 3:28 PM, Jonas Lundgren <[email protected]
> <mailto:[email protected]>> wrote:
>
> Dear Barry,
>
> I have tried what you suggested on a (not too large) test example on a bigger
> cluster that I have access to, using -redistribute_ksp_initial_guess_nonzero
> 0 and 1, respectively. The average timings during the 5 first (design)
> iterations are 8.7 s (state) and 7.9 s (adjoint) for the case with zero
> initial guesses and 5.0 s (state) and 5.7 s (adjoint) for the cases with
> nonzero initial guesses. These solvings are the bottleneck of my program,
> accounting for about 60-90% of the total computational time, depending on
> various parameters. The program is basically consisting of the loop: solve
> state > solve adjoint > update design > repeat. This is repeated for a couple
> of hundred iterations.
>
> From my experience, the number of iterations to convergence in each
> state/adjoint solve will decrease when increasing the (design) iterative
> counter (i.e. the longer the process has gone on for) IF the initial guess is
> the solution to the previous solve. This is because the design update is
> smaller in the end of the process than in the beginning, and a smaller design
> update leads to smaller changes in state/adjoint solution between subsequent
> (design) iterations. This means that the numbers provided above are on the
> low side: most likely the savings can be even more in the end of the design
> process.
>
> Best regards,
> Jonas Lundgren
>
> Från: Barry Smith <[email protected] <mailto:[email protected]>>
> Skickat: den 21 augusti 2023 20:13
> Till: Jonas Lundgren <[email protected] <mailto:[email protected]>>
> Kopia: [email protected] <mailto:[email protected]>
> Ämne: Re: [petsc-users] (Sub) KSP initial guess with PCREDISTRIBUTE
>
>
> When you use 2 KSP so that you can use the previous "solution" as the
> initial guess for the next problem, how much savings do you get? In
> iterations inside PCREDISTRIBUTE and in time (relative to the entire linear
> solver time and relative to the entire application run)? You can get this
> information running with -log_view
>
> That is, run the 2 KSP simulation twice, once with the inner
> KSPSetNonzeroInitialGuess() on and once with it off and compare the times for
> the two cases.
>
> Thanks
>
> Barry
>
> Using KSPSetNonzeroInitialGuess() requires an extra matrix-vector product
> and preconditioner application, so I would like to verify that you have a
> measurable performance improvement with the initial guess.
>
>
>
>
>
>
>
>
>
> On Aug 21, 2023, at 7:06 AM, Jonas Lundgren via petsc-users
> <[email protected] <mailto:[email protected]>> wrote:
>
> Dear PETSc users,
>
> I have a problem regarding the setting of initial guess to KSP when using
> PCREDISTRIBUTE as the preconditioner. (The reason to why I use PCREDISTRIBUTE
> is because I have a lot of fixed DOF in my problem, and PCREDISTRIBUTE
> successfully reduces the problem size and therefore speeds up the solving).
>
> First, some details:
> - I use a version of PETSc 3.19.1
> - The KSP I use is KSPPREONLY, as suggested in the manual pages of
> PCREDISTRIBUTE:https://petsc.org/release/manualpages/PC/PCREDISTRIBUTE/
> - I use KSPBCGSL as sub-KSP. I can perfectly well solve my problem
> using this as my main KSP, but the performance is much worse than when using
> it as my sub-KSP (under KSPPREONLY+PCREDISTRIBUTE) due to the amount of fixed
> DOF
> - I am first solving a state problem, then an adjoint problem using
> the same linear operator.
> - The adjoint vector is used as sensitivity information to update a
> design. After the design update, the state+adjoint problems are solved again
> with a slightly updated linear operator. This is done for hundreds of
> (design) iteration steps
> - I want the initial guess for the state problem to be the state
> solution from the previous (design) iteration, and same for the adjoint
> problem
> - I am aware of the default way of setting a custom initial guess:
> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE) together with providing the actual
> guess in the x vector in the call to KSPSolve(ksp, b, x)
>
> The main problem is that PCREDISTRIBUTE internally doesn’t use the input
> solution vector (x) when calling KSPSolve() for the sub-KSP. It zeroes out
> the solution vector (x) when starting to build x = diag(A)^{-1} b in the
> beginning of PCApply_Redistribute(), and uses red->x as the solution
> vector/initial guess when calling KSPSolve(). Therefore, I cannot reach the
> sub-KSP with an initial guess.
>
> Additionally, KSPPREONLY prohibits the use of having a nonzero initial guess
> (the error message says “it doesn’t make sense”). I guess I can remove the
> line raising this error and recompile the PETSc libraries, but it still won’t
> solve the previously mentioned problem, which seems to be the hard nut to
> crack.
>
> So far, I have found out that if I create 2 KSP object, one each for the
> state and adjoint problems, it is enough with calling
> KSPSetInitialGuessNonzero(subksp, PETSC_TRUE) on the subksp. It seems as if
> the variable red->x in PCApply_Redistribute() is kept untouched in memory
> between calls to the main KSP and therefore is used as (non-zero) initial
> guess to the sub-KSP. This has been verified by introducing
> PetscCall(PetscObjectCompose((PetscObject)pc,"redx",(PetscObject)red->x)); in
> PCApply_Redistribute(), recompiling the PETSc library, and then inserting a
> corresponding PetscObjectQuery((PetscObject)pc, "redx", (PetscObject
> *)&redx); in my own program source file.
>
> However, I would like to only create 1 KSP to be used with both the state and
> adjoint problems (same linear operator), for memory reasons. When I do this,
> the initial guesses are mixed up between the two problems: the initial guess
> for the adjoint problem is the solution to the state problem in the current
> design iteration, and the initial guess for the state problem is the solution
> to the adjoint problem in the previous design iteration. These are very bad
> guesses that increases the time to convergence in each state/adjoint solve.
>
> So, the core of the problem (as far as I can understand) is that I want to
> control the initial guess red->x in PCApply_Redistribute().
>
> The only solution I can think of is to include a call to PetscObjectQuery()
> in PCApply_Redistribute() to obtain a vector with the initial guess from my
> main program. And then I need to keep track of the initial guesses for my
> both problems in my main program myself (minor problem). This is maybe not
> the neatest way, and I do not know if this approach affects the performance
> negatively? Maybe one call each to PetscObjectQuery() and
> PetscObjectCompose() per call to PCApply_Redistribute() is negligible?
>
> Is there another (and maybe simpler) solution to this problem? Maybe I can
> SCATTER_FORWARD the input x vector in PCApply_Redistribute() before it is
> zeroed out, together with allowing for non-zero initial guess in KSPPREONLY?
>
> Any help is welcome!
>
>
> Best regards,
> Jonas Lundgren
