I am a bit confused. Without the initial guess computation, there are still a bunch of events I don't understand
MatTranspose 79 1.0 4.0598e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMultSym 110 1.0 1.7419e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMatMultNum 90 1.0 1.2640e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMatMatMultSym 20 1.0 1.3049e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatRARtSym 25 1.0 1.2492e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMatTrnMultSym 25 1.0 8.8265e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatTrnMultNum 25 1.0 2.4820e+02 1.0 6.83e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 275 MatTrnMatMultSym 10 1.0 7.2984e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatTrnMatMultNum 10 1.0 9.3128e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 in addition there are many more VecMAXPY then VecMDot (in GMRES they are each done the same number of times) VecMDot 5588 1.0 1.7183e+03 1.0 2.06e+13 1.0 0.0e+00 0.0e+00 0.0e+00 8 10 0 0 0 8 10 0 0 0 12016 VecMAXPY 22412 1.0 8.4898e+03 1.0 4.17e+13 1.0 0.0e+00 0.0e+00 0.0e+00 39 20 0 0 0 39 20 0 0 0 4913 Finally there are a huge number of MatMultAdd 258048 1.0 1.4178e+03 1.0 6.10e+13 1.0 0.0e+00 0.0e+00 0.0e+00 7 29 0 0 0 7 29 0 0 0 43025 Are you making calls to all these routines? Are you doing this inside your MatMult() or before you call KSPSolve? The reason I wanted you to make a simpler run without the initial guess code is that your events are far more complicated than would be produced by GMRES alone so it is not possible to understand the behavior you are seeing without fully understanding all the events happening in the code. Barry > On Jun 14, 2024, at 1:19 AM, Yongzhong Li <[email protected]> > wrote: > > Thanks, I have attached the results without using any KSPGuess. At low > frequency, the iteration steps are quite close to the one with KSPGuess, > specifically > > KSPGuess Object: 1 MPI process > type: fischer > Model 1, size 200 > > However, I found at higher frequency, the # of iteration steps are > significant higher than the one with KSPGuess, I have attahced both of the > results for your reference. > > Moreover, could I ask why the one without the KSPGuess options can be used > for a baseline comparsion? What are we comparing here? How does it relate to > the performance issue/bottleneck I found? “I have noticed that the time taken > by KSPSolve is almost two times greater than the CPU time for matrix-vector > product multiplied by the number of iteration” > > Thank you! > Yongzhong > > From: Barry Smith <[email protected] <mailto:[email protected]>> > Date: Thursday, June 13, 2024 at 2:14 PM > To: Yongzhong Li <[email protected] > <mailto:[email protected]>> > Cc: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>>, > [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>>, Piero Triverio > <[email protected] <mailto:[email protected]>> > Subject: Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance > Issue > > > Can you please run the same thing without the KSPGuess option(s) for a > baseline comparison? > > Thanks > > Barry > > > On Jun 13, 2024, at 1:27 PM, Yongzhong Li <[email protected] > <mailto:[email protected]>> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hi Matt, > > I have rerun the program with the keys you provided. The system output when > performing ksp solve and the final petsc log output were stored in a .txt > file attached for your reference. > > Thanks! > Yongzhong > > From: Matthew Knepley <[email protected] <mailto:[email protected]>> > Date: Wednesday, June 12, 2024 at 6:46 PM > To: Yongzhong Li <[email protected] > <mailto:[email protected]>> > Cc: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>>, > [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>>, Piero Triverio > <[email protected] <mailto:[email protected]>> > Subject: Re: [petsc-maint] Assistance Needed with PETSc KSPSolve Performance > Issue > > 你通常不会收到来自 [email protected] <mailto:[email protected]> 的电子邮件。了解这一点为什么很重要 > <https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!djGfJnEhNJROfsMsBJy5u_KoRKbug55xZ64oHKUFnH2cWku_Th1hwt4TDdoMd8pWYVDzJeqJslMNZwpO3y0Et94d31qk-oCEwo4$> > > On Wed, Jun 12, 2024 at 6:36 PM Yongzhong Li <[email protected] > <mailto:[email protected]>> wrote: > Dear PETSc’s developers, I hope this email finds you well. I am currently > working on a project using PETSc and have encountered a performance issue > with the KSPSolve function. Specifically, I have noticed that the time taken > by KSPSolve is > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Dear PETSc’s developers, > I hope this email finds you well. > I am currently working on a project using PETSc and have encountered a > performance issue with the KSPSolve function. Specifically, I have noticed > that the time taken by KSPSolve is almost two times greater than the CPU time > for matrix-vector product multiplied by the number of iteration steps. I use > C++ chrono to record CPU time. > For context, I am using a shell system matrix A. Despite my efforts to > parallelize the matrix-vector product (Ax), the overall solve time remains > higher than the matrix vector product per iteration indicates when multiple > threads were used. Here are a few details of my setup: > Matrix Type: Shell system matrix > Preconditioner: Shell PC > Parallel Environment: Using Intel MKL as PETSc’s BLAS/LAPACK library, > multithreading is enabled > I have considered several potential reasons, such as preconditioner setup, > additional solver operations, and the inherent overhead of using a shell > system matrix. However, since KSPSolve is a high-level API, I have been > unable to pinpoint the exact cause of the increased solve time. > Have you observed the same issue? Could you please provide some experience on > how to diagnose and address this performance discrepancy? Any insights or > recommendations you could offer would be greatly appreciated. > > For any performance question like this, we need to see the output of your > code run with > > -ksp_view -ksp_monitor_true_residual -ksp_converged_reason -log_view > > Thanks, > > Matt > > Thank you for your time and assistance. > Best regards, > Yongzhong > ----------------------------------------------------------- > Yongzhong Li > PhD student | Electromagnetics Group > Department of Electrical & Computer Engineering > University of Toronto > https://urldefense.us/v3/__http://www.modelics.org__;!!G_uCfscf7eWS!cpzruOwzb5N1ZGsGKL8RbNWCwCC7xZghRWjYeSbdL5VZd4fq0dIKpA21KkD9s30f3YZvEW-b_U_OktT0STuBwcM$ > > <https://urldefense.us/v3/__http://www.modelics.org__;!!G_uCfscf7eWS!cuLttMJEcegaqu461Bt4QLsO4fASfLM5vjRbtyNhWJQiInbjgNwkGNdkFE1ebSbFjOUatYB0-jd2yQWMWzqkDFFjwMvNl3ZKAr8$> > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cpzruOwzb5N1ZGsGKL8RbNWCwCC7xZghRWjYeSbdL5VZd4fq0dIKpA21KkD9s30f3YZvEW-b_U_OktT0QwJXMa4$ > > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!djGfJnEhNJROfsMsBJy5u_KoRKbug55xZ64oHKUFnH2cWku_Th1hwt4TDdoMd8pWYVDzJeqJslMNZwpO3y0Et94d31qkNOuenGA$> > <ksp_petsc_log.txt> > > <ksp_petsc_log.txt><ksp_petsc_log_noguess.txt>
