I did not know about -log_trace. Good to know, Thanks On Fri, Apr 23, 2021 at 8:19 AM Stefano Zampini <[email protected]> wrote:
> > > >> I'm not talking about timings, I am talking about seeing where a code >> dies. >> > > When codes die within the MPI implementation is basically impossible to > provide a backtrace from PETSc, but the MPI implementation may offer some > verbose messaging. > If this happens quite early in the code (and I presume here we are failing > somewhere in the 1-to-all pattern when distributing the plex data > structure) running with -log_trace may get you some insight on where this > is happening (provided that the offending PETSc routine calls > LogEventBegin/End) > > In my experience, the sad story is that when there are software > vendor failures with a large number of processes, I haven't found anything > more efficient than a few bisection steps with good old printfs.... > > >> >> >>> As a side note, one would expect that this kind of behavior is >>> consistent across the library. >>> But in fact, it isn’t. If you do something like -sub_ksp_view_pmat file >>> with, for example, DD preconditioners, you end up with a single file with >>> all local matrices scrambled in. >>> It would be nice to have the same logic, i.e., file.%p, and be able to >>> filter ranks that actually print to file like you suggest. >>> These kind of options are less critical than -info so I can just insert >>> them in the option database after initialization, if rank == view_this_rank >>> PetscOptionsInsertString(), but doing that using command line arguments >>> would be easier and more elegant, IMO. >>> >>> Thanks, >>> Pierre >>> >>> Barry >>> >>> On Apr 22, 2021, at 11:14 AM, Mark Adams <[email protected]> wrote: >>> >>> OK, so -info out.%p would do what I asked. Alas, what I really need it >>> to reduce the output with 128K processes. >>> Thanks, >>> >>> On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet <[email protected]> wrote: >>> >>>> >>>> >>>> On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <[email protected]> >>>> wrote: >>>> >>>> Not out of the box no. There is -info ::self which makes info only >>>> print from calls made on PETSC_COMM_SELF, but I don’t think that’s what you >>>> want. Best advice would be to print to a file and grep the file for any >>>> lines starting with the rank number you want. >>>> >>>> >>>> There is no need for grep, IMHO, just do -info file and then open >>>> file.%p >>>> >>>> Thanks, >>>> Pierre >>>> >>>> Best regards, >>>> >>>> Jacob Faibussowitsch >>>> (Jacob Fai - booss - oh - vitch) >>>> >>>> On Apr 22, 2021, at 10:16, Mark Adams <[email protected]> wrote: >>>> >>>> Can I make -info only print from one process? >>>> Mark >>>> >>>> >>>> >>>> >>> >>> > > -- > Stefano >
