Dear all, this is a follow-up to an old mail concerning non-repeatibility issues in a parallel environment.
We are solving the steady 3D RANS eqns using Newtons's algorithm. All equations (turbulence included) are fully coupled. Our non-linear convergence history shows remarkable non-repeatibility among subsequent parallel runs (see the enclosed plot referred to as MeTis partitioning). We believe (but of course cannot rule out other reasons) that this is due to the fact that within those "ghosted" nodes that are shared by more than two sub-domains the rhs will be slightly different among subsequent runs depending on the order by which contributions are accumulated in these interfacial nodes. Since we push convergence down to machine accuracy, this may not be irrelevant. We then devised an alternative partitioning (though applicable only on very simple geometries) that guarantees that "ghosted" nodes are shared by not more than two procs. This guarantees that not more than two contributions will be accumulated in the ghosted nodes. Using this partitioning (referred to as geometric in the enclosed plots) subsequent runs give indeed identical results. In all cases, of course, we start from identical initial solutions. A colleague of ours has suggested the following: > This could be a problem of GMRES least-squares > system being poorly conditioned. I would try set a higher (very high in > fact) the tolerance for the inner (linear system) solve and see whether > the outer (Newton) iterations become more predictable. > The other possibility is indeed in the communication "inexectness". Are > there any asynchronous communications used in PETSc or in your code. > Change all you can to synchronous communications and see whether results > become more stable. Could you comment in particular on this latter? Is there a way we can modify PETSc's behaviour via command line options or otherwise? Regards, Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 -------------- next part -------------- A non-text attachment was scrubbed... Name: geometricPartitioning.pdf Type: application/x-download Size: 3634 bytes Desc: not available URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080311/214b5a29/attachment.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: MetisPartitioning.pdf Type: application/x-download Size: 8193 bytes Desc: not available URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080311/214b5a29/attachment-0001.bin>
