Lucas,
it's hard to tell why this might be so slow. In general, I *think* what
you are doing is apply the AMG to the entire, coupled system? That is
unusual, though maybe not strictly wrong. In practice, however, we
almost always apply multigrid preconditioners only to the elliptic
blocks of coupled systems. You can see that in the preconditioner that
is discussed in the "Possibilities for extensions" section of step-22,
where we need to solve an elliptic problem as part of a preconditioner
for Stokes, and this is the approach that is also taken in a number of
other related programs (31, 32, 56). The experience of the community is
that applying multigrid (geometric or algebraic) to the entire coupled
problem is just fraught with so many problems that it is hard to get
right and harder yet to efficiently implement.
Best
W.
On 3/3/22 10:22, Lucas Myers wrote:
*** Caution: EXTERNAL Sender ***
Hi folks,
I'm trying to solve a 3D problem in parallel using an AMG
preconditioner, but the performance is bad. I'm wondering if I can get
some advice from someone who has experience choosing and tuning
multigrid preconditioners, particularly for problems in 3D.
For context:
* The problem is vector-valued (5 components) and elliptic(ish)
* I use GMRES w/BoomerAMG preconditioner for asymmetric matrices,
everything is default
* In 2D, scaling is good even down to 5,000 DoFs per processor, and
number of iterations is independent of problem size
* In 3D the scaling is bad. Adding more processors after about 50,000
DoFs/processor actually slows the program down and sometimes gives
memory errors.
* Taking away the preconditioner in 3D gives a ~20x speedup at 16
processors, and strong scaling is linear. However, the number of
iterations increases with problem size.
Given all that, I have some questions:
1. Why might it be that the memory and wall-time scaling is so bad in 3D?
2. Are there any examples lying around deal.II of folks using multigrid
for 3D problems in parallel? All the tutorials that I looked at were
in 2D, including the 2 examples used in the distributed computing paper.
3. Might there be an easy way to fix it while still using BoomerAMG? I
know the Hypre documentation
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhypre.readthedocs.io%2Fen%2Flatest%2Fsolvers-boomeramg.html&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=P8OcXGNcXuwWJtLFRCdDsxvIJpKKpkA2p5%2BQSxdkDKE%3D&reserved=0>
gives some parameter recommendations, but I'm not so sure (1) how to
set those options via deal.II (I think they are not available via
the AdditionalData interface), or (2) whether those will work. Does
anyone have experience with this?
4. Might the Trilinos AMG preconditioner work any better for this
problem by default? And if not, is there a systematic way to tune it
(particularly using a deal.II interface) to work better for a 3D
problem?
5. Might the in-house GMG methods work better? And if so, do the
matrix-free methods stand a chance of performing better even if I
have to use some complicated functions in the matrix assembly (for
my weak form I have a 5-component nonlinear function of my solution
vector which has to be inverted via Newton's method across my domain).
If more context on the particular problem would be helpful I can
certainly give details. Mostly I'm looking for intuition, general
suggestions, or pointers to good references. Any help is much appreciated.
Kind regards,
Lucas
--
The deal.II project is located at http://www.dealii.org/
<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dealii.org%2F&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=NJZnFeCaXecCwK6xyRuBwI033BIPlGrkD5E%2F%2Fy4%2B6ew%3D&reserved=0>
For mailing list/forum options, see
https://groups.google.com/d/forum/dealii?hl=en
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fforum%2Fdealii%3Fhl%3Den&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CGYWQ0dRtj%2Fhxr6m7t4DdR0BcU59lM25AuMwIrB4Ns4%3D&reserved=0>
---
You received this message because you are subscribed to the Google
Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to dealii+unsubscr...@googlegroups.com
<mailto:dealii+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dealii/7bcfdab4-ea5e-47ea-82f5-c78bc9e3ffd8n%40googlegroups.com
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fdealii%2F7bcfdab4-ea5e-47ea-82f5-c78bc9e3ffd8n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xPHuLbXYG8SbVTOQJ0PLCz4JqiaFJC7y%2F3AmI5V9N34%3D&reserved=0>.
--
------------------------------------------------------------------------
Wolfgang Bangerth email: bange...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/
--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see
https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dealii/a82f079f-7f79-6e0f-61f0-9510f86e7b76%40colostate.edu.
