Hi Bruno, No, unfortunately I have not played with the parameter values. Hypre has a set of recommended parameter values for 3D problems, but I think some are not exposed to the deal.II interface so I have avoided digging into the source.
I chose PETSc somewhat arbitrarily, so if you are more familiar with the Trilinos equivalents I can switch to that. Glancing at the documentation it appears that all of the Trilinos parameters are manipulable via the deal.II interface using the Teuchos::ParameterList object -- do you know if this is true? If, so it might be a better option. With regards to actually choosing parameters: Is there some (reasonably) systematic way to tune them? It looks like the ML documentation gives some tools to get started on profiling with the different parameters (section 6.6) but I'm unsure whether the utility methods are necessarily accessible via the deal.II interface. Additionally, it looks like there are quite a few parameter values to choose. Do you just futz with each one individually and look at the speed? Or is there some particular order or hierarchy to choosing the parameters? Is there some manner in which I can analyze my actual problem to optimize some choices? And finally, do you have recommendations for 3D problems rather than 2D? Thanks so much for the help - Lucas On Friday, March 4, 2022 at 4:59:22 PM UTC-6 blais...@gmail.com wrote: > Maybe I can add something there. > Although it's clearly not ideal to use the AMG preconditioner on the whole > matrix, it can work quite well sometimes. > Have you played around with the smoother and the coarsener you use or are > you using the default values? I have found that you need to tune the AMG > preconditioner parameters a bit to get a decent. My experience is limited > to the Trilinos AMG preconditioner (and not the MueLU one), but if you > define your constant modes correctly and use adequate smoother and > coarsener, it's really not all that bad. > Feel free to reach out if you feel I could help you with this. > Best > Bruno > > On Friday, March 4, 2022 at 4:13:51 p.m. UTC-5 Wolfgang Bangerth wrote: > >> >> Lucas, >> it's hard to tell why this might be so slow. In general, I *think* what >> you are doing is apply the AMG to the entire, coupled system? That is >> unusual, though maybe not strictly wrong. In practice, however, we >> almost always apply multigrid preconditioners only to the elliptic >> blocks of coupled systems. You can see that in the preconditioner that >> is discussed in the "Possibilities for extensions" section of step-22, >> where we need to solve an elliptic problem as part of a preconditioner >> for Stokes, and this is the approach that is also taken in a number of >> other related programs (31, 32, 56). The experience of the community is >> that applying multigrid (geometric or algebraic) to the entire coupled >> problem is just fraught with so many problems that it is hard to get >> right and harder yet to efficiently implement. >> >> Best >> W. >> >> >> On 3/3/22 10:22, Lucas Myers wrote: >> > *** Caution: EXTERNAL Sender *** >> > >> > Hi folks, >> > >> > I'm trying to solve a 3D problem in parallel using an AMG >> > preconditioner, but the performance is bad. I'm wondering if I can get >> > some advice from someone who has experience choosing and tuning >> > multigrid preconditioners, particularly for problems in 3D. >> > >> > For context: >> > >> > * The problem is vector-valued (5 components) and elliptic(ish) >> > * I use GMRES w/BoomerAMG preconditioner for asymmetric matrices, >> > everything is default >> > * In 2D, scaling is good even down to 5,000 DoFs per processor, and >> > number of iterations is independent of problem size >> > * In 3D the scaling is bad. Adding more processors after about 50,000 >> > DoFs/processor actually slows the program down and sometimes gives >> > memory errors. >> > * Taking away the preconditioner in 3D gives a ~20x speedup at 16 >> > processors, and strong scaling is linear. However, the number of >> > iterations increases with problem size. >> > >> > Given all that, I have some questions: >> > >> > 1. Why might it be that the memory and wall-time scaling is so bad in >> 3D? >> > 2. Are there any examples lying around deal.II of folks using multigrid >> > for 3D problems in parallel? All the tutorials that I looked at were >> > in 2D, including the 2 examples used in the distributed computing >> paper. >> > 3. Might there be an easy way to fix it while still using BoomerAMG? I >> > know the Hypre documentation >> > < >> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhypre.readthedocs.io%2Fen%2Flatest%2Fsolvers-boomeramg.html&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=P8OcXGNcXuwWJtLFRCdDsxvIJpKKpkA2p5%2BQSxdkDKE%3D&reserved=0> >> >> >> > gives some parameter recommendations, but I'm not so sure (1) how to >> > set those options via deal.II (I think they are not available via >> > the AdditionalData interface), or (2) whether those will work. Does >> > anyone have experience with this? >> > 4. Might the Trilinos AMG preconditioner work any better for this >> > problem by default? And if not, is there a systematic way to tune it >> > (particularly using a deal.II interface) to work better for a 3D >> > problem? >> > 5. Might the in-house GMG methods work better? And if so, do the >> > matrix-free methods stand a chance of performing better even if I >> > have to use some complicated functions in the matrix assembly (for >> > my weak form I have a 5-component nonlinear function of my solution >> > vector which has to be inverted via Newton's method across my domain). >> > >> > If more context on the particular problem would be helpful I can >> > certainly give details. Mostly I'm looking for intuition, general >> > suggestions, or pointers to good references. Any help is much >> appreciated. >> > >> > Kind regards, >> > Lucas >> > >> > -- >> > The deal.II project is located at http://www.dealii.org/ >> > < >> https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dealii.org%2F&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=NJZnFeCaXecCwK6xyRuBwI033BIPlGrkD5E%2F%2Fy4%2B6ew%3D&reserved=0> >> >> >> > For mailing list/forum options, see >> > https://groups.google.com/d/forum/dealii?hl=en >> > < >> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fforum%2Fdealii%3Fhl%3Den&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CGYWQ0dRtj%2Fhxr6m7t4DdR0BcU59lM25AuMwIrB4Ns4%3D&reserved=0> >> >> >> > --- >> > You received this message because you are subscribed to the Google >> > Groups "deal.II User Group" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an email to dealii+un...@googlegroups.com >> > <mailto:dealii+un...@googlegroups.com>. >> > To view this discussion on the web visit >> > >> https://groups.google.com/d/msgid/dealii/7bcfdab4-ea5e-47ea-82f5-c78bc9e3ffd8n%40googlegroups.com >> >> > < >> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fdealii%2F7bcfdab4-ea5e-47ea-82f5-c78bc9e3ffd8n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xPHuLbXYG8SbVTOQJ0PLCz4JqiaFJC7y%2F3AmI5V9N34%3D&reserved=0>. >> >> >> >> >> -- >> ------------------------------------------------------------------------ >> Wolfgang Bangerth email: bang...@colostate.edu >> www: http://www.math.colostate.edu/~bangerth/ >> > -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/7295b0a3-fd8d-4059-b9ed-9961ed723519n%40googlegroups.com.
