Run jstat on your jvm and watch the activity on the newgen and oldgen, I think I ran it via jstat --gccause, but there are many options. You may have to run jstatd
You can resize newgen and oldgen maximum sizes independent of the default - I had to do this, and you can up the number of cleaning threads to each. If you have the cpu, something to consider. I also set the "start threshold" on the oldgen from default of 70% full to 30% to get more runway, and increased the threads to clean oldgen. Had to get very aggressive with it for our system, and this worked very well indeed. New objects that are too large for newgen can be created in the oldgen, btw. I'd bet those humongous allocations are (have to be) in the oldgen. If oldgen cleanup can't keep up with new (large) objects being created there as well as newgen objects getting promoted to oldgen, 30% of runway might not be enough. Use docvalues! This was all with CMS a few years ago, but I was able to keep the same settings when I switched us over to G1. This is my own personal experience, as best I can remember it anyway. YMMV ________________________________ From: Webster Homer <webster.ho...@milliporesigma.com> Sent: Thursday, June 24, 2021 2:55 PM To: solr-u...@lucene.apache.org <solr-u...@lucene.apache.org>; users@solr.apache.org <users@solr.apache.org> Subject: Solr GC tuning Advice I am seeing intermittent GC suspensions. We see one or two nodes suddenly go to 100% GC suspension. We are using Solr 7.7.2 on Centos 7.7 with 64G of memory. We are using G1GC GC_TUNE="-XX:+PerfDisableSharedMem \ -XX:G1HeapRegionSize=16m \ -XX:+UseG1GC \ -XX:MaxGCPauseMillis=250 \ -XX:+ParallelRefProcEnabled" -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=128M -XX:+PrintAdaptiveSizePolicy" SOLR_JAVA_MEM="-Xms32768m -Xmx32768m" At first we had the G1HeapRegionSize set to 8m. I saw a lot of humongous allocations so we upped it to 16m. That helped a little, I think, but we still see the issue. On several posts on Solr GC configuration I noticed that -XX:+UseLargePages is set, but little to say why this would be useful. For it to work Large Memory Pages needs to be enabled as described here https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oracle.com%2Fjava%2Ftechnologies%2Fjavase%2Flargememory-pages.html&data=04%7C01%7C%7Cc84852f908a74d962c2108d937201fb8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637601433371381488%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fMg9SpMdjBf9Ao%2F3ji5nsbgQcvzBCqmKa4z7nOb4Ok%3D&reserved=0 What benefit does Solr get from this? Could it help with GC suspensions? We currently have 2 shards on our collections and are thinking of going to 3 shards to lower the memory pressure. I'm not an expert in GC tuning so any advice would be appreciated. This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click merckgroup.com/disclaimer<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.merckgroup.com%2Fen%2Flegal-disclaimer%2Fmail-disclaimer.html&data=04%7C01%7C%7Cc84852f908a74d962c2108d937201fb8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637601433371381488%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=reCITK1Xv2dTzybwQy9Bt716mwmq6gkUeo7G50EQZ%2B0%3D&reserved=0> to access the German, French, Spanish, Portuguese, Turkish, Polish and Slovak versions of this disclaimer. Please find our Privacy Statement information by clicking here merckgroup.com/en/privacy-statement.html<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.merckgroup.com%2Fen%2Fprivacy-statement.html&data=04%7C01%7C%7Cc84852f908a74d962c2108d937201fb8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637601433371381488%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lgfZA6lVxzfecFRMMhGUkk%2B4gqXez%2Fh42ObscqbV9B8%3D&reserved=0>
