On 8/26/22 14:18, Sidharth Negi wrote:
The disk space taken by the index of both Solr versions was about ~35 GB and the number of docs ~30 million in both.
Unless that system is handling insanely complex queries that chew up lots of memory, I would not expect it to need more than about 8GB of heap with that index size, and quite possibly even less.
48GB total system memory would probably work if the server is not handling anything other than that Solr index.
If you share solr GC logs that cover a day or more of heavy usage (several days being better), I might be able to determine a more precise heap size recommendation. Unfortunately there is no generic way to calculate heap requirements based on data statistics. GC logs provide the best info, otherwise it requires experimentation. My guess of 8GB is just that -- a guess. It might be wrong, either too high or too low.
Let me run an experiment using the same GC settings on both to see if that works. Is there anything else we can do to narrow down the reason for sure? All slaves combined will have to serve over 80k requests per second once we set the number of slaves such that the CPU usage of all remains well below 70% at peaks.
Very high query rate, but it sounds like you're going to size appropriately to deal with it.
Interesting to note that when I ran the experiment with Solr 9, the CPU usage was about the same as Solr 6.
That is interesting. Solr 9 also uses G1 and log4j2, so those are probably not the culprit. Deploying the latest release would be our recommendation, because only extremely bad bugs, mostly security issues, are going to be fixed in new 8.x releases. It is very unlikely that a new 8.11.x version will address this issue.
Thanks, Shawn
