Hi Dominique, Thanks, But I still have one confusion, Please help me with it.
Pretty sure the issue is caused by caches size at new searcher warmup time. We use leader-follower architecture with a replication interval of 3 hours. This means every 3 hours we get a commit and the searcher warms up. Right? We have frequent indexing, It isn't possible that we don't get any document indexed in 3 hours period. So the new searcher warms up 7-8 times a day. If the issue was due to a new searcher warm-up, We would have got the issue many times a day. But we get this OutOfMemory Issue once or twice a month or sometimes once in two months, so some combination of events must be triggering it. Further, In solrConfig we have <maxWarmingSearchers>2</maxWarmingSearchers> does this mean that at some time up to 3 searchers(1 active and 2 still warming up their caches) might be holding to their caches and this resulting in the issue? Sould we reduce this to 1 ? what impact would it have if we reduced it ? On Wed, Aug 11, 2021 at 2:15 AM Dominique Bejean <dominique.bej...@eolya.fr> wrote: > Pretty sure the issue is caused by caches size at new searcher warmup time. > > Dominique > > > Le mar. 10 août 2021 à 09:07, Satya Nand <satya.n...@indiamart.com> a > écrit : > >> Hi Dominique, >> >> You don't provide information about the number of documents. Anyway, all >>> your cache size and mostly initial size are big. Cache are stored in JVM >>> heap. >> >> Document count is 101893353. >> >> About cache size, most is not always better. Did you make some performance >>> benchmarks in order to set these values ? >> >> We increased cache size in the hope to reduce some response time, We >> heavily use group queries with 7-8 boost factors. The average response time >> on this document set is 136 ms. We receive approx 65 requests/second in >> peak hours. The replication interval is 3 hours. >> >> The most strange thing about is that system keeps running for days >> without any issue, So I believe cache size should not be an issue. If the >> cache size had been the culprit, the issue would have been frequent. isn't >> it? >> >> >> >> On Mon, Aug 9, 2021 at 6:44 PM Dominique Bejean < >> dominique.bej...@eolya.fr> wrote: >> >>> Hi, >>> >>> You don't provide information about the number of documents. Anyway, all >>> your cache size and mostly initial size are big. Cache are stored in JVM >>> heap. >>> >>> About cache size, most is not always better. Did you make some >>> performance >>> benchmarks in order to set these values ? >>> >>> Try with the default values, after a few hours check cumulative caches >>> statistics in order to decide if you need to increase their sizes. The >>> objective is not to have cumulative_hitratio to 100%. There isn't ideal >>> value as it is really related to your datas, to the user's queries, to >>> how >>> you build your queries ... but 70% is a good value. At some point >>> increasing the size again and again won't increase cumulative_hitratio a >>> lot as it is a logarithmic curve. >>> >>> Check also the heap usage also with your JVM GC logs and a tool like >>> gceasy.io >>> >>> Regards >>> >>> Dominique >>> >>> >>> >>> >>> Le lun. 9 août 2021 à 07:44, Satya Nand <satya.n...@indiamart.com >>> .invalid> >>> a écrit : >>> >>> > Hi, >>> > We are facing a strange issue in our solr system. Most of the days it >>> keeps >>> > running fine but once or twice in a month, we face OutofMemory on solr >>> > servers. >>> > >>> > We are using Leader-Follower architecture, one Leader and 4 followers. >>> > Strangely we get OutofMemory error on all follower servers. >>> > Before the OutOfMemory this exception is found on all servers. >>> > >>> > Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall >>> > search-central-prd-solr-temp1 >>> > ERROR: null:java.lang.NullPointerException >>> search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173) >>> > search-central-prd-solr-temp1 >>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440) >>> > search-central-prd-solr-temp1 >>> > at >>> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723) >>> > search-central-prd-solr-temp1 >>> > at >>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) >>> > search-central-prd-solr-temp1 >>> > at org.eclipse.jetty.server.Server.handle(Server.java:534) >>> > search-central-prd-solr-temp1 >>> > at >>> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) >>> > search-central-prd-solr-temp1 >>> > at >>> > >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) >>> > search-central-prd-solr-temp1 >>> > >>> > *Configuration* >>> > Index size- 95 GB >>> > Heap 30 GB >>> > Ram-96 GB >>> > Polling Interval - 3 Hours >>> > Caching- >>> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000" >>> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache" >>> size="30000" >>> > initialSize="1000" autowarmCount="100" /> <documentCache class= >>> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" /> >>> > >>> > -- >>> > >>> > >>> >> >> >> --
