Pretty sure the issue is caused by caches size at new searcher warmup time.
Dominique Le mar. 10 août 2021 à 09:07, Satya Nand <satya.n...@indiamart.com> a écrit : > Hi Dominique, > > You don't provide information about the number of documents. Anyway, all >> your cache size and mostly initial size are big. Cache are stored in JVM >> heap. > > Document count is 101893353. > > About cache size, most is not always better. Did you make some performance >> benchmarks in order to set these values ? > > We increased cache size in the hope to reduce some response time, We > heavily use group queries with 7-8 boost factors. The average response time > on this document set is 136 ms. We receive approx 65 requests/second in > peak hours. The replication interval is 3 hours. > > The most strange thing about is that system keeps running for days without > any issue, So I believe cache size should not be an issue. If the cache > size had been the culprit, the issue would have been frequent. isn't it? > > > > On Mon, Aug 9, 2021 at 6:44 PM Dominique Bejean <dominique.bej...@eolya.fr> > wrote: > >> Hi, >> >> You don't provide information about the number of documents. Anyway, all >> your cache size and mostly initial size are big. Cache are stored in JVM >> heap. >> >> About cache size, most is not always better. Did you make some performance >> benchmarks in order to set these values ? >> >> Try with the default values, after a few hours check cumulative caches >> statistics in order to decide if you need to increase their sizes. The >> objective is not to have cumulative_hitratio to 100%. There isn't ideal >> value as it is really related to your datas, to the user's queries, to how >> you build your queries ... but 70% is a good value. At some point >> increasing the size again and again won't increase cumulative_hitratio a >> lot as it is a logarithmic curve. >> >> Check also the heap usage also with your JVM GC logs and a tool like >> gceasy.io >> >> Regards >> >> Dominique >> >> >> >> >> Le lun. 9 août 2021 à 07:44, Satya Nand <satya.n...@indiamart.com >> .invalid> >> a écrit : >> >> > Hi, >> > We are facing a strange issue in our solr system. Most of the days it >> keeps >> > running fine but once or twice in a month, we face OutofMemory on solr >> > servers. >> > >> > We are using Leader-Follower architecture, one Leader and 4 followers. >> > Strangely we get OutofMemory error on all follower servers. >> > Before the OutOfMemory this exception is found on all servers. >> > >> > Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall >> > search-central-prd-solr-temp1 >> > ERROR: null:java.lang.NullPointerException search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844) >> > search-central-prd-solr-temp1 >> > at >> > >> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173) >> > search-central-prd-solr-temp1 >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440) >> > search-central-prd-solr-temp1 >> > at >> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723) >> > search-central-prd-solr-temp1 >> > at >> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) >> > search-central-prd-solr-temp1 >> > at >> > >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >> > search-central-prd-solr-temp1 >> > at >> > >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) >> > search-central-prd-solr-temp1 >> > at >> > >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >> > search-central-prd-solr-temp1 >> > at >> > >> > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) >> > search-central-prd-solr-temp1 >> > at org.eclipse.jetty.server.Server.handle(Server.java:534) >> > search-central-prd-solr-temp1 >> > at >> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) >> > search-central-prd-solr-temp1 >> > at >> > >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) >> > search-central-prd-solr-temp1 >> > >> > *Configuration* >> > Index size- 95 GB >> > Heap 30 GB >> > Ram-96 GB >> > Polling Interval - 3 Hours >> > Caching- >> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000" >> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache" >> size="30000" >> > initialSize="1000" autowarmCount="100" /> <documentCache class= >> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" /> >> > >> > -- >> > >> > >> > > >
