Hi Shawn,
>
>
> Do you have the actual OutOfMemoryError exception? Can we see that?
> There are several resources other than heap memory that will result in
> OOME if they are exhausted. It's important to be investigating the
> correct resource.
*Exception:*
Aug, 04 2021 15:38:36 org.apache.solr.servlet.HttpSolrCall
search-central-prd-solr-temp1
ERROR: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java
heap space search-central-prd-solr-temp1
at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:676)
search-central-prd-solr-temp1
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:544)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
search-central-prd-solr-temp1
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at org.eclipse.jetty.server.Server.handle(Server.java:534)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
search-central-prd-solr-temp1
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
search-central-prd-solr-temp1
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
search-central-prd-solr-temp1
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
search-central-prd-solr-temp1
at java.lang.Thread.run(Thread.java:748)
search-central-prd-solr-temp1
Caused by: java.lang.OutOfMemoryError: Java heap space
search-central-prd-solr-temp1
at org.apache.lucene.util.FixedBitSet.clone(FixedBitSet.java:480)
search-central-prd-solr-temp1
at
org.apache.solr.search.DocSetBase.intersection(DocSetBase.java:124)
search-central-prd-solr-temp1
at org.apache.solr.search.BitDocSet.intersection(BitDocSet.java:40)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1482)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1968)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
search-central-prd-solr-temp1
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
search-central-prd-solr-temp1
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
search-central-prd-solr-temp1
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
search-central-prd-solr-temp1
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
search-central-prd-solr-temp1
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
search-central-prd-solr-temp1
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at org.eclipse.jetty.server.Server.handle(Server.java:534)
search-central-prd-solr-temp1
On Tue, Aug 10, 2021 at 6:30 AM Shawn Heisey <apa...@elyograg.org> wrote:
> On 8/8/2021 11:43 PM, Satya Nand wrote:
> > We are facing a strange issue in our solr system. Most of the days it
> keeps
> > running fine but once or twice in a month, we face OutofMemory on solr
> > servers.
> >
> > We are using Leader-Follower architecture, one Leader and 4 followers.
> > Strangely we get OutofMemory error on all follower servers.
> > Before the OutOfMemory this exception is found on all servers.
>
> Do you have the actual OutOfMemoryError exception? Can we see that?
> There are several resources other than heap memory that will result in
> OOME if they are exhausted. It's important to be investigating the
> correct resource.
>
> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache"
> size="30000"
> > initialSize="1000" autowarmCount="100" /> <documentCache class=
> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
>
> If you have five million documents (including those documents that have
> been deleted) in a core, then each filterCache entry for that core will
> be 625000 bytes, plus some unknown amount of overhead to manage the
> entry. Four thousand of them will consume 2.5 billion bytes. If you
> have multiple cores each with many documents, the amount of memory
> required for the filterCache could get VERY big.
>
> Until we can see the actual OOME exception, we won't know what resource
> you need to investigate. It is frequently NOT memory.
>
> Thanks,
> Shawn
>
--
