Re: Solr JVM Heap becomes full and stops when we try to restart

2021-05-10 Thread Toke Eskildsen
Subject: Solr JVM Heap becomes full and stops when we try to restart Hi everyone, We have 3 cluster solr running in 3 different machines with an index size of 300 GB. RAM: 300 GB per node Heap - Xms: 240GB Xmx: 300GB Index size: 300GB GC_TUNE="-XX:+UseG1GC -XX:InitiatingHeapOccupancyPerce

Re: Solr JVM Heap becomes full and stops when we try to restart

2021-05-10 Thread Michael Gibney
In addition to the other advice so far, if applicable I'd strongly recommend disabling swap (especially considering that you've already tried varying heap size without the desired effect): https://solr.apache.org/guide/8_8/taking-solr-to-production.html#disabling-swap Some good background reading o

Re: Solr JVM Heap becomes full and stops when we try to restart

2021-05-10 Thread Walter Underwood
That heap is way, too big. Solr does NOT pull the entire index into the JVM heap. You need RAM that is not in the heap for the OS to keep the index in file buffers. Try again with: -Xms16G -Xmx16G If you run out of that, try 31G. Starting and maximum heap size should always be the same for a

Re: Solr JVM Heap becomes full and stops when we try to restart

2021-05-10 Thread Markus Jelsma
Hi, Reduce the heap even further, 30GB is still very high. For example, our largest index is around 50GB in total size. Divided in 4 shards and 3 replicas, each node runs happily with just 3.5GB heap, and serving two replicas each! Maybe try sizing the heap from the bottom up? You'll know it when

Re: Solr JVM Heap becomes full and stops when we try to restart

2021-05-10 Thread Vignan Malyala
Hi Radu, Thanks for replying. As suggested by you, we have changed to Java 11, reverted to solr defaults and removed ConcGCThreads and ParallelGCThreads. But the issue persists the same. We tried changing heaps multiple times with 30GB, 150GB, 250 GB, etc. It doesn't have effect. Solr still stops

Re: Solr JVM Heap becomes full and stops when we try to restart

2021-05-10 Thread Radu Gheorghe
Hi Vigz, If you have a 300GB of RAM machine, you’d want a lower heap size, to leave room for OS to cache files. Also, the real memory usage of the JVM would likely go beyond your RAM, in which case either the JVM memory will go to swap (which would kill performance) or the OS will kill the proc

Solr JVM Heap becomes full and stops when we try to restart

2021-05-09 Thread Vignan Malyala
Hi everyone, We have 3 cluster solr running in 3 different machines with an index size of 300 GB. RAM: 300 GB per node Heap - Xms: 240GB Xmx: 300GB Index size: 300GB GC_TUNE="-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=45 -XX:ConcGCThreads=6 -XX:ParallelGCThreads=30 -XX:G1ReservePercent=20