Subject: Solr JVM Heap becomes full and stops when we try to restart
Hi everyone,
We have 3 cluster solr running in 3 different machines with an index size of
300 GB.
RAM: 300 GB per node
Heap - Xms: 240GB Xmx: 300GB
Index size: 300GB
GC_TUNE="-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPerce
In addition to the other advice so far, if applicable I'd strongly
recommend disabling swap (especially considering that you've already
tried varying heap size without the desired effect):
https://solr.apache.org/guide/8_8/taking-solr-to-production.html#disabling-swap
Some good background reading o
That heap is way, too big. Solr does NOT pull the entire index into the JVM
heap.
You need RAM that is not in the heap for the OS to keep the index in file
buffers.
Try again with:
-Xms16G
-Xmx16G
If you run out of that, try 31G.
Starting and maximum heap size should always be the same for a
Hi,
Reduce the heap even further, 30GB is still very high. For example, our
largest index is around 50GB in total size. Divided in 4 shards and 3
replicas, each node runs happily with just 3.5GB heap, and serving two
replicas each!
Maybe try sizing the heap from the bottom up? You'll know it when
Hi Radu,
Thanks for replying.
As suggested by you, we have changed to Java 11, reverted to solr defaults
and removed ConcGCThreads and ParallelGCThreads. But the issue persists the
same.
We tried changing heaps multiple times with 30GB, 150GB, 250 GB, etc. It
doesn't have effect. Solr still stops
Hi Vigz,
If you have a 300GB of RAM machine, you’d want a lower heap size, to leave room
for OS to cache files. Also, the real memory usage of the JVM would likely go
beyond your RAM, in which case either the JVM memory will go to swap (which
would kill performance) or the OS will kill the proc
Hi everyone,
We have 3 cluster solr running in 3 different machines with an index size
of 300 GB.
RAM: 300 GB per node
Heap - Xms: 240GB Xmx: 300GB
Index size: 300GB
GC_TUNE="-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=45
-XX:ConcGCThreads=6
-XX:ParallelGCThreads=30
-XX:G1ReservePercent=20