Hi Florian, I haven't heard any reports of a memory leak triggered by the REINDEXCOLLECTION codepath, but such a leak is always possible. I'd love to see what you find if you're able to take a heap dump!
The typical way (afaik) to create heap dumps "on demand" is with jcmd or jmap. If that's not available on the Solr pods, it might be possible to "kubectl cp" one of those executables into your pods, and then run it to produce a dump. If that doesn't work, then the other option I'd recommend is making sure that Solr is running with the "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<file-or-dir-path>" JVM flags so that a heap dump is produced when OOM is eventually hit. (You'd want to make sure that the heap-dump path is something persistent, so that the heap dump won't be wiped out by a pod-restart.) Hopefully that helps a bit. Let us know if you're able to find anything! Best, Jason On Wed, Apr 30, 2025 at 6:55 AM Schieder Florian <florian.schie...@puzzleyou.de.invalid> wrote: > > Hello all, > > we are running Solr 9.8.1 on Kubernetes using the official Solr Operator > 0.9.1. > > After many asynchronous REINDEXCOLLECTION requests we are encountering a > significant RAM allocation increase which is itself neither surprising nor > problematic, but after those requests have successfully completed, the > consumption stays constant and does not decrease. Manually restarting the > SolrCloud deployment always resolves that issue, so for me it seems like > either the allocated blocks are afterwards not being freed up correctly or, > they *are* deallocated correctly but after the requests completed, the heap > itself is fragmented too much. > > If the heap allocation increased and increased after several > REINDEXCOLLECTION requests and we did not restart the pods, this led to OOM > crashes for us in the past. In this case, the REINDEXCOLLECTION async > requests were interrupted which required manual intervention. > > It's easy for me to reproduce this RAM allocation behavior, but after diving > into how to create JVM memory dumps with 'jcmd' and 'jmap', it seems to me > like it's not trivially possible to create such a heap dump with the current > tools installed into the Solr pods. However, I'm not that expert regarding > Solr and the Java ecosystem, so if there are any solutions to create memory > dumps to help the Solr development team tracing the RAM consumption issues > explained above, any ideas are welcome. > > Many thanks! > Florian Schieder
