On 3/8/23 04:16, Vincenzo D'Amore wrote:
I'm working on a Solr 4.10 master/slave cluster where every night around
100 cores are indexed and replicated. This is the configuration:
I hope you're aware that 4.10.0 was released 8.5 years ago. It's been
end of life for a very long time. Is upgrading to 9.x a viable option?
During the last 6 months the number of cores to be replicated almost
doubled and the slaves became very slow due very long garbage collection.
What version of Java are you using to run that? Have you tuned the
garbage collection? Solr 4.10 does have GC tuning out of the box, but
I've never seen GC options like that before, and I have no idea whether
they are effective. If you're not running on Java 11 or later, I would
strongly recommend G1GC. You would put this in solr.in.sh:
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=16m \
-XX:MaxGCPauseMillis=100 \
-XX:+ExplicitGCInvokesConcurrent \
-XX:+UseStringDeduplication \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA \
"
If you ARE running Java 11 or later, I would go with ZGC. I personally
wouldn't run a Solr version that old with anything newer than Java 8,
because Solr 7.0 was the first release that was guaranteed compatible
with Java 9.
GC_TUNE=" \
-XX:+UnlockExperimentalVMOptions \
-XX:+UseZGC \
-XX:+ParallelRefProcEnabled \
-XX:+ExplicitGCInvokesConcurrent \
-XX:+UseStringDeduplication \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA \
"
This situation is leading to having unresponsive slaves that return to work
normally only after a reboot.
In your experience do you know how I could limit the number of cores to be
replicated?
I haven't ever heard of something like that. Would be nice to have, but
if we did implement it, it would be in a 9.x version only, not applied
to anything older.
You should consider changing your nightly update procedure to not update
all the indexes at once. Tuning the GC would probably also help. The
options I have shown above are very good at eliminating most full GCs.
You should also consider increasing the heap size so there's more memory
for Java to work with. Can't make a recommendation about the heap size
without more info.
Thanks,
Shawn
