Also take a look at your Xmx and Xms values. I find this to be absolutely vital when taking into consideration the machine's own memory, and the index size. you want the OS to be able to put most of the index segments into memory, and want those two values (Xmx and Xms) to be identical, Just a thought. -Dave
On Wed, Mar 8, 2023 at 10:19 AM Vincenzo D'Amore <v.dam...@gmail.com> wrote: > Hi Shawn, thanks for the answer. Unfortunately the upgrade is not a viable > option and the cluster is using an old JVM 1.7. > On the other hand, as you suggested, we are changing the update procedure > to not update all the indexes in one shot and after every index build (or > bunch of indexes) starting the propagation process. > This I hope will relax the memory consumption and make the slave more > reliable. > > On Wed, Mar 8, 2023 at 3:37 PM Shawn Heisey <apa...@elyograg.org> wrote: > > > On 3/8/23 04:16, Vincenzo D'Amore wrote: > > > I'm working on a Solr 4.10 master/slave cluster where every night > around > > > 100 cores are indexed and replicated. This is the configuration: > > > > I hope you're aware that 4.10.0 was released 8.5 years ago. It's been > > end of life for a very long time. Is upgrading to 9.x a viable option? > > > > > During the last 6 months the number of cores to be replicated almost > > > doubled and the slaves became very slow due very long garbage > collection. > > > > What version of Java are you using to run that? Have you tuned the > > garbage collection? Solr 4.10 does have GC tuning out of the box, but > > I've never seen GC options like that before, and I have no idea whether > > they are effective. If you're not running on Java 11 or later, I would > > strongly recommend G1GC. You would put this in solr.in.sh: > > > > GC_TUNE=" \ > > -XX:+UseG1GC \ > > -XX:+ParallelRefProcEnabled \ > > -XX:G1HeapRegionSize=16m \ > > -XX:MaxGCPauseMillis=100 \ > > -XX:+ExplicitGCInvokesConcurrent \ > > -XX:+UseStringDeduplication \ > > -XX:+AlwaysPreTouch \ > > -XX:+UseNUMA \ > > " > > > > If you ARE running Java 11 or later, I would go with ZGC. I personally > > wouldn't run a Solr version that old with anything newer than Java 8, > > because Solr 7.0 was the first release that was guaranteed compatible > > with Java 9. > > > > GC_TUNE=" \ > > -XX:+UnlockExperimentalVMOptions \ > > -XX:+UseZGC \ > > -XX:+ParallelRefProcEnabled \ > > -XX:+ExplicitGCInvokesConcurrent \ > > -XX:+UseStringDeduplication \ > > -XX:+AlwaysPreTouch \ > > -XX:+UseNUMA \ > > " > > > > > This situation is leading to having unresponsive slaves that return to > > work > > > normally only after a reboot. > > > In your experience do you know how I could limit the number of cores to > > be > > > replicated? > > > > I haven't ever heard of something like that. Would be nice to have, but > > if we did implement it, it would be in a 9.x version only, not applied > > to anything older. > > > > > > You should consider changing your nightly update procedure to not update > > all the indexes at once. Tuning the GC would probably also help. The > > options I have shown above are very good at eliminating most full GCs. > > You should also consider increasing the heap size so there's more memory > > for Java to work with. Can't make a recommendation about the heap size > > without more info. > > > > Thanks, > > Shawn > > > > > -- > Vincenzo D'Amore >
