We haven't tested anything on 9, nor do we plan on going to 9 any time soon, as it's quite a bit of work as not only do we have our own patches of Solr *(providing custom analysers etc.,)*, but we also have a moderate extensive plugin of custom parsers etc., for our own business needs, so we'd also have to upgrade all of them to 9.X as well.
On Fri, 25 Nov 2022 at 17:43, matthew sporleder <msporle...@gmail.com> wrote: > What happens on 9.x?? :) > > > On Nov 25, 2022, at 11:33 AM, Richard Goodman <richa...@brandwatch.com> > wrote: > > > > Hi there, > > > > We have a cluster spread over 72 instances on k8s hosting around 12.5 > > billion documents (made up of 30 collections, each collection having 12 > > shards). We were originally using 7.7.2 and performance was okay enough > for > > us for our business needs. We then recently upgraded our cluster to > > v8.11.2, and have noticed a drop in performance. I appreciate that there > > have been a lot of changes from 7.7.2 to 8.11.2, but I have been > collecting > > metrics, and although the configuration (instance type and resource > > allocation, start up opts) are the same, we are completely at a loss as > to > > why it's performing worse, and was wondering if anyone had any guidance? > > > > I recently stumbled across the tickets; > > > > - SOLR-15840 <https://issues.apache.org/jira/browse/SOLR-15840> - > > Performance degradation with http2 > > - SOLR-16099 <https://issues.apache.org/jira/browse/SOLR-16099> - HTTP > > Client threads can hang > > > > In particular which sparked interest, and so we spun up a parallel > cluster > > with -Dsolr.http1=true, and there was no difference in performance. We're > > testing a couple of other ideas, such as different DirectoryFatory *(as I > > saw a message from someone in the Solr Slack about there being an issue > > with the MMap directory and vm.max_map_count)*, some GC settings, but are > > really open to any suggestions. We're also happy if it'll help with any > > performance related topics to use this cluster to test patches at a large > > scale to see if it'll help with performance *(more specifically to the > two > > Solr tickets listed above)*. > > > > I thought it would be useful to show some metrics I collected where we > had > > 2 clusters spun up, 1 being 7.7.2 and 1 being 8.11.2 where the 8.11.2 > > cluster was the active, and all traffic was being shadow loaded into the > > 7.7.2 cluster to compare against. It's important to note that both > clusters > > had the same configuration, here is a list to name a few: > > > > - G1GC garbage collector > > - TLOG replication > > - 27Gi Memory per instance > > - 16Gi assigned to -XmX and -Xms > > - 16 cores > > - -XX:G1HeapRegionSize=4m > > - -XX:G1ReservePercent=20 > > - -XX:InitiatingHeapOccupancyPercent=35 > > > > One metric that did stand out, was that 8.11.2 was churning through *a > lot* of > > eden space in the heap, which can be seen in some of the screenshots of > > metrics below; > > > > Total Memory Usage: > > 7.7.2 > > > > > > 8.11.2 > > > > > > Total Used G1 Pools > > 7.7.2 > > > > > > 8.11.2 > > > > > > And finally, the overall thread pool > > 7.7.2 > > > > > > 8.11.2 > > > > > > Any guidance or requests to test for performance wise would be > appreciated. > > > > Thanks, > > > > Richard > -- Richard Goodman (he/him) | Senior Data Infrastructure engineer richa...@brandwatch.com NEW YORK | BOSTON | CHICAGO | TORONTO | *BRIGHTON* | LONDON | COPENHAGEN | BERLIN | STUTTGART | FRANKFURT | PARIS | BUDAPEST | SOFIA | CHENNAI | SINGAPORE | SYDNEY | MELBOURNE
