Awesome follow up. Thanks.
On Tue, Mar 8, 2022 at 8:00 AM Dominic Humphries <domi...@adzuna.com.invalid> wrote: > > Hey folks, > > you might remember me posting a while back about a problem we were having > with upgrading: We were using 8.3 with no major problems, but when we tried > upgrading to the latest Solr we saw slower response times and increased > failures, to the point where we couldn't even consider upgrading. > > Initial suspicion fell on the JVM's GC because the heap was getting huge. > We also worried about thread creation, disk performance, and a few other > things. Along the way we spotted that our issue seemed to be connected to > external files that were getting read after every refresh, since emptying > the files cured the issue. > > But we needed them for boosts! > > Long story short, with some help from the good folks at > OpenSourceConnections, we finally got things resolved! > > It turns out that our problem was caused by refreshes causing followers to > switch to a new searcher, which blocked until it had imported the external > file. So first we tried adding the config suggested by the docs to set up > listeners: > <listener event="newSearcher" > class="org.apache.solr.schema.ExternalFileFieldReloader"/> > <listener event="firstSearcher" > class="org.apache.solr.schema.ExternalFileFieldReloader"/> > > These initially seemed to break everything - solr simply didn't want to > restart with these lines in place. Eventually though, we discovered that it > *did* restart if left long enough (over ten mins!) and after finally > starting up with this config in place, our performance issues largely went > away! > > We removed some soft commit config we had on our followers since there was > no real need for it there and it was part of the reason for slowdown; we > also changed to the StandardDirectoryFactory for best possible performance. > This brought the restart down to a mere(!) three minutes, which was just > about acceptable for our use case. All these changes together gave us > graceful switching over from old searcher to new without losing any > requests. > > We also added lifecycle hooks to our AWS instances to ensure that our > followers would be put into service as soon as they were fully refreshed > and up-to-date, but not before. This removed our worries about any slowness > on startup meaning out-of-date (or failed) search results. > > We're halfway through upgrading our Solr infrastructure now, to 8.11.1 and > we're seeing great results - we actually get less errors than with 8.3 - so > we're expecting to be fully updated by the end of the week, and with better > performance to boot! > > Thanks all for your advice in the long process of troubleshooting this > issue, and hopefully this write-up will give anybody else suffering in the > same way some ideas on how to solve it :) > > Dom
