Are you seeing any messages in the logs with "PERFORMANCE WARNING: Overlapping onDeckSearchers"? Can you elaborate on the autowarm configuration that you have? any "postCommit" events?
If you set the logger of "org.apache.solr.search.SolrIndexSearcher" to DEBUG level you should see when the searcher is open and how long it takes to warmup. On Wed, Dec 7, 2022 at 9:58 AM Matias Laino <matias.la...@passare.com.invalid> wrote: > I'm sorry but I'm not sure what you mean with metal, our servers are EC2 > instances if that helps in any way. > > MATIAS LAINO | DIRECTOR OF PASSARE REMOTE DEVELOPMENT > matias.la...@passare.com | +54 11-6357-2143 > > > -----Original Message----- > From: Dave <hastings.recurs...@gmail.com> > Sent: Wednesday, December 7, 2022 2:40 PM > To: users@solr.apache.org > Subject: Re: Near Real Time not working as expected > > Just out of curiosity are you using metal? And if so ran any disk io tests > to see if you may have a hardware problem on any of the nodes? A document > won’t be available until all the nodes have it so it just takes one to get > slow to slow you down > > > On Dec 7, 2022, at 9:45 AM, Matias Laino <matias.la...@passare.com.invalid> > wrote: > > > > > > Hi all, > > > > I recently had an issue with very high cpu usage on our Testing > SolrCloud cluster when sending data to Solr, I’ve tried several which > reduced the usage of CPU, now our testing SolrCloud is under an 8 core > machine with 32 gb de RAM (recently changed the heap to 21g as a test). > > When we push data to solr, it takes a couple of minutes for that > document to be available on search results, I’ve tried everything and > cannot find out what is going on, it was working perfectly fine until last > week when it suddenly started having this delay. > > > > Our configuration for NRT is very aggressive, 60s of auto commit with > open searcher false and 1s for auto soft commit, but it doesn’t matter what > configuration I try, it will always take a couple of minutes to have the > new document available on search results. > > > > I’ve tried modifying the cache configuration to use Caffeine, tried > removing max warming searchers values, tried modifying autoWarmCount to > different values and even tried, and still the same issue, it’s almost like > my configuration doesn’t matter. > > > > We are using a Solr 8.11 install in SolrCloud mode, 2 nodes, 1 Zookeeper > node. On each node we have 6 collections of around 10-11M records each > (numbers didn’t change much before and after this issue started). The total > amount of disk spaced used is 20.4gb, our heap is now 21gb. > > > > I’m kind of desperate since I’ll be on vacation starting the end of next > week and I haven’t been able to find out what is wrong with this, my fear > is if this happens to our production server, we won’t be able to know how > to fix it other than reinstalling Solr from scratch. > > > > Our prod server only has 1 collection of 11gb and is under a 4 core > servers with 16 gb of ram (8gb heap setup). > > > > Any help or pointer will be highly appreciated as I’m desperate. > > > > Thanks in advance! > > > > MATIAS LAINO | DIRECTOR OF PASSARE REMOTE DEVELOPMENT > > matias.la...@passare.com | +54 11-6357-2143 > > > > >
