Just out of curiosity are you using metal? And if so ran any disk io tests to see if you may have a hardware problem on any of the nodes? A document won’t be available until all the nodes have it so it just takes one to get slow to slow you down
> On Dec 7, 2022, at 9:45 AM, Matias Laino <matias.la...@passare.com.invalid> > wrote: > > > Hi all, > > I recently had an issue with very high cpu usage on our Testing SolrCloud > cluster when sending data to Solr, I’ve tried several which reduced the usage > of CPU, now our testing SolrCloud is under an 8 core machine with 32 gb de > RAM (recently changed the heap to 21g as a test). > When we push data to solr, it takes a couple of minutes for that document to > be available on search results, I’ve tried everything and cannot find out > what is going on, it was working perfectly fine until last week when it > suddenly started having this delay. > > Our configuration for NRT is very aggressive, 60s of auto commit with open > searcher false and 1s for auto soft commit, but it doesn’t matter what > configuration I try, it will always take a couple of minutes to have the new > document available on search results. > > I’ve tried modifying the cache configuration to use Caffeine, tried removing > max warming searchers values, tried modifying autoWarmCount to different > values and even tried, and still the same issue, it’s almost like my > configuration doesn’t matter. > > We are using a Solr 8.11 install in SolrCloud mode, 2 nodes, 1 Zookeeper > node. On each node we have 6 collections of around 10-11M records each > (numbers didn’t change much before and after this issue started). The total > amount of disk spaced used is 20.4gb, our heap is now 21gb. > > I’m kind of desperate since I’ll be on vacation starting the end of next week > and I haven’t been able to find out what is wrong with this, my fear is if > this happens to our production server, we won’t be able to know how to fix it > other than reinstalling Solr from scratch. > > Our prod server only has 1 collection of 11gb and is under a 4 core servers > with 16 gb of ram (8gb heap setup). > > Any help or pointer will be highly appreciated as I’m desperate. > > Thanks in advance! > > MATIAS LAINO | DIRECTOR OF PASSARE REMOTE DEVELOPMENT > matias.la...@passare.com | +54 11-6357-2143 > >
