Elaine, Thanks for the feedback. That's troubling to hear that your issues with SOLR and RHEL 7 weren' resolved..... Perhaps I should pursue an alternate O/S as well if needed
I finally got the customer to add the 64Gb SDD swap volume. Strange thing is with both the O/S tools and a Grafana node exporter running I don't see any usage of swap at all. Because all storage was switched to SSD and I just brought the RHEL 7 node into the cluster and I don't know if I have a fix yet. I appreciate the similarities.. Tomorrow will tell the tale pretty quickly. On Wed, Dec 8, 2021 at 6:13 PM Elaine Cario <etca...@gmail.com> wrote: > I'm late to the dance but FWIW, we also experienced some similar swap-like > issues when we upgraded from Centos 7.6 to Centos 7.9 (this was Solr 8.3) - > some of the Solr nodes would end up reading from disk like crazy, and query > response times would suffer accordingly. At one point we had 1/2 the nodes > (with 1 set of replicas) on 7.6 and the other 1/2 (with 2nd replicas) on > 7.9, and could see disk reads and io waits an order of magnitude higher on > 7.9, with all other things being equal. > > We never really solved it: after countless weeks of testing > various configurations, we threw up our hands and started migrating > everything to Amazon Linux 2 (there were other reasons for that, but this > was a definite driver). We also have some servers still hosting RedHat > Enterprise 7.9 so far without issues but these are also slated for > migration in the coming weeks. > > On Tue, Oct 26, 2021 at 8:11 AM Paul Russell <paul.russ...@qflow.com> > wrote: > > > I have a current SOLR cluster running SOLR 6.6 on RHEL 6 servers. All > SOLR > > instances use a 25G JVM on the RHEL 6 server configured with 64G of > memory > > managing a 900G collection. Measured response time to queries average > about > > 100ms. > > > > I am attempting to move the cluster to new RHEL 7 servers with the same > > configuration (8 cores/ 64G memory) and having performance issues. > > > > On the RHEL 7 servers the kswapd0 process is consuming up to 30% of the > CPU > > and response time is being measured at 500-1000 ms for queries. > > > > I tried using the vm.swappiness setting at both 0 and 1 and have been > > unable to change the behavior. If I trim the SOLR JVM to 16Gb response > > times get better and GC logs show the JVM is operating correctly.. > > > > Has anyone else had a similar issue? I have tried upgrading to SOLR 7.7.2 > > as part of the process and that hasn't helped. > > > > Any suggestions? > > >
