Re: SOLR Performance on RHEL 7

2021-12-08 Thread Paul Russell
Elaine, Thanks for the feedback. That's troubling to hear that your issues with SOLR and RHEL 7 weren' resolved. Perhaps I should pursue an alternate O/S as well if needed I finally got the customer to add the 64Gb SDD swap volume. Strange thing is with both the O/S tools and a Grafana node e

Re: SOLR Performance on RHEL 7

2021-12-08 Thread Elaine Cario
I'm late to the dance but FWIW, we also experienced some similar swap-like issues when we upgraded from Centos 7.6 to Centos 7.9 (this was Solr 8.3) - some of the Solr nodes would end up reading from disk like crazy, and query response times would suffer accordingly. At one point we had 1/2 the no

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Shawn Heisey
On 10/26/21 10:58 AM, Paul Russell wrote: Currently we are averaging about 5.5k requests a minute for this collection that is supported by a 3 node SOLR cluster. RHEL6 (Current Servers) and RHEL 7 (New Servers) are both utilizing OpenJDK8. Older servers have an older version 8.131 new servers ha

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Walter Underwood
How big are the indexes? Improving performance with a smaller heap could mean that the indexes were not fitting in the file buffers. You can verify this by looking at iostats with the different heap sizes. There should be almost no disk reads while Solr is handling queries. If there is disk IO,

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Dave
I have always preferred completely turning off swap on solr dedicated machines, and especially if you can’t use an SSD. > On Oct 26, 2021, at 12:59 PM, Paul Russell wrote: > > Thanks for all the helpful information. > > Currently we are averaging about 5.5k requests a minute for this collect

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Paul Russell
Thanks for all the helpful information. Currently we are averaging about 5.5k requests a minute for this collection that is supported by a 3 node SOLR cluster. RHEL6 (Current Servers) and RHEL 7 (New Servers) are both utilizing OpenJDK8. Older servers have an older version 8.131 new servers have

Re: SOLR Performance on RHEL 7

2021-10-26 Thread dmitri maziuk
On 2021-10-26 10:24 AM, Shawn Heisey wrote: ... I don't think swap is the problem.  Disabling swap entirely would be a good test to confirm.  For general server use cases, I would not recommend that action, but for dedicated systems with plenty of memory like what is described in this thread, r

Re: SOLR Performance on RHEL 7

2021-10-26 Thread matthew sporleder
On Tue, Oct 26, 2021 at 11:25 AM Shawn Heisey wrote: > > On 10/26/21 8:34 AM, Michael Gibney wrote: > > In my experience, running Solr on CentOS 7 (comparable to RHEL 7) -- on > > VMWare, but "ballooning" was _not_ the issue -- I found that setting > > vm.swappiness=0 or 1 did not actually prevent

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Shawn Heisey
On 10/26/21 8:34 AM, Michael Gibney wrote: In my experience, running Solr on CentOS 7 (comparable to RHEL 7) -- on VMWare, but "ballooning" was _not_ the issue -- I found that setting vm.swappiness=0 or 1 did not actually prevent swapping. Notwithstanding Shawn's excellent suggestions above, if y

Re: SOLR Performance on RHEL 7

2021-10-26 Thread matthew sporleder
On Tue, Oct 26, 2021 at 8:11 AM Paul Russell wrote: > > I have a current SOLR cluster running SOLR 6.6 on RHEL 6 servers. All SOLR > instances use a 25G JVM on the RHEL 6 server configured with 64G of memory > managing a 900G collection. Measured response time to queries average about > 100ms. > >

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Michael Gibney
In my experience, running Solr on CentOS 7 (comparable to RHEL 7) -- on VMWare, but "ballooning" was _not_ the issue -- I found that setting vm.swappiness=0 or 1 did not actually prevent swapping. Notwithstanding Shawn's excellent suggestions above, if you still suspect that swapping is the issue a

Re: SOLR Performance on RHEL 7

2021-10-26 Thread Shawn Heisey
On 10/26/21 6:10 AM, Paul Russell wrote: I have a current SOLR cluster running SOLR 6.6 on RHEL 6 servers. All SOLR instances use a 25G JVM on the RHEL 6 server configured with 64G of memory managing a 900G collection. Measured response time to queries average about 100ms. Congrats on getting t