On 10/26/21 6:10 AM, Paul Russell wrote:
I have a current SOLR cluster running SOLR 6.6 on RHEL 6 servers. All SOLR
instances use a 25G JVM on the RHEL 6 server configured with 64G of memory
managing a 900G collection. Measured response time to queries average about
100ms.
Congrats on getting that performance. With the numbers you have
described, I would not expect to see anything that good.
On the RHEL 7 servers the kswapd0 process is consuming up to 30% of the CPU
and response time is being measured at 500-1000 ms for queries.
How long are you giving the system, and how many queries have been
handled by the cluster before you begin benchmarking? The only way the
old cluster could see performance that good is handling a LOT of queries
... enough that the OS can figure out how to effectively cache the index
with limited memory. By my calculations, your systems have less than
40GB of free memory to cache a 900GB index. And that assumes that Solr
is the only software running on these systems.
I tried using the vm.swappiness setting at both 0 and 1 and have been
unable to change the behavior.
Did you see any information other than kswapd0 CPU usage that led you to
this action? I would not expect swap to be the problem with this, and
your own experiments seem to say the same.
If I trim the SOLR JVM to 16Gb response
times get better and GC logs show the JVM is operating correctly..
Sounds like you have a solution. Is there a problem with simply
changing the heap size? If everything works with a lower heap size,
then the lower heap size is strongly encouraged. You seem to be making
a point here about the JVM operating correctly with a 16GB heap. Are
you seeing something in GC logs to indicate incorrect operation with the
higher heap? Solr 6.x uses CMS for garbage collection. You might see
better GC performance by switching to G1. Switching to another collector
would require a much newer Java version, one that is probably not
compatible with Solr 6.x. Here is the GC_TUNE setting (goes in
solr.in.sh) for newer Solr versions:
GC_TUNE=('-XX:+UseG1GC' \
'-XX:+PerfDisableSharedMem' \
'-XX:+ParallelRefProcEnabled' \
'-XX:MaxGCPauseMillis=250' \
'-XX:+UseLargePages' \
'-XX:+AlwaysPreTouch' \
'-XX:+ExplicitGCInvokesConcurrent')
If your servers have more than one physical CPU and NUMA architecture,
then I would strongly recommend adding "-XX:+UseNUMA" to the argument
list. Adding it on systems with only one NUMA node will not cause problems.
I would not expect the problem to be in the OS, but I could be wrong.
It is possible that changes in the newer kernel make it less efficient
at figuring out proper cache operation, and that would affect Solr.
Usually things get better with an upgrade, but you never know.
It seems more likely to be some other difference between the systems.
Top culprit in my mind is Java. Are the two systems running the same
version of Java from the same vendor? What I would recommend for Solr
6.x is the latest OpenJDK 8. In the past I would have recommended
Oracle Java, but they changed their licensing, so now I go with
OpenJDK. Avoid IBM Java or anything that descends from it -- it is
known to have bugs running Lucene software. If you want to use a newer
Java version than Java 8, you'll need to upgrade Solr. Upgrading from
6.x to 8.x is something that requires extensive testing, and a complete
reindex from scratch.
I would be interested in seeing the screenshot described here:
https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue
RHEL uses gnu top.
My own deployments use Ubuntu. Back when I did have access to large
Solr installs, they were running on CentOS, which is effectively the
same as RHEL. I do not recall whether they were CentOS 6 or 7.
Thanks,
Shawn
