On 11/17/21 8:24 AM, Derek C wrote:
Now that you mention it..... Just after I sent the email I did start looking at top and I was seeing 100%, 200%, 300% CPU usage (and the VM only has 4 cores so I was looking at maxed out cores). I've grabbed screenshots now and I'm not seeing those high numbers but I've also grabbed a screenshot of the VM CPU usage (it's a VM in proxmox).
VMs make things a little muddy. There could be something going on at the VM host level that affects the VM, especially if the total amount of memory given to all the VMS is larger than the amount of memory actually installed on the VM host. Such oversubscription is extremely common in virtualized environments, and if that is happening, it will absolutely destroy performance across the board.
I'm not sure how to understand the solr_gc.log file (but I'd like to)
The best *easy* resource is what you were already told about -- the gceasy.io website. There are a number of other tools available that can analyze the logs locally with more detail, but the important bits are shown by gceasy. It's what I use.
There are no significant GC pauses. The amount of heap space used shown by the GC logs is extremely small, far smaller than the 12GB you had mentioned before. If that level of heap usage is typical, you could most likely drastically reduce the max heap size, leaving more memory available for the OS to use for disk caching. Effective disk caching is the key to making Solr perform well.
The solr.log file is missing. That contains a lot of the info I was hoping to see.
The screenshot from top looks like your Solr instance has somewhere in the ballpark of 20GB of index data, but there is only 4GB of data in the disk cache. That would be about right if you are using a 12GB heap on a system with 16GB of memory. Maybe the VM and/or the physical host doesn't have enough memory?
Thanks, Shawn
