On 8/10/2021 11:17 PM, Satya Nand wrote:
Thanks for explaining it so well. We will work on reducing the filter
cache size and auto warm count.
Though I have one question.
If your configured 4000 entry filterCache were to actually fill up, it
would require nearly 51 billion bytes, and that's just for the one
core
with 101 million documents. This is much larger than the 30GB
heap you
have specified ... I am betting that the filterCache is the reason
you're hitting OOME.
As you can see from the below screenshots the filter cache is almost
full and the heap is approx 18-20 GB. I think this means heap is not
actually taking 51 GB of space. Otherwise, the issue would have been
very frequent if the full cache had been taking ~50 GB of space. I
also believed the solr uses some compressed data structures to
accumulate its cache, That' how it is able to store the cache in less
memory. Isn't it?
Also, the issue is not very frequent. It comes once or twice a month,
Where all follower servers stop working at the same time due to
OutOfMemory error.
We can't see any of the images. The mailing list software stripped
them. Most attachments do not come through -- you'll need to find a
file sharing website and give us links. Dropbox is a good choice, and
there are others.
The cache may not be getting full, but each entry is over 12 megabytes
in size, so it will not need to be full to cause problems. It does not
get compressed. Solr (actually Lucene) does use compression in the
index file formats. It would be possible to compress the bitmap for a
filterCache entry, but that would slow things down when there is a cache
hit. I have no idea how much it would slow things down.
The cache warming probably isn't the problem. That's only going to
(temporarily) add 100 new entries to a new cache, then the old cache
will be gone. If the filterCache is indeed the major memory usage, it's
probably queries that cause it to get large.
I'm curious whether the 101 million document count is for one shard
replica or for the whole collection. How many documents are in all the
shard replicas handled by one Solr instance?
Thanks,
Shawn
