Hi Shawn, Please find the images. *Filter cache stats:* https://drive.google.com/file/d/19MHEzi9m3KS4s-M86BKFiwmnGkMh3DGx/view?usp=sharing *Heap stats* https://drive.google.com/file/d/1Q62ea-nFh9UjbcVcBJ39AECWym6nk2Yg/view?usp=sharing
I'm curious whether the 101 million document count is for one shard > replica or for the whole collection. How many documents are in all the > shard replicas handled by one Solr instance? We are not using solr cloud. We are using standalone solr with Master-slave architecture. 101 million documents are in one core. On Wed, Aug 11, 2021 at 5:20 PM Shawn Heisey <apa...@elyograg.org> wrote: > On 8/10/2021 11:17 PM, Satya Nand wrote: > > Thanks for explaining it so well. We will work on reducing the filter > > cache size and auto warm count. > > > > Though I have one question. > > > > If your configured 4000 entry filterCache were to actually fill up, > it > > would require nearly 51 billion bytes, and that's just for the one > > core > > with 101 million documents. This is much larger than the 30GB > > heap you > > have specified ... I am betting that the filterCache is the reason > > you're hitting OOME. > > > > > > As you can see from the below screenshots the filter cache is almost > > full and the heap is approx 18-20 GB. I think this means heap is not > > actually taking 51 GB of space. Otherwise, the issue would have been > > very frequent if the full cache had been taking ~50 GB of space. I > > also believed the solr uses some compressed data structures to > > accumulate its cache, That' how it is able to store the cache in less > > memory. Isn't it? > > > > Also, the issue is not very frequent. It comes once or twice a month, > > Where all follower servers stop working at the same time due to > > OutOfMemory error. > > We can't see any of the images. The mailing list software stripped > them. Most attachments do not come through -- you'll need to find a > file sharing website and give us links. Dropbox is a good choice, and > there are others. > > The cache may not be getting full, but each entry is over 12 megabytes > in size, so it will not need to be full to cause problems. It does not > get compressed. Solr (actually Lucene) does use compression in the > index file formats. It would be possible to compress the bitmap for a > filterCache entry, but that would slow things down when there is a cache > hit. I have no idea how much it would slow things down. > > The cache warming probably isn't the problem. That's only going to > (temporarily) add 100 new entries to a new cache, then the old cache > will be gone. If the filterCache is indeed the major memory usage, it's > probably queries that cause it to get large. > > I'm curious whether the 101 million document count is for one shard > replica or for the whole collection. How many documents are in all the > shard replicas handled by one Solr instance? > > Thanks, > Shawn > > > --
