Re: Crashes caused by high deleted .dvd file mmap counts

Hi, you can try to further reduce the sharedArenaMaxPermits down to 1 (which restores the old behaviour). I don't know what Opensearch is doing with those many docvalues updates, but you're already very close at limits. If you need so many indexes in one machine and if you don't care of "slow

Re: Crashes caused by high deleted .dvd file mmap counts

Hi Uwe, Setting -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=64 didn't seem to help and we're still seeing restarts. A question about your response: what are normal update ratios? Each of our machines is running 32 OpenSearch shards (Lucene indexes), each with about 52 segments.

Re: Crashes caused by high deleted .dvd file mmap counts

Hi, Did the sharedArenaMaxPermits=64 help. Actually sorry for the answer, I did not recognize that you were talking about doc values updates. I just saw deleted. But basically the issue is the same: Every update or delete will create a new file belonging to same segment. As each segment by de

Re: Crashes caused by high deleted .dvd file mmap counts

On Wed, May 7, 2025 at 3:48 PM Justin Borromeo wrote: > > > One thing I don't understand is why does the list of deleted mmapped > fields only include doc values files? If your theory is correct and this > is caused by deletes being updated over and over, wouldn't we expect only > .liv files to b

Re: Crashes caused by high deleted .dvd file mmap counts

Hi Uwe, Thanks for the response. We've tried setting sharedArenaMaxPermits to 64; I'll update this thread once we get some data. One thing I don't understand is why does the list of deleted mmapped fields only include doc values files? If your theory is correct and this is caused by deletes bei

Re: Crashes caused by high deleted .dvd file mmap counts

Hi, this could be related to a bug or limitation of the following change: 1. GITHUB#13570 ,GITHUB#13574 ,GITHUB#13535 : Avoid performance degradation

Re: Crashes caused by high deleted .dvd file mmap counts

Hi Justin, This question is better for the OS community, as some of the settings are specific to OpenSearch. Will really appreciate, if you can create an OpenSearch issue . We can always follow up with the Lucene community, if it turns out

Crashes caused by high deleted .dvd file mmap counts

Hi all, After upgrading our OpenSearch cluster from 2.16.0 to 2.19.1 (moving from Lucene 9.10 to Lucene 9.12), our largest clusters started crashing with the following error: # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed t