RayBB commented on PR #5293:
URL: https://github.com/apache/solr/pull/5293#issuecomment-6957899920

   On Open Library (Solr 10.0.0, leader/follower), our hourly job does in-place 
doc-values updates, and over 13 days that filled the leader's disk with 66 GB 
of deleted files that only a restart freed. The cause is in the Lucene used by 
Solr 10.0 and 10.1: [`MMapDirectory` groups every update-generation file of a 
segment into one shared 
arena](https://github.com/apache/lucene/blob/releases/lucene/10.3.2/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L122-L139),
 and in practice that arena is never released, so superseded `.dvd` files stay 
memory-mapped and their disk space never comes back. Lucene 10.5 fixed the 
permit cap 
([apache/lucene#15908](https://github.com/apache/lucene/pull/15908)), so this 
PR fixes it for Solr. In a small repro (one segment, update one doc value, 
commit, reopen the reader, repeat):
   
   | Lucene | Deleted `.dvd` files still mapped |
   |---|---|
   | 10.4.0 (current Solr) | grows by 1 per update: 49 after 50, 199 after 200 |
   | 10.6.0 (this PR) | peaks at 19, then 0 from about update 25 through 200 |
   
   Id love to see this merged and in the next 10.x release. Until then, anyone 
stuck on Solr 10.0/10.1 can work around it with 
`-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1`, which gives 
each file its own arena (0 leaked files in the repro). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to