magibney commented on code in PR #13570:
URL: https://github.com/apache/lucene/pull/13570#discussion_r1682897378
##########
lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java:
##########
@@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) {
return filename;
}
+ /**
Review Comment:
> In addition about deletions and in-place updates: There is the ability to
extract the "generation" of a file. If the generation is not 0, the file is an
update to an already existing segment (like new deletions and docvalues
updates). As those should not be stay open forever, those should be a separate
key. Or alternatively handle all files with generation!=0 as a separate shared
arena.
We could have 2 Arenas per segment: 1 for segment files _without_
generation, 1 for segment files _with_ generation -- and the Arena for segment
files _with_ generation could have some arbitrary built-in limit to the number
of times (e.g., 1k?) the shared Arena can be used before `acquire()` returns
false. This would cause Arenas to get rotated out periodically, ensuring that
virtual memory would always be freed eventually.
In fact, I wonder if we might want some arbitrary limit to the number of
mappings allowed for _each_ `RefCountedSharedArena` (not just the
"with-generation" ones) ... I'm not sure -- if you map the same file with the
same Arena multiple times, does the Arena itself do any deduplication? If one
were to open a single Arena, and then use that Arena to repeatedly map the same
2G file, would the system eventually run out of virtual memory space?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
