msokolov commented on code in PR #12660:
URL: https://github.com/apache/lucene/pull/12660#discussion_r1367971976
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene95/Lucene95HnswVectorsWriter.java:
##########
@@ -50,13 +72,24 @@ public final class Lucene95HnswVectorsWriter extends
KnnVectorsWriter {
private final IndexOutput meta, vectorData, vectorIndex;
private final int M;
private final int beamWidth;
+ private final int numMergeWorkers;
+ private final ExecutorService exec;
private final List<FieldWriter<?>> fields = new ArrayList<>();
private boolean finished;
- Lucene95HnswVectorsWriter(SegmentWriteState state, int M, int beamWidth)
throws IOException {
+ Lucene95HnswVectorsWriter(SegmentWriteState state, int M, int beamWidth, int
numMergeWorkers)
+ throws IOException {
this.M = M;
this.beamWidth = beamWidth;
+ this.numMergeWorkers = numMergeWorkers;
+ if (numMergeWorkers <= 1) {
+ exec = null;
+ } else {
+ exec =
+ Executors.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors(), new
NamedThreadFactory("hnsw-merge"));
Review Comment:
So I *think* that these merge operations are triggered by a single
background merge thread? But we shouldn't rely on that and should provide
global control over resources dedicate to this, so something like that (thread
pool in Format) makes sense. However conceptually the Format doesn't seem right
to me - as a user, I would want to be able to control this from
IndexWriterConfig along with other merge-related parameters in IndexWriter like
buffer size and number of docs before flush. Possibly we could have other
concurrent processes running during background merge (like merging different
formats in parallel) and we don't want to have separate thread pools for all of
those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
