Michael McCandless created LUCENE-6209:
------------------------------------------
Summary: IndexWriter should confess when it stalls flushes
Key: LUCENE-6209
URL: https://issues.apache.org/jira/browse/LUCENE-6209
Project: Lucene - Core
Issue Type: Improvement
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: Trunk, 5.1
You tell IW how much RAM it's allowed to use to hold recently indexed documents
before they must be written to disk.
IW is willing to use up to 2X that amount for in-progress flushes.
If the in-progress flushes go over that limit, then IW will stall them,
hijacking indexing threads and having them wait until the in-progress flushes
are below 2X indexing buffer size again.
This is important back-pressure e.g. if you are indexing on a machine with many
cores but slowish IO.
Often when I profile an indexing heavy use case, even on fast IO (SSD) boxes, I
see the methods associated with this back-pressure taking unexpected time ...
yet IW never logs when it stalls/unstalls flushing. I think it should.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
