[
https://issues.apache.org/jira/browse/LUCENE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102524#comment-15102524
]
David Smiley commented on LUCENE-6979:
--------------------------------------
One proposal I have is that this helpful checking only occur when assertions
are enabled, since this seems like a good fit for that java feature.
Second, not mutually exclusive with the first, is that when
TokenStreamComponents catches this problem, it is in a position to close the
stream. This closes the entire stream, although an alternative might be to
localize this change to just the Tokenizer.setReader and maybe it could reset
itself, at the risk of not closing other possible filters that might need
closing.
> Tokenizer input state detection should reset state before throwing
> ------------------------------------------------------------------
>
> Key: LUCENE-6979
> URL: https://issues.apache.org/jira/browse/LUCENE-6979
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: David Smiley
>
> The Tokenizer will helpfully let you know that you're using it wrong in
> certain cases, like if you forget to close it. However it does this in a way
> that happens lazily (after the fact) but worse it keeps the state of the
> Tokenizer in a cranky state (i.e. if you try to use it again, you'll get an
> exception again). What makes this issue insidious is that Tokenizers are
> re-used via a ReuseStrategy in a ThreadLocal. So once you hit this bug,
> you're thread is, in a word, "poisoned". And what makes the stack trace a
> real head-scratcher is that it is not of the original "guilty" party that
> didn't close; it's likely some other caller, perhaps an indexing thread who
> isn't going to misuse the TokenStream, or at least hasn't yet. The error
> message could make that clearer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
