[
https://issues.apache.org/jira/browse/LUCENE-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891998#action_12891998
]
Yonik Seeley commented on LUCENE-2458:
--------------------------------------
bq. Before, one ctor used the version specified, the other hardcoded LUCENE_24.
Ah.... and that constructor is the one that's used everywhere in Solr (leading
me to believe that leaving Solr's default alone was deliberate).
bq. As i said before, this shouldnt and cannot be "per-token" and such english
centric hacks do not belong in the analysis api.
The ability of a filter to say "this token is actually indexed as two adjacent
tokens" is fundamental and not related to any specific language.
It can be *used* for language specific hacks perhaps... but it is not a hack
itself.
I never mentioned issues of back compat, but of changes to Solr's default
behavior, which I continue to think is the best.
I think the best way forward is to add a CJK field to solr that defaults to the
opposite behavior (i.e. treats split tokens as completely separate).
> queryparser makes all CJK queries phrase queries regardless of analyzer
> -----------------------------------------------------------------------
>
> Key: LUCENE-2458
> URL: https://issues.apache.org/jira/browse/LUCENE-2458
> Project: Lucene - Java
> Issue Type: Bug
> Components: QueryParser
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Blocker
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2458.patch, LUCENE-2458.patch, LUCENE-2458.patch,
> LUCENE-2458.patch
>
>
> The queryparser automatically makes *ALL* CJK, Thai, Lao, Myanmar, Tibetan,
> ... queries into phrase queries, even though you didn't ask for one, and
> there isn't a way to turn this off.
> This completely breaks lucene for these languages, as it treats all queries
> like 'grep'.
> Example: if you query for f:abcd with standardanalyzer, where a,b,c,d are
> chinese characters, you get a phrasequery of "a b c d". if you use cjk
> analyzer, its no better, its a phrasequery of "ab bc cd", and if you use
> smartchinese analyzer, you get a phrasequery like "ab cd". But the user
> didn't ask for one, and they cannot turn it off.
> The reason is that the code to form phrase queries is not internationally
> appropriate and assumes whitespace tokenization. If more than one token comes
> out of whitespace delimited text, its automatically a phrase query no matter
> what.
> The proposed patch fixes the core queryparser (with all backwards compat
> kept) to only form phrase queries when the double quote operator is used.
> Implementing subclasses can always extend the QP and auto-generate whatever
> kind of queries they want that might completely break search for languages
> they don't care about, but core general-purpose QPs should be language
> independent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
