The field type is derived from the distributed alphaOnlySort as follows:
<fieldType name="alphaOnlySortLim" class="solr.TextField"
sortMissingLast="true" omitNorms="true">
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.TrimFilterFactory" />
<filter class="solr.PatternReplaceFilterFactory"
pattern="([\x00-\x2F\x3A-\x40\x5B-\x60\x7B-\x9F\u2000-\u206F\uFEFF\uFFF9-\uFFFD])"
replacement=""
replace="all"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="(.{1,30})(.{31,})"
replacement="$1"
replace="all"/>
</analyzer>
</fieldType>
It reduces long lists of author names (100 and more authors) to the first 30
chars
for sorting and removes some illegal chars to keep sorting with utf8 solid.
Don't see any problems there.
Will check with admin/analysis page.
Bernd
Am 12.11.2012 14:28, schrieb Erick Erickson:
> First, sorting on tokenized fields is undefined/unsupported. You _might_
> get away with it if the author field always reduces to one token, i.e. if
> you're always indexing only the last name.
>
> I should say unsupported/undefined when more than one token is the result
> of analysis. You can do things like use the KeywordTokenizer followed by
> tranformations on the _entire_ input field (lowercasing is popular for
> instance).
>
> So somehow the analysis chain you have defined for this field grabs
> "Arslanagic"
> and translates it into "a". Synonyms? Stemming? Some "interesting" sequence?
>
> The fastest way to look at that would be in Solr's admin/analysis page.
> Just put Arslanagic into the index box and you should see which of the
> steps does the translation. Although changing it to "a" is really weird,
> it's almost certainly something you've defined in the indexing analysis
> chain.
>
> FWIW,
> Erick
>
>
> On Mon, Nov 12, 2012 at 8:19 AM, Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de> wrote:
>
>> Hi list,
>> a user reported wrong sorting of our search service running on solr.
>> While chasing this issue I traced it back through lucene into the index.
>> I have a text field for sorting
>> (stored,indexed,tokenized,omitNorms,sortMissingLast)
>> and three docs with author names.
>>
>> If I trace at org.apache.lucene.document.Document.add(IndexableField) while
>> indexing I can see all three author names added as field to each documents.
>>
>> After searching with *:* for the three docs and doing a sort the sorting
>> is wrong
>> because one of the author names is reduced to the first char, all other
>> chars are lost.
>>
>> So having the authors names (Alexander, Arslanagic, Brennmoen) indexed,
>> the result
>> of sorting ascending is (Arslanagic, Alexander, Brennmoen) which is wrong.
>> But this happens because the author "Arslanagic" is reduced to "a" during
>> indexing (???)
>> and if sorted "a" is before "alexander".
>>
>> Currently I use 4.0 but have the same issue with 3.6.1.
>>
>> Without tracing through tons of code:
>> - which is the last breakpoint for debugging to see the docs right before
>> they go into the index
>> - which is the first breakpoint for debugging to see the docs coming right
>> out of the index
>>
>> Regards
>> Bernd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
>
--
*************************************************************
Bernd Fehling Bielefeld University Library
Dipl.-Inform. (FH) LibTec - Library Technology
Universitätsstr. 25 and Knowledge Management
33615 Bielefeld
Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
