Hello, I'm witnessing a change in behavior between Lucene 4.9 and 5.4.1 that I don't quite understand. I'd like to track down what's happening under the hood. I'm working to update the dependencies of an open source geospatial resolution tool ( https://github.com/Berico-Technologies/CLAVIN), which uses Lucene. I've indexed the geonames.org database using both Lucene 4.9 and 5.4.1. We index on the Population of each city for later sorting on query.
When running a fuzzy query "bostn~" with Occur.MUST in 4.9, we get the expected result of Boston, where 6793534 is a boosted population. Here is the scoreDoc.toString(): *Boston: doc=19586055 score=NaN shardIndex=-1 fields=[2.971942, 6793534]* However, using 5.4.1, the fuzzy match with Occur.MUST returns "Basti Bosan" and "Boston Basin", both of which have a population of zero before returning Boston. *Basti Bosan: doc=11707183 score=NaN shardIndex=0 fields=[1.5721874, 0]* *Boston Basin: doc=12728320 score=NaN shardIndex=0 fields=[1.5721874, 0]Boston: doc=17515475 score=NaN shardIndex=0 fields=[1.4374285, 6793534]* I'm wondering if something with the FIELD_SCORE calculation changed between 4.9 and 5.4.1, or perhaps I've done something incorrect in building the index, etc. It's worth mentioning that for this test I have built an index w/ both 4.9 and 5.4.1 using the same geonames database to ensure consistency. Also, sort is set up with both versions in the same way: *private static final Sort POPULATION_SORT = new Sort(new SortField[] { SortField.FIELD_SCORE, * * new SortedNumericSortField(SORT_POP.key(), SortField.Type.LONG, true) * *});* With regard to building the index, in 4.9, we added the population sort field to the index like so: *doc.add(new LongField(SORT_POP.key(), geoName.getPopulation(), Field.Store.YES));* Because you can't sort on docValue = NONE anymore, in 5.4.1, we now add it like this: *doc.add(new LongField(SORT_POP.key(), geoName.getPopulation(), LONG_FIELD_TYPE_STORED_SORTED));* where LONG_FIELD_TYPE_STORED_SORTED is: *private static final FieldType LONG_FIELD_TYPE_STORED_SORTED = new FieldType();* *static { LONG_FIELD_TYPE_STORED_SORTED.setTokenized(false); LONG_FIELD_TYPE_STORED_SORTED.setOmitNorms(true); LONG_FIELD_TYPE_STORED_SORTED.setIndexOptions(IndexOptions.DOCS); LONG_FIELD_TYPE_STORED_SORTED .setNumericType(FieldType.NumericType.LONG);LONG_FIELD_TYPE_STORED_SORTED.setStored(true);LONG_FIELD_TYPE_STORED_SORTED.setDocValuesType(DocValuesType.NUMERIC);LONG_FIELD_TYPE_STORED_SORTED.freeze();}* I would greatly appreciate any insights here; and I'm happy to answer questions to unravel this a bit more. Thank you for your time! V/r, Jeremy