Re: How to change sorting *after* getting search results

2021-11-30 Thread Luís Filipe Nassif
one of the various > Rescorers. Have you looked at those? > > On Tue, Nov 30, 2021, 9:15 AM Luís Filipe Nassif > wrote: > >> Hi Lucene community, >> >> Our users could do very heavy searches and they are able to change the >> sorting criteria multiple times after ge

Re: How to change sorting *after* getting search results

2021-11-30 Thread Michael Sokolov
ery heavy searches and they are able to change the > sorting criteria multiple times after getting the results. We collect all > of them, this is important for our use case, disabling scoring if the > result size is too large to make the search faster. Currently we have our > own multi-thre

How to change sorting *after* getting search results

2021-11-30 Thread Luís Filipe Nassif
Hi Lucene community, Our users could do very heavy searches and they are able to change the sorting criteria multiple times after getting the results. We collect all of them, this is important for our use case, disabling scoring if the result size is too large to make the search faster. Currently

RE: early termination with query time sorting (but without index-time SortingMergePolicy)

2019-04-19 Thread Ayse Onalan
Formatting got messed up - fixed to make it more readable. -Original Message- From: Ayse Onalan Sent: Friday, April 19, 2019 1:29 PM To: java-user@lucene.apache.org Subject: early termination with query time sorting (but without index-time SortingMergePolicy) Hi Lucene users, I&#

early termination with query time sorting (but without index-time SortingMergePolicy)

2019-04-19 Thread Ayse Onalan
spec consistent with the segment's sort order, docs would be scored and collected in the sort property value order, and hence can be early terminated once page size is reached. This yields significant perf-gains. However index time sorting is limited to only one sort spec. We need the abili

Re: How to control missing value when sorting using *ValuesSource?

2019-04-02 Thread Adrien Grand
these SortFields, you get an > > Exception: > > throw new IllegalArgumentException("Missing value only > > works for numeric or STRING types"); > > > > But these *are* numeric types, or they seem that way. Internally they are > > label

Re: How to control missing value when sorting using *ValuesSource?

2019-03-31 Thread Michael Sokolov
t these *are* numeric types, or they seem that way. Internally they are > labeled as type CUSTOM. > > First, my question is whether there is a way to achieve the desired > behavior -- missing last semantics combined with sorting on ValuesSource? > > And if there isn't a good w

How to control missing value when sorting using *ValuesSource?

2019-03-27 Thread Michael Sokolov
G types"); But these *are* numeric types, or they seem that way. Internally they are labeled as type CUSTOM. First, my question is whether there is a way to achieve the desired behavior -- missing last semantics combined with sorting on ValuesSource? And if there isn't a good way, wo

Re: Possible bug/issue when index sorting is enabled.

2018-08-27 Thread Adrien Grand
per community, > > I may have hit a Lucene issue and I would like clarification if this is > indeed a bug. > > Steps: > 1. Enable index sorting for the index > 2. Add document with id=1 and yVal=Long.MaxValue > 3. Delete document with id, followed by add document with same id

Possible bug/issue when index sorting is enabled.

2018-08-26 Thread vish . ramachandran
Hi Lucene Developer community, I may have hit a Lucene issue and I would like clarification if this is indeed a bug. Steps: 1. Enable index sorting for the index 2. Add document with id=1 and yVal=Long.MaxValue 3. Delete document with id, followed by add document with same id with yVal=45. 4

Re: index sorting merge

2017-12-28 Thread Yonghui Zhao
Got it, thanks! my version is 4.10.4 which is tooo old 2017-12-29 5:31 GMT+08:00 Michael McCandless : > You should upgrade to newer versions of Lucene, where all segments are > sorted, not just merged segments. > > Mike McCandless > > http://blog.mikemccandless.com > > On Thu, Dec 28, 2017 at

Re: index sorting merge

2017-12-28 Thread Michael McCandless
You should upgrade to newer versions of Lucene, where all segments are sorted, not just merged segments. Mike McCandless http://blog.mikemccandless.com On Thu, Dec 28, 2017 at 11:13 AM, Yonghui Zhao wrote: > Hi, > > I specified a SortingMergePolicy in my case. I find only the first N-1 > segm

index sorting merge

2017-12-28 Thread Yonghui Zhao
Hi, I specified a SortingMergePolicy in my case. I find only the first N-1 segments are sorted as expected, the last segment is still disordered when I call forceMerge(N), N > 1, I think it is by design, but is there any way to make all segments sorted. Thanks !

Re: Maintaining sorting order (stored fields vs DocValue fields) while upgrading Lucene version

2017-07-02 Thread András Péteri
Hi, Note that If you are using Lucene directly, 5.x introduced LUCENE-6064 [1] [2], which adds checks to ensure that the sort field has a corresponding DocValue of the expected type. Indexed fields can only be used for sorting via an UninvertingReader, at a cost of increased heap usage [3]. Solr

Re: Maintaining sorting order (stored fields vs DocValue fields) while upgrading Lucene version

2017-06-29 Thread Erick Erickson
ike. Even in that case you'll have to upgrade eventually. And if you wind up re-indexing everything anyway, it seems like stopping at 5x is unnecessary. Best, Erick On Thu, Jun 29, 2017 at 6:45 PM, Florian Buetow wrote: > > Hi, > > > > I am in the process of updating a larg

Maintaining sorting order (stored fields vs DocValue fields) while upgrading Lucene version

2017-06-29 Thread Florian Buetow
Hi, I am in the process of updating a large index from Lucene 4.x to 5.x and have two questions related to the sorting order. 1. Is it correct that stored fields can only be sorted on if they become a DocValue field in 5.x? 2. When "updating" stored fields to DocValue fields , is i

Re: Docvalue - Sorting on Numeric Docvalue

2017-03-27 Thread Adrien Grand
y single-valued. Le ven. 24 mars 2017 à 10:19, aravinth thangasami < aravinththangas...@gmail.com> a écrit : > Hi all, > > I'm analysing sorting using doc values. > > Please correct me if I am wrong > > > 1.In SortedNumericDocvalueField, the sorting apply acr

Docvalue - Sorting on Numeric Docvalue

2017-03-24 Thread aravinth thangasami
Hi all, I'm analysing sorting using doc values. Please correct me if I am wrong 1.In SortedNumericDocvalueField, the sorting apply across field, not across segment 2.While Sorting the SortedNumericSelector forms a NumericDocvalue from SortedSetDocvaluesField So, I thought

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-23 Thread Erick Erickson
to lexicographic > sorting. i can understand, it is due to trie structure of LongField, > > But one more doubt, Will uninversion process happen in IntField / LongField > too? > > Thanks for the link mike. i will look into LongPoint in recent versions. > > -- > Kumaran R > &

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-23 Thread Kumaran Ramasubramanian
Thanks Erick and Mike. i am using lucene 4.10.4 directly. i have observed better performance in LongField compared to lexicographic sorting. i can understand, it is due to trie structure of LongField, But one more doubt, Will uninversion process happen in IntField / LongField too? Thanks for

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-23 Thread Michael McCandless
.. Mike McCandless http://blog.mikemccandless.com On Thu, Dec 22, 2016 at 10:37 PM, Erick Erickson wrote: > bq: Does this mean LongField/IntField just supports lexicographic > order in sorting? > > no on several counts. > > No numeric type (long, int, float, double or trie valu

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Erick Erickson
bq: Does this mean LongField/IntField just supports lexicographic order in sorting? no on several counts. No numeric type (long, int, float, double or trie values) support lexicographic sorting. That's the whole _point_ of having numeric types in the first place. Well, and efficient

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Kumaran Ramasubramanian
Thank you Adrien. "NumericDocValuesField is the one that supports sorting." Does this mean LongField/IntField just supports lexicographic order in sorting? - Kumaran R On Dec 22, 2016 11:28 PM, "Adrien Grand" wrote: Le jeu. 22 déc. 2016 à 18:50, Kumaran Ramasubrama

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Adrien Grand
Le jeu. 22 déc. 2016 à 18:50, Kumaran Ramasubramanian a écrit : > I want to provide sorting, range search and faceting in numeric fields. > > AFAIK, Purpose of different numeric field types are, > > NumericDocValuesField supports sorting and faceting > LongField/IntField suppor

Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Kumaran Ramasubramanian
Hi All, I want to provide sorting, range search and faceting in numeric fields. AFAIK, Purpose of different numeric field types are, NumericDocValuesField supports sorting and faceting LongField/IntField supports range query and sorting 1. Should i duplicate one field in above mentioned

Re: Problem sorting long integers

2016-12-14 Thread Jaime
100 : From: Jaime : Reply-To: java-user@lucene.apache.org : To: java-user@lucene.apache.org : Subject: Problem sorting long integers : : Hello, : : With Lucene 6.1.0, I'm trying to search sorting the results by a long column. : This is my indexing code: : : doc.add(new LongPoint(name, Long.

Re: Problem sorting long integers

2016-12-14 Thread Jaime
apache.org : Subject: Problem sorting long integers : : Hello, : : With Lucene 6.1.0, I'm trying to search sorting the results by a long column. : This is my indexing code: : : doc.add(new LongPoint(name, Long.parseLong(value))); : doc.add(new StoredField(name, value)); :

Re: Problem sorting long integers

2016-12-13 Thread Chris Hostetter
the order is wrong) : Date: Tue, 13 Dec 2016 18:30:09 +0100 : From: Jaime : Reply-To: java-user@lucene.apache.org : To: java-user@lucene.apache.org : Subject: Problem sorting long integers : : Hello, : : With Lucene 6.1.0, I'm trying to search sorting the results by a long column. : This

Problem sorting long integers

2016-12-13 Thread Jaime
Hello, With Lucene 6.1.0, I'm trying to search sorting the results by a long column. This is my indexing code: doc.add(new LongPoint(name, Long.parseLong(value))); doc.add(new StoredField(name, value)); doc.add(new NumericDocValuesField(SORT_FIELD_PREFIX + name, Long.pars

Re: BlockJoinQuery with sorting

2016-11-28 Thread ASKozitsin
https://lucene.apache.org/core/6_2_0/join/org/apache/lucene/search/join/ToParentBlockJoinSortField.html Fits Perfectly! Thanks for the link! -- View this message in context: http://lucene.472066.n3.nabble.com/BlockJoinQuery-with-sorting-tp4307405p4307672.html Sent from the Lucene - Java Users

Re: BlockJoinQuery with sorting

2016-11-28 Thread Mikhail Khludnev
price.value : 15, price.type : t3, isParent: 0} > - {id : 3, price.value : 40, price.type : t3, isParent: 0} > - {id : 3, isParent: 1} > > This solution works for everything, but if i want to make "global sorting", > I will face a problem: > Expected behaviour is that

Re: BlockJoinQuery with sorting

2016-11-28 Thread ASKozitsin
price.type : t3, isParent: 0} - {id : 3, isParent: 1} This solution works for everything, but if i want to make "global sorting", I will face a problem: Expected behaviour is that if I sort by price.value, I will retreive: for ascending: {id : 1}, {id : 3}, {id : 2} // find "

Re: BlockJoinQuery with sorting

2016-11-26 Thread Mikhail Khludnev
documents: > offering with price1, offering with price2, offering with price3 (all of > them have the same offering_id). > after this 3 documents, i put parent document with the same offering_id. > > So, everything is fine, I use BlockJoinQueries for retrieving ids without > duplicate

BlockJoinQuery with sorting

2016-11-25 Thread ASKozitsin
when I try to handle sorting by price value, I stuck with a problem. Seems, there is no ability to sort children documents without duplicates. (of course, i can use Set collection for collector result, but it is not really good). Please, help me to resolve the issue. Thanks in advance! -- View th

Re: sorting biginteger

2016-08-27 Thread Michael McCandless
; > 2016-08-27 19:51 GMT+02:00 Michael McCandless >> > : >> >> >> >> I think to sort properly you must also ensure all byte[] from those >> >> BigIntegers are the same length, and that you sign extend them? >> >> Mike McCandless &

Re: sorting biginteger

2016-08-27 Thread Michael McCandless
no reference for > sorting, > and SortedNumericDocValuesField accept long not biginteger. > > > I thought to sort so : > > BigInteger bi = (BigInteger) o; > byte[] b = bi.toByteArray(); > NumericUtils.bigIntToSortableBytes(bi, BigIntegerPoint.BYTES, b, 0); > doc.ad

sorting biginteger

2016-08-21 Thread Cristian Lorenzetto
I took a look for bigInteger point but i didnt see no reference for sorting, and SortedNumericDocValuesField accept long not biginteger. I thought to sort so : BigInteger bi = (BigInteger) o; byte[] b = bi.toByteArray(); NumericUtils.bigIntToSortableBytes(bi, BigIntegerPoint.BYTES, b, 0

Re: LUCENE-6766 index sorting and custom SortField

2016-08-17 Thread Michael McCandless
Index time sorting was always an experimental feature ... it's free to change, be removed, be restricted, etc. We should probably just keep it simple and require users to compile their sort needs into an indexed (as doc values) sort key. Mike McCandless http://blog.mikemccandless.com O

Re: LUCENE-6766 index sorting and custom SortField

2016-08-17 Thread Adrien Grand
Le mar. 16 août 2016 à 22:21, Andres de la Peña a écrit : > I think that autoserializable SortFields should be considerably easier to > use, keeping the purpose of LUCENE-6766: to ease the usage of index > sorting, thus not being an expert feature. > I don't think a feature

Re: LUCENE-6766 index sorting and custom SortField

2016-08-16 Thread Andres de la Peña
think that autoserializable SortFields should be considerably easier to use, keeping the purpose of LUCENE-6766: to ease the usage of index sorting, thus not being an expert feature. 2016-08-16 20:56 GMT+01:00 Michael McCandless : > It would be a custom Codec implementation, where your codec co

Re: LUCENE-6766 index sorting and custom SortField

2016-08-16 Thread Michael McCandless
n the >>> > > marshalled values of each of the columns in the primary key, so it >>> is not >>> > > trivial at all to compute an equivalent collated value to be indexed >>> in >>> > doc >>> > > values. >>> > >

Re: LUCENE-6766 index sorting and custom SortField

2016-08-16 Thread Andres de la Peña
gt;> > doc >> > > values. >> > > >> > > Maybe it could be possible to define how to do this >> > > serialization-deserialization when extending SortField. This way it >> will >> > be >> > > possible to recover this lost Lucene 5.x fe

Re: LUCENE-6766 index sorting and custom SortField

2016-08-16 Thread Michael McCandless
values. > > > > > > Maybe it could be possible to define how to do this > > > serialization-deserialization when extending SortField. This way it > will > > be > > > possible to recover this lost Lucene 5.x feature, don't you think so? > > > > &g

Re: LUCENE-6766 index sorting and custom SortField

2016-08-16 Thread Adrien Grand
ion-deserialization when extending SortField. This way it will > be > > possible to recover this lost Lucene 5.x feature, don't you think so? > > > > Thanks, > > > > 2016-08-14 23:09 GMT+01:00 Michael McCandless >: > > > >> Unfortunately, as of LUC

Re: LUCENE-6766 index sorting and custom SortField

2016-08-15 Thread Michael McCandless
n extending SortField. This way it will be > possible to recover this lost Lucene 5.x feature, don't you think so? > > Thanks, > > 2016-08-14 23:09 GMT+01:00 Michael McCandless : > >> Unfortunately, as of LUCENE-6766, index sorting only supports simple sort >> t

Re: LUCENE-6766 index sorting and custom SortField

2016-08-15 Thread Andres de la Peña
ou think so? Thanks, 2016-08-14 23:09 GMT+01:00 Michael McCandless : > Unfortunately, as of LUCENE-6766, index sorting only supports simple sort > types. This was needed because Lucene needs to be able to easily serialize > and de-serialize the sort order into the index. > > Can you

Re: LUCENE-6766 index sorting and custom SortField

2016-08-14 Thread Michael McCandless
Unfortunately, as of LUCENE-6766, index sorting only supports simple sort types. This was needed because Lucene needs to be able to easily serialize and de-serialize the sort order into the index. Can you compute your sort criteria and index it as a doc values field and then sort by that? Or

LUCENE-6766 index sorting and custom SortField

2016-08-14 Thread Andres de la Peña
Hi, LUCENE-6766 allows to define index sorting on IndexWriterConfig instead of defining a SortingMergePolicy. However, the new index sorting only supports some types of sort fields, and the old SortingMergePolicy, which didn't have this limitation, has been removed. What should do pro

Re: Sorting IndexSearcher results by LongPoint with 6.0

2016-05-27 Thread Uwe Schindler
able to do a range query on, sort by, and also retrieve in >the >document as a stored value, I will need to add it to the document three >times, as a NumericDocValuesField, as a LongPoint, and as a >StoredField. >Does that sound correct? > >On Thu, May 26, 2016 at 3:43 PM, Uwe

Re: Sorting IndexSearcher results by LongPoint with 6.0

2016-05-26 Thread Jeremy Friesen
. Does that sound correct? On Thu, May 26, 2016 at 3:43 PM, Uwe Schindler wrote: > Hi, > > Sorting does not work on indexed fields anymore (since Lucene 5), unless > you use UninvertingReader. Point values don't work with that because they > cannot be uninverted. > > For

Re: Sorting IndexSearcher results by LongPoint with 6.0

2016-05-26 Thread Uwe Schindler
Hi, Sorting does not work on indexed fields anymore (since Lucene 5), unless you use UninvertingReader. Point values don't work with that because they cannot be uninverted. For sorting it's the same rule for all field types: enable DocValues! You just have to add another field ins

Sorting IndexSearcher results by LongPoint with 6.0

2016-05-26 Thread Jeremy Friesen
I'm attempting to upgrade my project to Lucene 6.0, and have run into an issue with sorting my results. My documents have a timestamp field that was previously a StoredField with NumericType: Long. I've converted it to a LongPoint, which seems to work fine for range queries. My probl

Re: Sorting on child document field.

2016-05-20 Thread Pranaya Behera
Adding lucene user mailing list to it. On Thursday 19 May 2016 06:55 PM, Pranaya Behera wrote: Example would be: Lets say that I have a product document with regular fields as name, price, desc, is_parent. it has child documents such as CA:: fields as a,b,c,rank and another child document as

RE: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Uwe Schindler
Erickson > > Cc: java-user > > Subject: Re: Lucene 5.0.0 - StringField and Sorting > > > > Hi Erick, > > > > i guess you've muddled the lists - this is lucenes one, not solr. I know > > how to define it in solr but that wasn't the question as i am

RE: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Uwe Schindler
ons: - Reindex your stuff with DocValues fields enabled. This new field type is optimized for random access as used by sorting. This is the recommended approach. - Alternatively you can use UninvertingReader from the misc module and wrap your index reader. This "emulates" the Doc

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Torsten Krah
Thanks for going to fix the stale docs. Torsten Am Montag, den 25.04.2016, 11:09 -0400 schrieb Michael McCandless: > The Lucene level javadocs are definitely stale ... I'll fix. > > > You should separately add a SortedDocValuesField if you also need to > sort on this field. > > Mike McCandless

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Torsten Krah
Hi Erick, i guess you've muddled the lists - this is lucenes one, not solr. I know how to define it in solr but that wasn't the question as i am using pure lucene and it did not work as expected from the javadocs there. Cheers Torsten --

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Gimantha Bandara
but not tokenized: the entire > > * String value is indexed as a single token. For example > > * this might be used for a 'country' field or an 'id' > > * field, or any field that you intend to use for sorting > > * or access through the f

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Michael McCandless
the JavaDoc of StringField it says: > > /** A field that is indexed but not tokenized: the entire > * String value is indexed as a single token. For example > * this might be used for a 'country' field or an 'id' > * field, or any field that you intend to use fo

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Erick Erickson
This works fine for me in a current Solr (5x). What version are you using? Note that _any_ field you use for sorting _must_ be indexed="true" or, in recent Solrs, docValues="true", just like any other field. You can't sort on a field that only has stored='"true&

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Torsten Krah
but not tokenized: the entire > * String value is indexed as a single token. For example > * this might be used for a 'country' field or an 'id' > * field, or any field that you intend to use for sorting > * or access through

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-21 Thread Gimantha Bandara
avaDoc of StringField it says: > > /** A field that is indexed but not tokenized: the entire > * String value is indexed as a single token. For example > * this might be used for a 'country' field or an 'id' > * field, or any field that you intend to use f

Re: How does sorting work in Lucene?

2016-03-15 Thread Jake Clawson
-user@lucene.apache.org" Sent: Saturday, March 12, 2016 10:16 AM Subject: Re: How does sorting work in Lucene? We are using our own clustering mechanism using Hazelcast. Sorting works fine when the our server runs in standalone mode. Lucene returns the doc ids in the sorted order but the score is

Re: How does sorting work in Lucene?

2016-03-12 Thread Gimantha Bandara
We are using our own clustering mechanism using Hazelcast. Sorting works fine when the our server runs in standalone mode. Lucene returns the doc ids in the sorted order but the score is always 1.0. Is this expected? or am I doing something wrong? ( Please note that the doc id order is returned

Re: How does sorting work in Lucene?

2016-02-29 Thread Gimantha Bandara
cCollector then sort them > by the score and returning the Top scored Documents. I think using the > MultiReader can replace this logic. > > But I have some questions regarding sorting by a specific field/fields. > > 1. Does lucene sort at the search time or does it store sort i

How does sorting work in Lucene?

2016-02-28 Thread Gimantha Bandara
MultiReader can replace this logic. But I have some questions regarding sorting by a specific field/fields. 1. Does lucene sort at the search time or does it store sort information at the index time in some way? 2. How would I implement pagination for a sorted set of documents? I have several

Re: AW: Lucene 4.x -> 5 : IllegalStateException while sorting

2015-03-11 Thread shamik
27;ADSKDedup' (expected=SORTED). Use UninvertingReader or dex with docvalues." Does this mean that I need to re-index all the documents over again ? -- View this message in context: http://lucene.472066.n3.nabble.com/Lucene-4-x-5-IllegalStateException-while-sorting-tp4188210p

Lucene 5.0.0 - StringField and Sorting

2015-03-06 Thread Torsten Krah
Hi, looking at the JavaDoc of StringField it says: /** A field that is indexed but not tokenized: the entire * String value is indexed as a single token. For example * this might be used for a 'country' field or an 'id' * field, or any field that you intend to us

Lucene 5.0.0 - StringField and Sorting

2015-02-27 Thread Torsten Krah
Hi, looking at the JavaDoc of StringField it says: /** A field that is indexed but not tokenized: the entire * String value is indexed as a single token. For example * this might be used for a 'country' field or an 'id' * field, or any field that you intend to us

AW: Can't get distance sorting to work in Lucene Spatial 4.10.3

2015-02-26 Thread Simon Rainer
f: AW: Can't get distance sorting to work in Lucene Spatial 4.10.3 Hi David, thanks for these hints! Unfortunately that didn't change anything yet. I made the fixes that you suggested: val queryPoint = spatialCtx.makePoint(lon, lat) val args = new SpatialArgs(SpatialOpe

AW: Can't get distance sorting to work in Lucene Spatial 4.10.3

2015-02-25 Thread Simon Rainer
es(20, DistanceUtils.EARTH_MEAN_RADIUS_KM))) But while the swapped lon/lat would have been an explanation, it turns out that it doesn't matter what I set as the query point when defining the valueSource. No effect on the sorting. I wonder if I'm doing something wrong when indexing the sha

Re: Can't get distance sorting to work in Lucene Spatial 4.10.3

2015-02-25 Thread david.w.smi...@gmail.com
still a bug, perhaps start with the SpatialExample.java in a working state and iteratively modify it to work how you want it to, testing each time that the sort works. ~ David On Wed, Feb 25, 2015 at 7:18 AM, Simon Rainer wrote: > Hi! > > I have problems getting distance sorting

Can't get distance sorting to work in Lucene Spatial 4.10.3

2015-02-25 Thread Simon Rainer
Hi! I have problems getting distance sorting to work in Lucene Spatial. (I'm using v4.10.3.) I'm following the SpatialExample.java from the Lucene docs. My code is below (it's Scala, but translates 1:1 into Java). When I run the query, results don't seem to be affected b

AW: Lucene 4.x -> 5 : IllegalStateException while sorting

2015-02-23 Thread Clemens Wyss DEV
.de] Gesendet: Montag, 23. Februar 2015 17:46 An: java-user@lucene.apache.org Betreff: RE: Lucene 4.x -> 5 : IllegalStateException while sorting Hi, Solr uses DocValues and falls back to wrapping with UninvertingReader, if user have not indexed them (with negative startup performance and memory

RE: Lucene 4.x -> 5 : IllegalStateException while sorting

2015-02-23 Thread Uwe Schindler
-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Clemens Wyss DEV [mailto:clemens...@mysign.ch] > Sent: Monday, February 23, 2015 2:20 PM > To: java-user@lucene.apache.org > Subject: AW: Lucene 4.x -> 5 : IllegalStateExcep

AW: Lucene 4.x -> 5 : IllegalStateException while sorting

2015-02-23 Thread Clemens Wyss DEV
ile sorting Hi Clemens, I think this part of the release notes [1] applies to your case: * FieldCache is gone (moved to a dedicated UninvertingReader in the misc module). This means when you intend to sort on a field, you should index that field using doc values, which is much faster and less h

Re: Lucene 4.x -> 5 : IllegalStateException while sorting

2015-02-23 Thread András Péteri
than FieldCache. [1] https://wiki.apache.org/lucene-java/ReleaseNote50 On Mon, Feb 23, 2015 at 1:24 PM, Clemens Wyss DEV wrote: > After upgrading to Lucene 5 one of my unittest which tests sorting fails > with: > unexpected docvalues type NONE for field 'providertestfield'

Lucene 4.x -> 5 : IllegalStateException while sorting

2015-02-23 Thread Clemens Wyss DEV
After upgrading to Lucene 5 one of my unittest which tests sorting fails with: unexpected docvalues type NONE for field 'providertestfield' (expected=SORTED). Use UninvertingReader or index with docvalues What am I missing?

Re: Chinese sorting

2014-12-19 Thread Nils Knappmeier
Hi Tomoko, thank you for the detailed explanation and many thanks for trying out the analyzer for me. I think "Very good compared to Unicode codepoint based sorting" is good enough for me. I will just try and use that Analyzer and see how it satisfies our customer. Regards, N

Re: Chinese sorting

2014-12-18 Thread Tomoko Uchida
Yes, sorting Kanji is not so easy as Hiragana/Kanji. We simply expect that collators sort strings based on phonetics regardless of how they written in (Hiragana, Katakana, Kanji.) However a Kanji has multiple (usually 2 or 3) readings. We human naturally make judgement which reading is suitable

Re: Chinese sorting

2014-12-18 Thread Nils Knappmeier
Hi Tomoko, does sorting with Locala.JAPANESE also work for Kanji. Since Hiragana and Katakana are based on the phonetics, I guess it is easier to define a sorting order. But Kanji is more similar to the Chinese. Thanks, Nils On 17.12.2014 17:01, Tomoko Uchida wrote: Hi, Nils, I don&#

Re: Chinese sorting

2014-12-17 Thread Tomoko Uchida
s works for you... I would appreciate if you share your trial/research. Regards, Tomoko 2014-12-17 20:54 GMT+09:00 Nils Knappmeier : > > Hi, > > is there any implementation for a chinese collator in Lucene. I've seen > that there is a chinese analyzer which uses Hidden Markov Model

Chinese sorting

2014-12-17 Thread Nils Knappmeier
Hi, is there any implementation for a chinese collator in Lucene. I've seen that there is a chinese analyzer which uses Hidden Markov Models. But sorting seems to be an issue on its own and all my googling hasn't led to any results yet. I understand that this is not a trivial issu

Re: Lucene DocValuesField, SortedDocValuesField usage for filtering and sorting

2014-12-16 Thread Adrien Grand
On Tue, Dec 16, 2014 at 3:25 PM, Piotr Idzikowski wrote: > So for instance if I store documents with ie creation date and I have a > data (millions of documents) from last let's say 3 years and I'd like to do > range filter to get socs from some month only is it better to use ordinary > numeric qu

Re: Lucene DocValuesField, SortedDocValuesField usage for filtering and sorting

2014-12-16 Thread Piotr Idzikowski
> > >> So for instance if I store documents with ie creation date and I have a > data (millions of documents) from last let's say 3 years and I'd like to do > range filter to get socs from some month only is it better to use ordinary > numeric query instead of FieldCacheRangeQuery? > > Of course I

Re: Lucene DocValuesField, SortedDocValuesField usage for filtering and sorting

2014-12-16 Thread Piotr Idzikowski
make some optimization in my index and code. I would like to use > > DocValuesField to get values but also for filtering and sorting. So here > I > > have some questions: If I'd like to use range filter > > (FieldCacheRangeFilter) I need to store a value in XxxDocValuesField,

Re: Lucene DocValuesField, SortedDocValuesField usage for filtering and sorting

2014-12-16 Thread Adrien Grand
Hi Piotr, On Mon, Dec 15, 2014 at 9:43 PM, Piotr Idzikowski wrote: > Hello. > I am going to switch to newest (4.10.2) version of Lucene and I'd like to > make some optimization in my index and code. I would like to use > DocValuesField to get values but also for filtering and s

Lucene DocValuesField, SortedDocValuesField usage for filtering and sorting

2014-12-15 Thread Piotr Idzikowski
Hello. I am going to switch to newest (4.10.2) version of Lucene and I'd like to make some optimization in my index and code. I would like to use DocValuesField to get values but also for filtering and sorting. So here I have some questions: If I'd like to use range filter (FieldCacheR

Lucene DocValuesField, SortedDocValuesField usage for filtering and sorting

2014-12-15 Thread Piotr Idzikowski
Hello. I am going to switch to newest (4.10.2) version of Lucene and I'd like to make some optimization in my index and code. I would like to use DocValuesField to get values but also for filtering and sorting. So here I have some questions: If I'd like to use range filter (FieldCacheR

Re: Custom Sorting

2014-06-25 Thread Vitaly Funstein
As a compromise, you can base your custom sort function on values of stored fields in the same index - as opposed to fetching them from an external data store, or relying on internal sorting implementation in Lucene. It will still be relatively slow, but not nearly as slow as going out to a DB

Re: Custom Sorting

2014-06-25 Thread Erick Erickson
Sure, you can write a custom function, see: https://cwiki.apache.org/confluence/display/solr/Function+Queries And you can invoke your custom function since sorting by function is supported. But my point remains. To be performant, you'll have to cache the results. Which is what's

Re: Custom Sorting

2014-06-25 Thread Sandeep Khanzode
indexed by Lucene. Thanks again, --- Thanks n Regards, Sandeep Ramesh Khanzode On Wednesday, June 25, 2014 1:21 AM, Erick Erickson wrote: I'm a little confused here. Sure, sorting on a number of fields will increase memory, the basic idea here is that you need to cach

Re: Custom Sorting

2014-06-24 Thread Erick Erickson
I'm a little confused here. Sure, sorting on a number of fields will increase memory, the basic idea here is that you need to cache all the sort values (plus support structures) for performance reasons. If you create your own custom sort that goes out to a DB and gets the doc, you have

Custom Sorting

2014-06-24 Thread Sandeep Khanzode
Hi, I am trying to implement a sort order for search results in Lucene 4.7.2. If I want to use data for ordering that is not stored in Lucene as Fields, is there any way this can be done? Basically, I would have certain data that is associated logically to a document but stored elsewhere, like

Case-insensitive sorting - Lucene 4

2014-01-22 Thread Phil Herold
t java.io.IOException; import org.apache.lucene.index.IndexReader; import org.apache.lucene.search.FieldCache; import org.apache.lucene.search.FieldComparator; import org.apache.lucene.search.FieldComparatorSource; /** * Lucene FieldComparator for case-insensitive sorting. */ public class CaseInsensitiveStringComp

Sorting by locale

2014-01-03 Thread Mindaugas Žakšauskas
Hello, We are in progress of migrating from Lucene 3.6.2 to the latest 4.6.0. One of many issues we're having is the absence of SortField constructor which had Locale parameter. I was able to trace a similar problem/discussion at http://lucene.472066.n3.nabble.com/getLocale-of-SortField-td4076562

Re: Merging ordered segments without re-sorting.

2013-10-24 Thread Adrien Grand
https://issues.apache.org/jira/browse/LUCENE-4752?focusedCommentId=13605896&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13605896) and most of the time was not actually spent in sorting the doc IDs but merging stored fields (because we by-pass the specialized

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Arvind Kalyan
On Wed, Oct 23, 2013 at 2:45 PM, Adrien Grand wrote: > Hi, > > On Wed, Oct 23, 2013 at 10:19 PM, Arvind Kalyan wrote: > > Sorting is not an option for our case so we will most likely implement a > > variant that merges the segments in one pass. Using TimSort is great but &g

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Adrien Grand
Hi, On Wed, Oct 23, 2013 at 10:19 PM, Arvind Kalyan wrote: > Sorting is not an option for our case so we will most likely implement a > variant that merges the segments in one pass. Using TimSort is great but in > our case the 2 segments will be highly interspersed and would not benef

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Arvind Kalyan
Thanks again. Sorting is not an option for our case so we will most likely implement a variant that merges the segments in one pass. Using TimSort is great but in our case the 2 segments will be highly interspersed and would not benefit from the galloping in TimSort. In additional, if anyone

  1   2   3   4   5   6   7   8   >