Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-23 Thread Erick Erickson
And I frequently forget which list I'm on. Siiigggh... Listen to Mike ;) Best, Erick On Fri, Dec 23, 2016 at 3:40 AM, Kumaran Ramasubramanian wrote: > Thanks Erick and Mike. i am using lucene 4.10.4 directly. > > > i have observed better performance in LongField compared to lexicographic > sort

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-23 Thread Kumaran Ramasubramanian
Thanks Erick and Mike. i am using lucene 4.10.4 directly. i have observed better performance in LongField compared to lexicographic sorting. i can understand, it is due to trie structure of LongField, But one more doubt, Will uninversion process happen in IntField / LongField too? Thanks for th

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-23 Thread Michael McCandless
Note that Erick is giving you the Solr syntax below, but if you are using Lucene directly, that obviously doesn't apply (though the same general concepts do). I would strongly recommend not using uninversion: it's an archaic and costly option that Lucene only offered long ago because it didn't hav

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Erick Erickson
bq: Does this mean LongField/IntField just supports lexicographic order in sorting? no on several counts. No numeric type (long, int, float, double or trie values) support lexicographic sorting. That's the whole _point_ of having numeric types in the first place. Well, and efficient range queries

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Kumaran Ramasubramanian
Thank you Adrien. "NumericDocValuesField is the one that supports sorting." Does this mean LongField/IntField just supports lexicographic order in sorting? - Kumaran R On Dec 22, 2016 11:28 PM, "Adrien Grand" wrote: Le jeu. 22 déc. 2016 à 18:50, Kumaran Ramasubramanian a écrit : > I want

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

2016-12-22 Thread Adrien Grand
Le jeu. 22 déc. 2016 à 18:50, Kumaran Ramasubramanian a écrit : > I want to provide sorting, range search and faceting in numeric fields. > > AFAIK, Purpose of different numeric field types are, > > NumericDocValuesField supports sorting and faceting > LongField/IntField supports range query and

Re: sorting biginteger

2016-08-27 Thread Michael McCandless
Points cannot sort. They can only do range queries. So you need to convert your 128 bit BigInts into a sort-order-preserving byte[] (BigIntegerPoint.encodeDimension should do this correctly), and then index that byte[] using a SortedDocValuesField. Mike McCandless http://blog.mikemccandless.com

Re: sorting biginteger

2016-08-27 Thread Michael McCandless
I think to sort properly you must also ensure all byte[] from those BigIntegers are the same length, and that you sign extend them? Mike McCandless http://blog.mikemccandless.com On Sun, Aug 21, 2016 at 5:50 AM, Cristian Lorenzetto wrote: > I took a look for bigInteger point but i didnt see no

Re: Sorting IndexSearcher results by LongPoint with 6.0

2016-05-27 Thread Uwe Schindler
Hi Jeremy, Yes. That's right. The question is if you really need the stored field, but that's out of scope for this issue. Uwe Am 27. Mai 2016 01:21:48 MESZ, schrieb Jeremy Friesen : >Thanks for the help. So just to sum up, if I have a numeric field type >that >I want to be able to do a range q

Re: Sorting IndexSearcher results by LongPoint with 6.0

2016-05-26 Thread Jeremy Friesen
Thanks for the help. So just to sum up, if I have a numeric field type that I want to be able to do a range query on, sort by, and also retrieve in the document as a stored value, I will need to add it to the document three times, as a NumericDocValuesField, as a LongPoint, and as a StoredField. Do

Re: Sorting IndexSearcher results by LongPoint with 6.0

2016-05-26 Thread Uwe Schindler
Hi, Sorting does not work on indexed fields anymore (since Lucene 5), unless you use UninvertingReader. Point values don't work with that because they cannot be uninverted. For sorting it's the same rule for all field types: enable DocValues! You just have to add another field instance with sa

Re: Sorting on child document field.

2016-05-20 Thread Pranaya Behera
Adding lucene user mailing list to it. On Thursday 19 May 2016 06:55 PM, Pranaya Behera wrote: Example would be: Lets say that I have a product document with regular fields as name, price, desc, is_parent. it has child documents such as CA:: fields as a,b,c,rank and another child document as

Re: Merging ordered segments without re-sorting.

2013-10-24 Thread Adrien Grand
Hi, On Thu, Oct 24, 2013 at 12:20 AM, Arvind Kalyan wrote: > I will benchmark the available approach itself then, in that case. Will > revert back if the performance in unacceptable. For the record, last time I checked, indexing was 2x slower on average on a 10M document collection (see https://

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Arvind Kalyan
On Wed, Oct 23, 2013 at 2:45 PM, Adrien Grand wrote: > Hi, > > On Wed, Oct 23, 2013 at 10:19 PM, Arvind Kalyan wrote: > > Sorting is not an option for our case so we will most likely implement a > > variant that merges the segments in one pass. Using TimSort is great but > in > > our case the 2

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Adrien Grand
Hi, On Wed, Oct 23, 2013 at 10:19 PM, Arvind Kalyan wrote: > Sorting is not an option for our case so we will most likely implement a > variant that merges the segments in one pass. Using TimSort is great but in > our case the 2 segments will be highly interspersed and would not benefit > from th

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Arvind Kalyan
is that SortingMergePolicy performs sorting > after > > wrapping the 2 segments, correct? > > > > As I mentioned in my original email I would like to avoid the re-sorting > > and exploit the fact that the input segments are already sorted. > > > > > &g

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Shai Erera
is that SortingMergePolicy performs sorting after > wrapping the 2 segments, correct? > > As I mentioned in my original email I would like to avoid the re-sorting > and exploit the fact that the input segments are already sorted. > > > > On Wed, Oct 23, 2013 at 11:02

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Arvind Kalyan
Thanks, my understanding is that SortingMergePolicy performs sorting after wrapping the 2 segments, correct? As I mentioned in my original email I would like to avoid the re-sorting and exploit the fact that the input segments are already sorted. On Wed, Oct 23, 2013 at 11:02 AM, Shai Erera

Re: Merging ordered segments without re-sorting.

2013-10-23 Thread Shai Erera
Hi You can use SortingMergePolicy and SortingAtomicReader to achieve that. You can read more about index sorting here: http://shaierera.blogspot.com/2013/04/index-sorting-with-lucene.html Shai On Wed, Oct 23, 2013 at 8:13 PM, Arvind Kalyan wrote: > Hi there, I'm looking for pointers, suggesti

Merging ordered segments without re-sorting.

2013-10-23 Thread Arvind Kalyan
Hi there, I'm looking for pointers, suggestions on how to approach this in Lucene 4.5. Say I am creating an index using a sequence of addDocument() calls and end up with segments that each contain documents in a specified ordering. It is guaranteed that there won't be updates/deletes/reads etc hap

Re: sorting with lucene 4.3

2013-07-31 Thread Nicolas Guyot
ok i see. I tried the same test with randomized values on the numeric DV and now the search speed is low and constant. It's not gonna solve our issue since the values are relatively ordered in our case but it's good to know. On Wed, Jul 31, 2013 at 12:05 PM, Yonik Seeley wrote: > On Wed, Jul 3

Re: sorting with lucene 4.3

2013-07-31 Thread Yonik Seeley
On Wed, Jul 31, 2013 at 2:51 PM, Nicolas Guyot wrote: > I have written a quick test to reproduce the slower sorting with numeric DV. > In this test case, it happens only when reverse sorting. Right - I bet your numeric field is relatively ordered in the index. When this happens, there is always o

Re: sorting with lucene 4.3

2013-07-31 Thread Nicolas Guyot
thanks Adrien for the explanation, it's really much appreciated. I have written a quick test to reproduce the slower sorting with numeric DV. In this test case, it happens only when reverse sorting. About the sorting by page i mentioned, it was due to a mistake in our testcase, sorry about that.

Re: sorting with lucene 4.3

2013-07-30 Thread Adrien Grand
Hi, On Tue, Jul 30, 2013 at 8:19 PM, Nicolas Guyot wrote: > When sorting numerically, the search seems to take a bit of a while > compared to the lexically sorted search. > Also when sorting numerically the result is sorted within each page but no > globally as opposed to the lexical sorted searc

RE: Sorting by NumericField not working

2011-03-26 Thread Uwe Schindler
.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: 石玉明 [mailto:shiyuming@gmail.com] > Sent: Saturday, March 26, 2011 2:26 AM > To: java-user@lucene.apache.org > Subject: Re: Sorting by NumericField not working &g

Re: Sorting by NumericField not working

2011-03-25 Thread 石玉明
NumericField include int float double ... and so on. but your sort uses SortField.int . Maybe that is the key point. -- newbie of Lucene 2011-03-26 石玉明 发件人: Azhar Jassal 发送时间: 2011-03-25 22:23:46 收件人: java-user@lucene.apache.org 抄送: 主题: Sorting by NumericField not working Hi,

Re: Sorting by NumericField not working

2011-03-25 Thread Ian Lea
>From the javadoc for SortField: Fields must be indexed in order to sort by them. >From the javadoc for NumericField: public NumericField(String name, int precisionStep, Field.Store store, boolean index) -- Ian. P.S. Read the javadocs!

Re: Sorting by multiple dependent fields

2011-03-23 Thread Ahmet Arslan
> I'm searching for things near your location (as specified > by longitude and latitude).  I've got the search > working correctly (with the help of NumericField), but now I > need to sort the results by distance from you.  The > closer things appear at the top of the list.  There is a contrib pac

Re: Sorting a Lucene index

2010-08-25 Thread Lance Norskog
It is also possible to sort by function. This allows you to avoid storing an array of 1 int for all documents. It is slower than the raw Lucene sort. On Wed, Aug 25, 2010 at 1:46 AM, Toke Eskildsen wrote: > On Wed, 2010-08-25 at 07:16 +0200, Shelly_Singh wrote: >> I have 1 bln documents to sort.

RE: Sorting a Lucene index

2010-08-25 Thread Toke Eskildsen
On Wed, 2010-08-25 at 07:16 +0200, Shelly_Singh wrote: > I have 1 bln documents to sort. So, that would mean ( 8 bln bytes == 8GB RAM) > bytes. > All I have is 8 GB on my machine, so I do not think approach would work. This implies that your numeric value can be more than 2 billion. Are you sure

Re: Sorting a Lucene index

2010-08-25 Thread Ian Lea
19, 2010 7:18 PM > To: java-user@lucene.apache.org > Subject: Re: Sorting a Lucene index > > You haven't yet told us how many documents you're talking about here, so > it's > hard to have a good idea of what solutions are. That said, I'd just try > sorting f

RE: Sorting a Lucene index

2010-08-24 Thread Shelly_Singh
:18 PM To: java-user@lucene.apache.org Subject: Re: Sorting a Lucene index You haven't yet told us how many documents you're talking about here, so it's hard to have a good idea of what solutions are. That said, I'd just try sorting first. The sorting cache size will be something

Re: Sorting a Lucene index

2010-08-19 Thread Erick Erickson
[mailto:ansh...@gmail.com] > Sent: Wednesday, August 18, 2010 5:21 PM > To: java-user@lucene.apache.org > Subject: Re: Sorting a Lucene index > > Hi Shelly, > The search results so returned are sorted either by relevance, index order, > stored field, or custom order. > As yo

Re: Sorting a Lucene index

2010-08-19 Thread findbestopensource
shelly_si...@infosys.com > Phone: (M) 91 992 369 7200, (VoIP)2022978622 > > -Original Message- > From: Anshum [mailto:ansh...@gmail.com] > Sent: Wednesday, August 18, 2010 5:21 PM > To: java-user@lucene.apache.org > Subject: Re: Sorting a Lucene index > > Hi S

RE: Sorting a Lucene index

2010-08-18 Thread Shelly_Singh
: shelly_si...@infosys.com Phone: (M) 91 992 369 7200, (VoIP)2022978622 -Original Message- From: Anshum [mailto:ansh...@gmail.com] Sent: Wednesday, August 18, 2010 5:21 PM To: java-user@lucene.apache.org Subject: Re: Sorting a Lucene index Hi Shelly, The search results so returned are

Re: Sorting a Lucene index

2010-08-18 Thread Anshum
Hi Shelly, The search results so returned are sorted either by relevance, index order, stored field, or custom order. As you are saying that you would not be able to maintain the index order, you would have to do the sort at run time. Sorting on a stored field is not costly and you may use it comf

Re: Sorting and Empty (non-existing) Fields

2010-05-19 Thread Chris Hostetter
: Now I want to search something on the first field and want the results : sorted by relevance, then by the first field, then by the second field. first off: if your primary sort is on relevancy, there are going to be very few cases where your secondary sort comes into play -- the scoring form

Re: Sorting and Empty (non-existing) Fields

2010-05-18 Thread Rob Bygrave
BTW: Saw this in the SOLR docs... - If sortMissingLast="false" and sortMissingFirst="false" (the default), * then default lucene sorting will be used which places docs without the field first in an ascending sort and last in a descending sort.* On Wed, May 19, 2010 at 4

Re: Sorting and Empty (non-existing) Fields

2010-05-18 Thread Rob Bygrave
I'm not a Lucene Guru so hopefully you get a more definitive response. I believe this means you want a way to specify ... "Nulls High" / "Nulls Low" for your field (in this case you want Nulls High I believe). I haven't seen support for that (but it might exist). Looking at StringValComparator I'

Re: Sorting case insensitive wildcard query (with highlight)

2010-03-12 Thread Ian Lea
Can you just lowercase a dedicated sort field and leave the others alone. -- Ian. On Fri, Mar 12, 2010 at 10:47 AM, Kev Kilroy wrote: > > Hi, > > I'm using Lucene 2.4.1 with Hibernate Search 3.1.1. I have objects in the > index, for each field I index as follows: > > @Fields( value = { >    

Re: Sorting case insensitive wildcard queries

2010-02-19 Thread kevinkilroy
Hi Erick, Thanks for your input. I have 2 annotation fields (Sorry, I don't have access to the code at the moment to paste in). but basically, I am sorting on an un-tokenized field. The problem I have is that I am using wildcard queries, & from what I believe they don't get the filters applied

Re: Sorting case insensitive wildcard queries

2010-02-19 Thread Erick Erickson
You can index (but not store) the field you want to sort on in a separate field then sort on that field. How are you sorting anyway? Your message leaves open the possibility that you're sorting on a tokenized field, which is unsupported. Fields you use to sort should be untokenized. HTH Erick On

RE: Sorting issues resolved in 3.0?

2009-12-03 Thread Uwe Schindler
In 3.0 nothing changed about that (and also in 2.9). Only that the FieldCache is now segment-wise which makes IndexReader.reopen be faster. But you are still able to do your own sorting with own structures, you just have to write your own TopDocsCollector. - Uwe Schindler H.-H.-Meier-Allee 63

Re: Sorting field contating NULL values consumes field cache memory

2009-07-21 Thread Shai Erera
FWIW, I had implemented a sort-by-payload feature which performs quite well. It has a very small memory footprint (actually close to 0), and reads values from a payload. Payloads, at least from my experience, perform better than stored fields. On a comparison I've once made, the sort-by-payload fe

Re: Sorting field contating NULL values consumes field cache memory

2009-07-21 Thread Chris Hostetter
: Right now, you can't really do anything about it. In the future, with the : new FieldCache API that may go in, you could plug in a custom implementation : that makes tradeoffs for a sparse array of some kind. The docid is currently : the index into the array, but with a custom impl you may be ab

Re: Sorting field contating NULL values consumes field cache memory

2009-07-20 Thread Ganesh
pointers and it is not part of initial search. Regards Ganesh - Original Message - From: "Mark Miller" To: Sent: Monday, July 20, 2009 10:21 PM Subject: Re: Sorting field contating NULL values consumes field cache memory > Right now, you can't really do anything about

Re: Sorting field contating NULL values consumes field cache memory

2009-07-20 Thread Mark Miller
Right now, you can't really do anything about it. In the future, with the new FieldCache API that may go in, you could plug in a custom implementation that makes tradeoffs for a sparse array of some kind. The docid is currently the index into the array, but with a custom impl you may be able to use

Re: Sorting field contating NULL values consumes field cache memory

2009-07-20 Thread Ganesh
Any ideas on this?? Regards Ganesh - Original Message - From: "Ganesh" To: Sent: Friday, July 17, 2009 2:42 PM Subject: Sorting field contating NULL values consumes field cache memory I am doing sorting on DateTime with minute resolution. I am having 90 million of records and sortin

Re: Sorting fields while searching!

2009-06-01 Thread Erick Erickson
It's really unclear to me what PhysicianFieldInfo.FIRST_NAME_EXACT.toString() returns. I assume the intent is to return a field name, but how that relates to FIRST_NAME_EXACT(Field.Store.YES, Field.Index.UN_TOKENIZED) doesn't mean anything to me. Could you provide some details? Note that if you s

Re: Sorting by relevance and a field

2009-02-13 Thread Michael McCandless
SortField.FIELD_SCORE lets you sort by relevance. So then make a Sort that contains an array of two SortFields, eg: new Sort(new SortField[] {SortField.FIELD_SCORE, new SortField(myField)}) and pass that when searching. Lucene will then sort first by score, and when there are ties, sec

Re: Sorting documents without a query

2008-12-06 Thread Khawaja Shams
Hi Shivaraj, I would recommend John's approach and let Lucene deal with the sorting. Nonetheless, you can traverse through the terms by using the following method: http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/index/IndexReader.html#terms(org.apache.lucene.index.Term) Basically, yo

Re: Sorting documents without a query

2008-12-06 Thread Shivaraj Tenginakai
Thanks John, MatchAllDocsQuery works, however I was unable to locate an example for traversing the term table. Could you please point me to one? Regards, Shivaraj On Fri, Dec 5, 2008 at 4:35 AM, John Wang <[EMAIL PROTECTED]> wrote: > The obvious way is to use use MatchAllDocsQuery with Sort

Re: Sorting documents without a query

2008-12-05 Thread John Wang
The obvious way is to use use MatchAllDocsQuery with Sort parameters on the searcher, e.g. searcher.search(new MatchAllDocsQuery(),sort); If you only care about 1 sort spec (e.g. no secondary sort to break ties) it may be faster just traversing the term table since that is already sorted. -John

Re: Sorting posting lists before intersection

2008-10-13 Thread Renaud Delbru
Hi, Paul Elschot wrote: This could be done, but since not all scorers will be TermScorers it will be necessary to add a method to Scorer (or perhaps even to its DocIdSetIterator superclass): public abstract int estimatedDocFreq(); and implement this for all existing instances. TermScorer co

Re: Sorting posting lists before intersection

2008-10-13 Thread Renaud Delbru
Andrzej Bialecki wrote: Renaud Delbru wrote: Hi Andrzej, sorry for the late reply. I have looked at the code. As far as I understand, you sort the posting lists based on the first doc skip. The first posting list will be the one who have the first biggest document skip. Do the sparseness of

Re: Sorting posting lists before intersection

2008-10-13 Thread Paul Elschot
Op Monday 13 October 2008 17:00:06 schreef Andrzej Bialecki: > Renaud Delbru wrote: > > Hi Andrzej, > > > > sorry for the late reply. > > > > I have looked at the code. As far as I understand, you sort the > > posting lists based on the first doc skip. The first posting list > > will be the one who

Re: Sorting posting lists before intersection

2008-10-13 Thread Andrzej Bialecki
Renaud Delbru wrote: Hi Andrzej, sorry for the late reply. I have looked at the code. As far as I understand, you sort the posting lists based on the first doc skip. The first posting list will be the one who have the first biggest document skip. Do the sparseness of posting lists is a good p

Re: Sorting posting lists before intersection

2008-10-13 Thread Renaud Delbru
Hi Andrzej, sorry for the late reply. I have looked at the code. As far as I understand, you sort the posting lists based on the first doc skip. The first posting list will be the one who have the first biggest document skip. Do the sparseness of posting lists is a good predictor for sampling

Re: Sorting with ParallelReader

2008-09-26 Thread Ivan Vasilev
Sorry about the spam with this thread. We started using ParallelReader in our app and we have some bug in the app with the sorts. I tested with simple standalone app ParallelReader and discovered that sort works in the same way perfectly as with the other Readers. Sorry once again. Best Regards

Re: Sorting posting lists before intersection

2008-09-17 Thread Jason Rutherglen
It would be a good feature in Lucene to be able to sort, or perhaps store the postings in term frequency sorted order. Thoughts? On Wed, Sep 17, 2008 at 9:33 AM, Andrzej Bialecki <[EMAIL PROTECTED]> wrote: > Renaud Delbru wrote: >> >> Hi all, >> >> I am wondering if Lucene implements the query op

Re: Sorting posting lists before intersection

2008-09-17 Thread Andrzej Bialecki
Renaud Delbru wrote: Hi all, I am wondering if Lucene implements the query optimisation that consists of ordering the posting lists based on the term frequency before intersection ? If yes, could somebody point me to the java class / method that implements such strategy ? Lucene trunk: Conj

RE: Sorting in lucene through Document boosting

2008-09-15 Thread Dragan Jotanovic
PM To: java-user@lucene.apache.org Subject: Re: Sorting in lucene through Document boosting 15 sep 2008 kl. 14.08 skrev Dragan Jotanovic: > I made simple Similarity implementation: > public float tf(float arg0) { > return 1f; > } Why do you touch the term f

Re: Sorting in lucene through Document boosting

2008-09-15 Thread Karl Wettin
15 sep 2008 kl. 14.08 skrev Dragan Jotanovic: I made simple Similarity implementation: public float tf(float arg0) { return 1f; } Why do you touch the term frequency? Is that prehaps unrelated to what's discussed in this thread? karl

Re: Sorting

2008-08-05 Thread Erick Erickson
"__AMSUNTOK__" + fieldName. > > Where fieldName was the name of the tokenized field. > > > > > > Bob Hastings > > Ancept Inc. > > > > > > > > > > Mark Miller <[EMAIL PROTECTED]> > > 08/05/2008 02:38 PM > &

Re: Sorting

2008-08-05 Thread Andre Rubin
quot;__AMSUNTOK__" + fieldName. > Where fieldName was the name of the tokenized field. > > > Bob Hastings > Ancept Inc. > > > > > Mark Miller <[EMAIL PROTECTED]> > 08/05/2008 02:38 PM > Please respond to > java-user@lucene.apache.org > > > To &g

Re: Sorting

2008-08-05 Thread Robert . Hastings
he.org To java-user@lucene.apache.org cc Subject Re: Sorting Hey Andre, The reason the javadoc says the field should not be tokenized stems from the issue you point out. What you want to do is possible of course, but making the Lucene code change would complicate a process that can be quit

Re: Sorting

2008-08-05 Thread Mark Miller
Hey Andre, The reason the javadoc says the field should not be tokenized stems from the issue you point out. What you want to do is possible of course, but making the Lucene code change would complicate a process that can be quite memory and cpu intensive on large collections. Done right, it

Re: Sorting case-insensitively

2008-07-18 Thread Chris Hostetter
: > if you could submit a test case that ... : See my e-mail dated July 3, 2008. Sorry: i ment open a bug (in Jira) and submit a JUnit test case. I also ment something even simpler so the lower casing doesn't confuse the issue ie: class IdentitySortComparator extends SortCompar

Re: Sorting case-insensitively

2008-07-14 Thread Paul J. Lucas
On Jul 10, 2008, at 2:24 PM, Chris Hostetter wrote: if you could submit a test case that reproduces this using a trivial subclass (just return the orriginal String as the Comparable) that can help us verify the bug and the fix. See my e-mail dated July 3, 2008. Assuming i'm right, I don'tr

Re: Sorting case-insensitively

2008-07-10 Thread Chris Hostetter
: But how does the built-in STRING sort work with null values then? And how do : I make a SortComparitor that works? Built in string sorting uses FieldCache.DEFAULT.getStringIndex() ... any doc without a value ends up without an assignment in StringIndex.order[], so it gets the default value o

Re: Sorting case-insensitively

2008-07-10 Thread Paul J. Lucas
On Jul 9, 2008, at 10:14 PM, Chris Hostetter wrote: I'm going to guess you have a doc where that field doesn't have a value. ordinarily that's fine, but maybe SortComparator doesn't handle that case very well. But how does the built-in STRING sort work with null values then? And how do I

Re: Sorting case-insensitively

2008-07-09 Thread Chris Hostetter
: But when my code runs, I get a NullPointerException in Lucene's : SortComparator, : line 54 which reads: : : return cachedValues[i.doc].compareTo (cachedValues[j.doc]); : : because cachedCalues[i.doc] is null. But why is it null? I'm going to guess you have a doc where that field doesn

Re: Sorting case-insensitively

2008-07-04 Thread Paul J . Lucas
On Jun 30, 2008, at 8:08 PM, Paul J. Lucas wrote: On Jun 30, 2008, at 8:55 PM, Paul J. Lucas wrote: If I have a SortField with a type of STRING, is there any way to sort in a case-insensitive manner? Couldn't I also use a custom SortComparator? OK, so I tried that by doing: SortFi

Re: Sorting case-insensitively

2008-07-01 Thread Erik Hatcher
On Jun 30, 2008, at 11:08 PM, Paul J. Lucas wrote: On Jun 30, 2008, at 7:00 PM, Erik Hatcher wrote: On Jun 30, 2008, at 8:55 PM, Paul J. Lucas wrote: If I have a SortField with a type of STRING, is there any way to sort in a case-insensitive manner? Only if you unify the case (lower case

Re: Sorting case-insensitively

2008-06-30 Thread Paul J. Lucas
On Jun 30, 2008, at 7:00 PM, Erik Hatcher wrote: On Jun 30, 2008, at 8:55 PM, Paul J. Lucas wrote: If I have a SortField with a type of STRING, is there any way to sort in a case-insensitive manner? Only if you unify the case (lower case everything) on the client side that you send to Solr

Re: Sorting case-insensitively

2008-06-30 Thread Erik Hatcher
On Jun 30, 2008, at 8:55 PM, Paul J. Lucas wrote: If I have a SortField with a type of STRING, is there any way to sort in a case-insensitive manner? Only if you unify the case (lower case everything) on the client side that you send to Solr, but in general no. You can use a text field ty

Re: Sorting issues

2008-06-27 Thread Erick Erickson
PM > Please respond to > java-user@lucene.apache.org > > > To > java-user@lucene.apache.org > cc > > Subject > Re: Sorting issues > > > > > > > I can't really help since I've never had to go into the guts of Lucene and > see where sorting is a

Re: Sorting issues

2008-06-27 Thread Robert . Hastings
.apache.org To java-user@lucene.apache.org cc Subject Re: Sorting issues I can't really help since I've never had to go into the guts of Lucene and see where sorting is applied, so I don't know where to point you . But the sorting has always worked for me, and I don&#

Re: Sorting issues

2008-06-27 Thread Erick Erickson
; "Erick Erickson" <[EMAIL PROTECTED]> > 06/27/2008 12:19 PM > Please respond to > java-user@lucene.apache.org > > > To > java-user@lucene.apache.org > cc > > Subject > Re: Sorting issues > > > > > > > That's surprising. Co

Re: Sorting issues

2008-06-27 Thread Robert . Hastings
respond to java-user@lucene.apache.org To java-user@lucene.apache.org cc Subject Re: Sorting issues That's surprising. Could you post a brief example of your index and search code? It sounds like you're saying docs 1, 2, 3 all have category aaa docs 4, 5, 6 all have category bbb docs 7,

Re: Sorting issues

2008-06-27 Thread Erick Erickson
That's surprising. Could you post a brief example of your index and search code? It sounds like you're saying docs 1, 2, 3 all have category aaa docs 4, 5, 6 all have category bbb docs 7, 8, 9 all have category ccc But if you search for category:bbb you don't get docs 4, 5, and 6 Is this a fair

Re: Sorting consumes hundreds of MBytes RAM

2008-04-26 Thread Eran Sevi
PROTECTED]> wrote: > On Mon, Apr 14, 2008, Chris Hostetter wrote about "Re: Sorting consumes > hundreds of MBytes RAM": > > : And question #2: what am I going to do against it? Index sharding? > > > > The only suggestion i can offer is to take a look at LUCENE-

Re: Sorting consumes hundreds of MBytes RAM

2008-04-25 Thread Nadav Har'El
On Mon, Apr 14, 2008, Chris Hostetter wrote about "Re: Sorting consumes hundreds of MBytes RAM": > : And question #2: what am I going to do against it? Index sharding? > > The only suggestion i can offer is to take a look at LUCENE-769 ... it > takes a completley differn

Re: Sorting consumes hundreds of MBytes RAM

2008-04-17 Thread Otis Gospodnetic
Timo - correct and correct. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Timo Nentwig <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Tuesday, April 15, 2008 3:22:15 AM Subject: Re: Sorting consumes hundreds of MBytes RAM W

Re: Sorting consumes hundreds of MBytes RAM

2008-04-15 Thread Timo Nentwig
odnetic <[EMAIL PROTECTED]> Reply-To: java-user@lucene.apache.org To: java-user@lucene.apache.org Subject: Re: Sorting consumes hundreds of MBytes RAM Timo, That is true. The only think I can recommend at the moment is to make sure you specify the correct data type. If your sort field is a nume

Re: Sorting consumes hundreds of MBytes RAM

2008-04-14 Thread Chris Hostetter
: How does this work internally? It seems as if all data for this field found in : the entire index is read into memory (?). You can think of it as an "inverted-inverted index" Lucene needs a data structure it can usefor fast lookups where the key is the docId and the value is something "com

Re: Sorting consumes hundreds of MBytes RAM

2008-04-13 Thread Otis Gospodnetic
Timo, That is true. The only think I can recommend at the moment is to make sure you specify the correct data type. If your sort field is a numeric field, make that explicit. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Timo Nentwig <[EMA

Re: Sorting VS Scoring

2008-04-02 Thread Erick Erickson
; From: Erick Erickson [mailto:[EMAIL PROTECTED] > Sent: Wednesday, April 02, 2008 11:12 AM > To: java-user@lucene.apache.org > Subject: Re: Sorting VS Scoring > > The problem here is that you'll have to keep deleting and > adding your documents in order to update the counter field

RE: Sorting VS Scoring

2008-04-02 Thread John Xiao
apache.org Subject: Re: Sorting VS Scoring The problem here is that you'll have to keep deleting and adding your documents in order to update the counter field for all of these solutions, and I doubt that's what you really want to do. There is much discussion of updating a document that&#x

Re: Sorting VS Scoring

2008-04-02 Thread Erick Erickson
The problem here is that you'll have to keep deleting and adding your documents in order to update the counter field for all of these solutions, and I doubt that's what you really want to do. There is much discussion of updating a document that's already in the index, but I don't think it's there y

Re: sorting a doc field takes more time

2008-03-17 Thread Grant Ingersoll
Sorting is dependent on the values in the fields. What is actually in the fields? But, yes, in general, sorting is going to be slower than just raw search. It's extra operations. It also looks like you are using the AUTO SortField, which means you are relying on Lucene to figure out how

Re: sorting a doc field takes more time

2008-03-14 Thread sandyg
HI, thnx for reply field ,documents ,sort and sort field all are lucene classes and after getting the results at the time of displaying am using sort class to sort the reults based on particular field the code for sorting Query query = parser.parse(queryString); Sort

Re: sorting a doc field takes more time

2008-03-13 Thread Grant Ingersoll
What's in "field"? What are your docs? More info is needed to help... -Grant On Mar 13, 2008, at 6:50 AM, sandyg wrote: Hi, Thnxs for spending time for the problem. When sorting the results of lucene search it takes more time and not looks not that much usefull can any one help Below i

Re: Sorting by multiple fields (conditions influencing each other?!)

2008-01-13 Thread Otis Gospodnetic
As far as I know, the secondary sort really kicks in only when there is a tie caused by the primary sort, so the secondary sort should not be affecting the primary sort. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Tobias Lohr <[EMAIL PROT

Re: Sorting on tokenized fields

2008-01-08 Thread Michael Prichard
yes, no worries. i just check in advance what fields are available and build the Sort object accordingly. Eventually BCC would be there...but not necessary so at first. Anyway, got it to work! Thanks for your help. All the best, Michael On Jan 8, 2008, at 4:37 PM, Doron Cohen wrote: H

Re: Sorting on tokenized fields

2008-01-08 Thread Doron Cohen
Hi Michael, I think you mean the exception thrown when you search and sort with a field that was not yet indexed: RuntimeException: field "BBC" does not appear to be indexed I think the current behavior is correct, otherwise an application might (by a bug) attempt to sort by a wrong field, th

Re: Sorting on tokenized fields

2008-01-08 Thread Ryan McKinley
my mistake, I thought I was looking at the solr mailing list ;) If you change your analyzer, it does not change the tokens that are already in the index -- you will need to re-index for any changes to take effect. ryan Michael Prichard wrote: Meaning that it says "field is not indexed". Wh

Re: Sorting on tokenized fields

2008-01-08 Thread Michael Prichard
Meaning that it says "field is not indexed". Where is sortMissingLastAttribute? thanks. On Jan 8, 2008, at 4:13 PM, Ryan McKinley wrote: what do you mean by "fail"? -- there is the sortMissingLast attribute Michael Prichard wrote: ok... i should read the manual more often. i went ahead a

Re: Sorting on tokenized fields

2008-01-08 Thread Ryan McKinley
what do you mean by "fail"? -- there is the sortMissingLast attribute Michael Prichard wrote: ok... i should read the manual more often. i went ahead and just added untokenized, unstored sort fields question, if I put a field in to sort of but say I have not indexed any as of yet...will

Re: Sorting on tokenized fields

2008-01-08 Thread Michael Prichard
ok... i should read the manual more often. i went ahead and just added untokenized, unstored sort fields question, if I put a field in to sort of but say I have not indexed any as of yet...will the Sort fail? For example, say I have a BCC field and nothing has been indexed with that yet

  1   2   3   >