Re: Newbie Question - Lucene Sorting NOT Ignoring NULL values

2008-10-22 Thread Reetha Hariharan
Hi Yonik: A simple RangeFilter did the trick > public class NotNullRangeFilter extends org.apache.lucene.search.Filter { >private RangeFilter rangeFilter; >public BitSet bits(IndexReader reader) throws IOException { >rangeFilter = RangeFilter.More("lastUpdatedDate"/*Field to sort*

Re: Newbie Question - Lucene Sorting NOT Ignoring NULL values

2008-10-14 Thread Yonik Seeley
You're on the right track I think... perhaps try using RangeFilter directly rather than creating your own class. Something like: Filter filter = RangeFilter.More("lastUpdatedDate",""); searcher.search(query, filter) If that works for you, then the next step would be to look at CachingWrapperFilt

Re: Newbie Question - Lucene Sorting NOT Ignoring NULL values

2008-10-14 Thread Reetha Hariharan
Hi Yonik, Thanks for your reply. In my case I don't want those document that has Null value for the field that I am willing to sort I tried writing my own filter using RangeFilter, but it doesn't work. I used something like the following in my custom filter. public class NotNullRangeFilter ext

Re: Newbie Question - Lucene Sorting NOT Ignoring NULL values

2008-10-14 Thread Yonik Seeley
On Tue, Oct 14, 2008 at 4:35 AM, Reetha Hariharan <[EMAIL PROTECTED]> wrote: > I am searching using one field, say X and want to sort the results using > another, say Y (Which can have null values). But I am expecting Sort to > ignore all the null values and just sort only records that has values i

Newbie Question - Lucene Sorting NOT Ignoring NULL values

2008-10-14 Thread Reetha Hariharan
Hi, I am a newbie. I just configured lucene using hibernate search. But I find that the sorting doesn't ignore null values. I am searching using one field, say X and want to sort the results using another, say Y (Which can have null values). But I am expecting Sort to ignore all the null values

Re: Lucene sorting case-sensitive by default?

2008-01-15 Thread Antony Bowesman
Erick Erickson wrote: doc.add( new Field( "f", "This is Some Mixed, case Junk($*%& With Ugly SYmbols", Field.Store.YES, Field.Index.TOKENIZED)); pr

RE: Lucene sorting case-sensitive by default?

2008-01-15 Thread Toke Eskildsen
On Mon, 2008-01-14 at 10:58 -0500, Alex Wang wrote: > Toke, you mentioned "Using a Collator works but does take a fair amount > of memory", can you please elaborate a little more on that. Thanks. We have an index with 10 million records that takes up 37GB. Practically all records have a title, whi

RE: Lucene sorting case-sensitive by default?

2008-01-14 Thread Alex Wang
No problem Erick. Thanks for clarifying it. Alex -Original Message- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Monday, January 14, 2008 12:35 PM To: java-user@lucene.apache.org Subject: Re: Lucene sorting case-sensitive by default? Sorry, I was confused about this for the

Re: Lucene sorting case-sensitive by default?

2008-01-14 Thread Erick Erickson
se, or take any action based on > this message or any information herein. If you have received this > message in error, please advise the sender immediately by reply e-mail > and delete this message. Thank you for your cooperation. > > > -Original Message- > From: Erick E

RE: Lucene sorting case-sensitive by default?

2008-01-14 Thread Alex Wang
you for your cooperation. -Original Message- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Monday, January 14, 2008 11:24 AM To: java-user@lucene.apache.org Subject: Re: Lucene sorting case-sensitive by default? Several things: 1> do you need to display all the fields? Wo

Re: Lucene sorting case-sensitive by default?

2008-01-14 Thread Erick Erickson
ate a little more on that. Thanks. > > Alex > > -Original Message- > From: Toke Eskildsen [mailto:[EMAIL PROTECTED] > Sent: Monday, January 14, 2008 3:13 AM > To: java-user@lucene.apache.org > Subject: Re: Lucene sorting case-sensitive by default? > > On Fri, 2008

RE: Lucene sorting case-sensitive by default?

2008-01-14 Thread Alex Wang
ache.org Subject: Re: Lucene sorting case-sensitive by default? On Fri, 2008-01-11 at 11:40 -0500, Alex Wang wrote: > Looks like Lucene is separating upper case and lower case while sorting. As Tom points out, default sorting uses natural order. It's worth noting that this implies that defa

Re: Lucene sorting case-sensitive by default?

2008-01-14 Thread Toke Eskildsen
On Fri, 2008-01-11 at 11:40 -0500, Alex Wang wrote: > Looks like Lucene is separating upper case and lower case while sorting. As Tom points out, default sorting uses natural order. It's worth noting that this implies that default sorting does not produce usable results as soon as you use non-ASCI

Re: Lucene sorting case-sensitive by default?

2008-01-11 Thread Erick Erickson
I've often stored a special sort field that's lower-cased. On Jan 11, 2008 11:40 AM, Alex Wang <[EMAIL PROTECTED]> wrote: > Hi All, > > > > I was searching my index with sorting on a field called "Label" which is > not tokenized, here is what came back: > > > > Extended Sites Catalog Asset Store

Re: Lucene sorting case-sensitive by default?

2008-01-11 Thread Tom Emerson
String fields are sorted using natural (lexicographic) order. For characters in ASCII range this means uppercase letters will sort before lowercase letters (e.g., 'A' U+0041 sorts before 'a' U+0061). This behaviour is documented on in the JavaDocs for org.apache.lucene.search.Sort. -tree On

Lucene sorting case-sensitive by default?

2008-01-11 Thread Alex Wang
Hi All, I was searching my index with sorting on a field called "Label" which is not tokenized, here is what came back: Extended Sites Catalog Asset Store Extended Sites Catalog Asset Store SALES Print Catalog 2 Print catalog test Test Print Catalog Test refresh catalog print test 3

RE: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Doron Cohen
ed the sortable > fields as 'stored' then the process above would preserve those > fields and hence sorting would proceed as normal. > > Does that make sense? > > Thanks > > Alan > > -Original Message- > From: Erick Erickson [mailto:[EMAIL PROTECTED

RE: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Alan Boshier
ould preserve those fields and hence sorting would proceed as normal. Does that make sense? Thanks Alan -Original Message- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Thu 9/14/2006 18:00 To: java-user@lucene.apache.org Cc: Subject: Re: Newbie question: lucene sort

Re: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Yonik Seeley
On 9/14/06, Alan Boshier <[EMAIL PROTECTED]> wrote: That was my understanding (that they had to be indexed) but making them stored seems to have fixed the problem we were seeing, which is odd. Not being an expert on how lucene works internally, I'm struggling to see how this change could have ma

Re: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Erick Erickson
Original Message- From: Mordo, Aviran (EXP N-NANNATEK) [mailto:[EMAIL PROTECTED] Sent: Thu 9/14/2006 17:18 To: java-user@lucene.apache.org Cc: Subject:RE: Newbie question: lucene sorting problems and stored fields AFIK, the field has to be indexed, but I don't think it

Re: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Yonik Seeley
On 9/14/06, Alan Boshier <[EMAIL PROTECTED]> wrote: Is it a requirement when creating a field for sorting to make it stored? No, stored doesn't matter... it must be indexed though. -Yonik http://incubator.apache.org/solr Solr, the open-source Lucene search server -

RE: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Alan Boshier
ssage- From: Mordo, Aviran (EXP N-NANNATEK) [mailto:[EMAIL PROTECTED] Sent: Thu 9/14/2006 17:18 To: java-user@lucene.apache.org Cc: Subject:RE: Newbie question: lucene sorting problems and stored fields AFIK, the field has to be indexed, but I don't think it has to be stored

RE: Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Mordo, Aviran (EXP N-NANNATEK)
apache.org Subject: Newbie question: lucene sorting problems and stored fields Hi We are seeing intermittent problems with searches that use sorted fields (in lucene 1.4.3). If we add the fields to our Documents as 'unstored' then we start to see results that have been sorted by Document I

Newbie question: lucene sorting problems and stored fields

2006-09-14 Thread Alan Boshier
Hi We are seeing intermittent problems with searches that use sorted fields (in lucene 1.4.3). If we add the fields to our Documents as 'unstored' then we start to see results that have been sorted by Document ID. The problem goes away if we add the fields as 'stored'. Is it a requirement when

Re: lucene sorting

2006-04-05 Thread Chris Hostetter
I don't know if there is anyway for a Custom Sort to access the lucene score -- but another approach that works very well is to use the FunctionQuery classes from Solr... http://incubator.apache.org/solr/docs/api/org/apache/solr/search/function/package-summary.html ...you can make a FunctionQuer

lucene sorting

2006-04-05 Thread Gian Marco Tagliani
Hi, I need to change the lucene sorting to give just a bit more relevance to the recent documents (but i don't want to sort by date). I'd like to mix the lucene score with the date of the document. I'm following the example in "Lucene in Action", chapter 6. I