Lucene 4.0 Payloads

2011-03-17 Thread Alex vB
Hello everybody, I am currently experimenting with Lucene 4.0 and would like to add payloads. Payload should only be added once per term on the first position. My current code looks like this: public final boolean incrementToken() throws java.io.IOException { String term = characterAttr.toStri

Re: Understanding the IndexWriter-Infostream log

2011-03-17 Thread Michael McCandless
Hi Tom, Answers below... On Thu, Mar 17, 2011 at 1:19 PM, Burton-West, Tom wrote: > Hello all, > > We have very large documents with large numbers of unique terms.   Our > documents average about 800,000 KB and about 200,000 tokens.  In trying to > understand how often the ramBuffer gets flush

RE: RE: ParallelMultisearcher

2011-03-17 Thread Uwe Schindler
Hi Ganesh, this method is also in 2.9.1, it is just inherited from the superclass! You have to also look at the complete javadocs. Not every method that needs to be called must be already implemented in the used class. For PMS in 2.9 it's just not fatal to not close it, but you should call it: htt

Re: RE: ParallelMultisearcher

2011-03-17 Thread Ganesh
There is no ParallelMultiSearcher.close() in 2.9.1 and its been added in 3.0. Thanks for pointing out. I am not aware of. Regards Ganesh - Original Message - From: "Uwe Schindler" To: Sent: Thursday, March 17, 2011 11:27 PM Subject: [Bulk] RE: ParallelMultisearcher > Hi Ganesh, It's

RE: lucene

2011-03-17 Thread Uwe Schindler
Hi, Why not use PerFieldAnalyzerWrapper to provide the same thing and that's already available? - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Rafał Lenarczyk [mailto:rafal.lenarczyk...@gmail.com] > Se

RE: ParallelMultisearcher

2011-03-17 Thread Uwe Schindler
Hi Ganesh, It's definitely in 3.0.3: @Override public void close() throws IOException { executor.shutdown(); super.close(); } This code is in the downloaded .src.zip file of Apache Lucene 3.0.3. Are you sure that you call close() on the ParallelMultiSearcher (e.g. when you are done

lucene

2011-03-17 Thread Rafał Lenarczyk
Hy, This mail should be written by developers. I'm java developer and use your product in my application. I use special QueryParser like MultiFieldQueryParser I use static method parse where I must set lucene Version, string tab with queries, string tab with field, string tab with flags and analyz

Re: ParallelMultisearcher

2011-03-17 Thread Ganesh
I agree but the chang logs of 3.0.3 didn't mention about that. I tested and found that the same problem exists with 3.0.3. I am bit confused. http://lucene.apache.org/java/3_0_3/changes/Changes.html This is show stopper for us as few customers facing big problems. The application is crashing fre

RE: ParallelMultisearcher

2011-03-17 Thread Uwe Schindler
Yes, it's fixed in SVN since June. Lucene 3.0.3 should contain the fix. When you call ParallelMultiSearcher#close, the Executor is shutdown. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Devon H. O'Del

Understanding the IndexWriter-Infostream log

2011-03-17 Thread Burton-West, Tom
Hello all, We have very large documents with large numbers of unique terms. Our documents average about 800,000 KB and about 200,000 tokens. In trying to understand how often the ramBuffer gets flushed to disk we turned on the IndexWriter log. true With the Solr default setting of ramBuffer

Re: ParallelMultisearcher

2011-03-17 Thread Devon H. O'Dell
2011/3/17 Ganesh : > Is this bug https://issues.apache.org/jira/browse/LUCENE-2249 got fixed in > 3.0.3? The linked ticket shows that it was fixed in 3.0.3. --dho > Regards > Ganesh > > - Original Message - > From: "Ganesh" > To: > Sent: Thursday, March 17, 2011 7:03 PM > Subject: Re:

Re: ParallelMultisearcher

2011-03-17 Thread Ganesh
Is this bug https://issues.apache.org/jira/browse/LUCENE-2249 got fixed in 3.0.3? Regards Ganesh - Original Message - From: "Ganesh" To: Sent: Thursday, March 17, 2011 7:03 PM Subject: Re: ParallelMultisearcher I tested the same with Lucene 2.9.1 and found very less thread count b

Re: FieldSelector with Lucene 2.3.2

2011-03-17 Thread Chris Bamford
Thanks David, I realised my mistake after I sent the original mail. Kind regards - Chris -Original Message- From: David Causse To: java-user@lucene.apache.org Sent: Thu, 17 Mar 2011 11:22 Subject: Re: FieldSelector with Lucene 2.3.2 On Wed, Mar 16, 2011 at 03:17:41PM +,

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread shrinath.m
On Thu, Mar 17, 2011 at 7:42 PM, Ian Lea [via Lucene] < ml-node+2693597-476158808-376...@n3.nabble.com> wrote: > if you go the MultiFieldQueryParser route you are going to need some > arrays. > Which other route can I take to do a all field search ? (Keeping in mind that I don't know what fields

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread Ian Lea
if you go the MultiFieldQueryParser route you are going to need some arrays. You specifically asked if you needed an array of BooleanClause.Occur and the answer to that specific question is still no. I can't tell you any more on this than you can get from reading the javadocs. -- Ian. On Thu,

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread shrinath.m
On Thu, Mar 17, 2011 at 7:07 PM, Ian Lea [via Lucene] < ml-node+2693447-1032232993-376...@n3.nabble.com> wrote: > static Query parse(Version matchVersion, String[] queries, String[] > fields, Analyzer analyzer) > Parses a query which searches on the fields specified. > Well, at the end

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread Ian Lea
I was referring to the MultiFieldQueryParser.parse method that doesn't require an array of BooleanClause.Occur flags. static Queryparse(Version matchVersion, String[] queries, String[] fields, Analyzer analyzer) Parses a query which searches on the fields specified. Good to hear yo

Re: ParallelMultisearcher

2011-03-17 Thread Ganesh
I tested the same with Lucene 2.9.1 and found very less thread count but with 3.0.2 it is very high. Do i need to revert back to Lucene 2.9.1. Regards Ganesh - Original Message - From: "Ganesh" To: Sent: Thursday, March 17, 2011 5:17 PM Subject: ParallelMultisearcher Hello all,

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread shrinath.m
On Thu, Mar 17, 2011 at 5:40 PM, Ian Lea [via Lucene] < ml-node+2693169-1481141743-376...@n3.nabble.com> wrote: > OK, yes, if you do need to "specify certain fields as required, and > others as prohibited" then you'll need to build and populate that > array. There is also a parse method that does

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread Ian Lea
OK, yes, if you do need to "specify certain fields as required, and others as prohibited" then you'll need to build and populate that array. There is also a parse method that doesn't require the array if you just want to search all the fields. I hope that the 2_3_1 links don't indicate that you a

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread shrinath.m
On Thu, Mar 17, 2011 at 4:48 PM, Ian Lea [via Lucene] < ml-node+2693036-1270863785-376...@n3.nabble.com> wrote: > > I'd have to create a similarly sized array of Boolean.Occur too, isn't > it? > > What for? http://lucene.apache.org/java/2_3_1/api/org/apache/lucene/queryParser/MultiFieldQueryPars

ParallelMultisearcher

2011-03-17 Thread Ganesh
Hello all, Is there any issue with ParallelMultiSearcher in Lucene 3.0.2? If we search more frequently then OutOfMemoryError is triggered or It is throwing "Not able to create native thread".. It is working with 2.9.1. I generated Heap Dump and found lot of entries for ParallelMultiSearcher.

Re: FieldSelector with Lucene 2.3.2

2011-03-17 Thread David Causse
On Wed, Mar 16, 2011 at 03:17:41PM +, Chris Bamford wrote: > Hi > > I am trying to speed up access to the data in my results Documents > and was wondering if FieldSelector might be the way forward? > After my search, I end up with an ArrayList of Documents, from each > of which I need to extra

Re: Re: How to do an all field search without using a "catchall" fieldor "MultiFieldQueryParser" ?

2011-03-17 Thread Ian Lea
> I'd have to create a similarly sized array of Boolean.Occur too, isn't it? What for? > 1. Is that how SIREn implements it? No idea. > 2. Is that optimal solution if I'm going to have an index of a billion docs > with varying number of fields? Probably not. I always use a catchall field. I

RE: About highlighter

2011-03-17 Thread Pierre GOSSE
500 is the max size of text fragments to be returned by highlight. It shouldn't be the problem here, as far as I understand highlight. Gong li, how is defined the field "contents" ? Is it the only field on which the search is made ? Pierre -Message d'origine- De : Ian Lea [mailto:ian..