RE: TopDocCollector vs TopScoreDocCollector (semantics changed in 4.0, not backward comptabile)

2013-01-23 Thread saisantoshi
Here is the way I implemented a collector class. Appreciate if you could let me know of any issues.. public class MyCollector extends PositiveScoresOnlyCollector { private IndexReader indexReader; public MyCollector (IndexReader indexReader,PositiveScoresOnlyCollector topScor

Re: Extending TopScoreDocCollector to write a custom collector

2013-01-23 Thread saisantoshi
Here is the way I implemented a collector class. Appreciate if you could let me know of any issues.. public class MyCollector extends PositiveScoresOnlyCollector { private IndexReader indexReader; public MyCollector (IndexReader indexReader,PositiveScoresOnlyCollector topScor

Extending TopScoreDocCollector to write a custom collector

2013-01-23 Thread saisantoshi
I would like to write a custom collector ( similar to the one which is inside the source of TopScoreDocCollector like InOrderTopScoreDocCollector). The reason for extending this is because InOrderTopScoreDocCollector and OutOfOrderTopScoreDocCollector are private to the class and I really wanted

Re: IndexWriter.optimize() is removed in 4.0?

2013-01-23 Thread Denis Bazhenov
All the given reasons applies to some systems but not to all of them. But distributed systems (with index being built on dedicated machine) with index warm up on index reopen can successfully mitigate both IO and cold-cache overhead. On Jan 24, 2013, at 8:58 AM, Steve Rowe wrote: > Hi Sai, >

RE: TopDocCollector vs TopScoreDocCollector (semantics changed in 4.0, not backward comptabile)

2013-01-23 Thread saisantoshi
I am sorry but I am confused looking at the change logs and the enhancements done. Since we are jumping from 2.4 - 4.0. Could you please point me to any example code that extends one of the new collectors.. that would help a lot or it would be great if you could give some pointers on how we can mo

RE: TopDocCollector vs TopScoreDocCollector (semantics changed in 4.0, not backward comptabile)

2013-01-23 Thread Uwe Schindler
This has been changed in Lucene 2.9, its nothing new in Lucene 4.0. Read the changes logs of Lucene 2.9/3.0, there is explained what you need to do. Uwe - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From:

TopDocCollector vs TopScoreDocCollector (semantics changed in 4.0, not backward comptabile)

2013-01-23 Thread saisantoshi
Our current search implementation (based on 2.4.0) uses a collector extending the TopDocCollector class public class MyHitCollector extends TopDocsCollector { private IndexReader indexReader; private CustomFilter customFilter; public MyHitCollector (IndexReader indexReader, int numbe

Re: IndexWriter.optimize() is removed in 4.0?

2013-01-23 Thread Steve Rowe
Hi Sai, Check out Simon Willnauer's blog post about this: Steve On Jan 23, 2013, at 4:49 PM, saisantoshi wrote: > There is no optimize() method in 4.0. I looked at the 3.6 docs and it did > mention the followin

IndexWriter.optimize() is removed in 4.0?

2013-01-23 Thread saisantoshi
There is no optimize() method in 4.0. I looked at the 3.6 docs and it did mention the following below. Does the following below mean that we no longer need this method and should not be used anymore. Is there any supplement method that we need to use as it is deprecated as of version 3.6.0 /* Depr

RE: Are Search Index directories backward comptabile? ( when upgrading to latest lucene version)

2013-01-23 Thread saisantoshi
Thanks. We decided to delete the existing index directories and recreate it once we upgrade to 4.0 (unless we hit any major api blockers during compilation, we will prefer to go to 3.6.2 first and then later to 4.0). Thanks, Sai. -- View this message in context: http://lucene.472066.n3.nabble.

RE: Are Search Index directories backward comptabile? ( when upgrading to latest lucene version)

2013-01-23 Thread Uwe Schindler
No, there are only those 2 options. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: saisantoshi [mailto:saisantosh...@gmail.com] > Sent: Wednesday, January 23, 2013 8:11 AM > To: java-user@lucene.apache.

Re: NullPointer Exception in MemoryIndex.MemoryIndexReader

2013-01-23 Thread Michael McCandless
This is not expected: the method should return null if the field isn't in the FieldInfos. Can you open an issue? Thanks! Mike McCandless http://blog.mikemccandless.com On Wed, Jan 23, 2013 at 11:27 AM, Ricky Pritchett wrote: > Greetings, > > Version: Lucene 4.1 > > I'm currently a user of the

Re: Are Search Index directories backward comptabile? ( when upgrading to latest lucene version)

2013-01-23 Thread Ian Lea
Lucene 4.x cannot read indexes created with 2.x. You can change and recompile your code to 4.x in one go. Since you can reindex, I'd make all the code changes and then recreate the indexes using 4.x. With a bit of testing along the way of course. -- Ian. On Wed, Jan 23, 2013 at 7:34 AM, sais

Re: MMapDirectory performance - Are searchable field values contiguously stored in FS block?

2013-01-23 Thread Michael McCandless
Are the additional rarely used 48 fields used for searching? Or, for looking up stored fields? If it's for searching then you should see good locality (efficient use of the OS's IO cache) from the posting lists: each field's postings are stored in a single chunk of the files, then the next field'

MMapDirectory performance - Are searchable field values contiguously stored in FS block?

2013-01-23 Thread Gili Nachum
Hi, I have a search workload that focuses on two fields in my 1GB index. I get very good performance when loaded the index via MmapDirectory. I attribute this performance to the Operating System File System (FS OS) cache, that keeps the most recently used FS blocks RAM resident. I would like to

Re: FacetedSearch and MultiReader

2013-01-23 Thread Nicola Buso
Thanks Shai, I'm trying your solution and it's working, I need to check some number to test it. As I said we are aware having big indexes, than I use facets only on subsets, but if it will result in performances issues too than I'll for sure take a look into facet sampling. Nicola. On Wed, 2013

Re: FacetedSearch and MultiReader

2013-01-23 Thread Shai Erera
> > I think we should open an issue to provide support for distributed > faceting? > Opened https://issues.apache.org/jira/browse/LUCENE-4710. BTW Nicola, I remember you said something about TBs of indexes. I just wanted to point out that if you have really large indexes, with many documents, the