RE: updateDocument (somtimes) no longer deleting documents after Update to 4.6

2014-02-24 Thread Uwe Schindler
Hi, it looks like your filters are implemented in a wrong way: - First, in Lucene 3 and 4, filters are applied by segment. Means, they have to calculate the DocIdSet of matched documents for each index segment separately. On updating, the document is "deleted" (hidden) on the old segment and re

Re: updateDocument (somtimes) no longer deleting documents after Update to 4.6

2014-02-24 Thread Erick Erickson
I suspect you're finding the old doc that is simply marked as deleted. Did you check for that? One quick way to see if this is even in the right ballpark would be to do a forceMerge. If the problem disappears, then this is relevant I'd guess. Warning: The operative word here is "guess", I haven't

Re: updateDocument (somtimes) no longer deleting documents after Update to 4.6

2014-02-24 Thread nospam
Hm it looks like this is somehow caused by the filters we are using for searching. I took one of the MY_UNIQUE_BUSINESS_ID ids, used in our applications search functionality and debuged the lucene search a little more. If I specify null for the filters I only get one result (which is correct).

Re: Single segment merge in lucene possible?

2014-02-24 Thread Ravikumar Govindarajan
Thanks Uwe. Will try your suggestion -- Ravi On Sat, Feb 22, 2014 at 4:20 AM, Uwe Schindler wrote: > Yeah, as mentioned, UpgradeIndexMergePolicy is easiest to customize to do > that: You only have to override shouldUpgradeSegment() and return true for > the segment to be merged, false otherwis

Re: updateDocument (somtimes) no longer deleting documents after Update to 4.6

2014-02-24 Thread nospam
I'll see if I can dig a little bit deeper into the 3.6 behavior, for now I'm trying to get it running on 4.6 (as the index file is also a lot smaller - on 3.6 it was about 2 GB for about 9000 documents, with 4.6 it's only about 200 MB). And yes the business ID is indexed - otherwhise I would

Re: updateDocument (somtimes) no longer deleting documents after Update to 4.6

2014-02-24 Thread Michael McCandless
The 30 second turnaround time in 3.6.x is absurd; if you turn on IndexWriter's infoStream maybe it'd give a clue. Or, capture a few stack traces and post them. How are you creating the luceneDocumentToIndex? You must ensure that the business ID is in fact indexed as a field in the document, othe

updateDocument (somtimes) no longer deleting documents after Update to 4.6

2014-02-24 Thread nospam
Hi there, we recently updated our application from lucene 3.0 to 3.6 with the effect that (albeit using the SearchManager functionality as described on http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html) calls to searcherManager.maybeRefresh() were incredibly slow.