Re: background merge hit exception

2014-04-09 Thread Jason Wee
Hi Jose, Thank you for very informative response. I have commented out the line of codes that does the forceMerge(50) and commit() while the indexing is happening. Also increase the ram buffer size iwc.setRAMBufferSizeMB(512.0); and after index is done, then only forceMerge and commit but this

Re: background merge hit exception

2014-04-08 Thread Jose Carlos Canova
Hi Jason, No, the StrackTrace shows clearly the cause of the errror occurred during the merge into a single index file segment(forgeMerge parameter defines the number of desired segments at end). During the indexing of a document, Lucene might decide to create a new segment of the information ext

Re: background merge hit exception

2014-04-08 Thread Jason Wee
Hello Jose, Thank you for your response, I took a closer look. Below are my responses: > Seems that you want to force a max number of segments to 1, // you're done adding documents to it): // writer.forceMerge(1); writer.close(); Yes, the line of code is uncommented be

Re: background merge hit exception

2014-04-05 Thread Jose Carlos Canova
Seems that you want to force a max number of segments to 1, On a previous thread someone answered that the number of segments will affect the Index Size, and is not related with Index Integrity (like size of index may vary according with number of segments). on version 4.6 there is a small issue o

Re: background merge hit exception

2011-09-12 Thread Koji Sekiguchi
I've got some follow-up from the user. > Is it possible disk filled up? Though I'd expect an IOE during write > or close in that case. > > In this case nothing should be lost in the index: the merge simply > refused to commit itself, since it detected something went wrong. But > I believe we al

Re: background merge hit exception

2011-09-10 Thread Simon Willnauer
On Sat, Sep 10, 2011 at 12:58 PM, Michael McCandless wrote: > On Fri, Sep 9, 2011 at 6:50 PM, Koji Sekiguchi wrote: Also: what java version are they running? We added this check originally as a workaround for a JRE bug... but usually when that bug strikes the file size is very clos

Re: background merge hit exception

2011-09-10 Thread Michael McCandless
On Fri, Sep 9, 2011 at 6:50 PM, Koji Sekiguchi wrote: >>> Also: what java version are they running? We added this check >>> originally as a workaround for a JRE bug... but usually when that bug >>> strikes the file size is very close (like off by just 1 byte or 8 >>> bytes or something). >> >> I t

Re: background merge hit exception

2011-09-09 Thread Koji Sekiguchi
Also: what java version are they running? We added this check originally as a workaround for a JRE bug... but usually when that bug strikes the file size is very close (like off by just 1 byte or 8 bytes or something). I think they are using 1.6, but I should ask the minor number. Could you show

Re: background merge hit exception

2011-09-09 Thread Koji Sekiguchi
Hi Mike, This wouldn't be caused by the "NFS happily deletes open files" problem (= Stale NFS file handle error). But this could in theory be caused by the NFS client somehow being wrong about the file's metadata (file length). It's sort of odd because I would expect since the client wrote the

Re: background merge hit exception

2011-09-09 Thread Michael McCandless
Interesting... This wouldn't be caused by the "NFS happily deletes open files" problem (= Stale NFS file handle error). But this could in theory be caused by the NFS client somehow being wrong about the file's metadata (file length). It's sort of odd because I would expect since the client wrote

Re: background merge hit exception

2009-02-26 Thread Michael McCandless
Also: what sorts of stored fields are you storing? Binary? Compressed? Text with unicode characters? Roughly how many stored fields per document? Mike vivek sar wrote: Hi, We ran into the same issue (corrupted index) using Lucene 2.4.0. There was no outage or system reboot - not su

Re: background merge hit exception

2009-02-26 Thread Michael McCandless
The exception you hit during optimize indicates some corruption in the stored fields file (_X.fdt). Then, the exception hit during CheckIndex was different -- the postings entry (_r0y.frq) for term "desthost:wir" was supposed to have 6 entries but had 0. Did "CheckIndex -fix" allow you to optim

Re: background merge hit exception

2009-02-25 Thread vivek sar
Hi, We ran into the same issue (corrupted index) using Lucene 2.4.0. There was no outage or system reboot - not sure how could it get corrupted. Here is the exception, Caused by: java.io.IOException: background merge hit exception: _io5:c66777491 _nh9:c10656736 _taq:c2021563 _s8m:c1421051 _uh5:

Re: background merge hit exception

2009-01-03 Thread Brian Whitman
> > > It's very strange that CheckIndex -fix did not resolve the issue. After > fixing it, if you re-run CheckIndex on the index do you still see that > original one broken segment present? CheckIndex should have removed > reference to that one segment. > I just ran it again, and it detected the

Re: background merge hit exception

2009-01-03 Thread Michael McCandless
It's remotely possible you hit a JVM bug in the past and that caused the corruption. EG there is at least one JVM bug lurking that can affect Lucene (though apparently with an OS-level fault): https://issues.apache.org/jira/browse/LUCENE-1342 I don't know much about Amazon's elastic block stor

Re: background merge hit exception

2009-01-02 Thread Brian Whitman
So my apologies for the duplicate comments, I went to go get proof of duplicates and was confused as we apparently have duplicates across different shards now in our distributed setup (a bug on our end.) I assumed when I saw duplicates that it was the same problem as last time. Still doesn't help m

Re: background merge hit exception

2009-01-02 Thread Michael McCandless
So you have a segment (_tej) with 22201 docs, all but 30 of which are deleted, and somehow one of the posting lists in _tej.frq is referencing an out-of-bound docID 34950. Odd... Are you sure the IO system doesn't have any consistency issues? What environment are you running on (machine, OS, fil

Re: background merge hit exception

2009-01-02 Thread Michael McCandless
Also, this (Solr server going down during an add) should not be able to cause this kind of corruption. Mike Yonik Seeley wrote: > On Fri, Jan 2, 2009 at 3:47 PM, Brian Whitman wrote: > > I will but I bet I can guess what happened -- this index has many > duplicates > > in it as well (same uniqu

Re: background merge hit exception

2009-01-02 Thread Yonik Seeley
On Fri, Jan 2, 2009 at 3:47 PM, Brian Whitman wrote: > I will but I bet I can guess what happened -- this index has many duplicates > in it as well (same uniqueKey id multiple times) - this happened to us once > before and it was because the solr server went down during an add. That should no lon

Re: background merge hit exception

2009-01-02 Thread Brian Whitman
Here's checkindex: NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene', so assertions are enabled Opening index @ /vol/solr/data/index/ Segments file=segments_vxx numSegments=8 version=FORMAT_HAS_PROX [Lucene 2.4] 1 of 8: name=_ks4 docCount=2504982 compound=fal

Re: background merge hit exception

2009-01-02 Thread Brian Whitman
I will but I bet I can guess what happened -- this index has many duplicates in it as well (same uniqueKey id multiple times) - this happened to us once before and it was because the solr server went down during an add. We may have to re-index, but I will run checkIndex now. Thanks (Thread for dupe

Re: background merge hit exception

2009-01-02 Thread Michael McCandless
It looks like your index has some kind of corruption. Were there any other exceptions prior to this one, or, any previous problems with the OS/IO system? Can you run CheckIndex (java org.apache.lucene.index.CheckIndex to see usage) and post the output? Mike Brian Whitman wrote: > I am getting

Re: Background merge hit exception

2008-09-22 Thread Michael McCandless
OK I found one path whereby optimize would detect that the ConcurrentMergeScheduler had hit an exception while merging in a BG thread, and correctly throw an IOException back to its caller, but fail to set the root cause in that exception. I just committed it, so it should be fixed in 2.

Re: Background merge hit exception

2008-09-19 Thread Michael McCandless
vivek sar wrote: Thanks Mike for the insight. I did check the stdout log and found it was complaining of not having enough disk space. I thought we need only x2 of the index size. Our index size is 10G (max) and we had 45G left on that parition - should it still complain of the space? Is ther

Re: Background merge hit exception

2008-09-18 Thread vivek sar
Thanks Mike for the insight. I did check the stdout log and found it was complaining of not having enough disk space. I thought we need only x2 of the index size. Our index size is 10G (max) and we had 45G left on that parition - should it still complain of the space? Some comments/questions on o

Re: Background merge hit exception

2008-09-18 Thread Michael McCandless
Lucene tries to carry forward the root cause exception from the merge, into that IOException that optimize throws. But it doesn't always succeed in doing so; I'll open a Jira issue and try to figure out why this is the case. All the exception "means" is that the optimize didn't finish -- you sti

Re: background merge hit exception

2008-07-21 Thread Michael McCandless
Did you see a "caused by" in your stack trace? Or, a separate unhandled exception in a thread? Is this reproducible? Also, did you have a reader open on the index when you ran the optimize? If so, you'd need 2X the index size (eg 20 GB with the example below) free. Mike sandyg wrote

Re: background merge hit exception

2008-07-21 Thread sandyg
Hi, thx for the reply. But i had enough space in my desk . Michael McCandless-2 wrote: > > > Normally when optimize throws this exception, either it includes a > "caused by" in the exception, or, you should have seen a stack trace > from a merge thread printed to your stderr. > > One quic

Re: background merge hit exception

2008-07-12 Thread Michael McCandless
Normally when optimize throws this exception, either it includes a "caused by" in the exception, or, you should have seen a stack trace from a merge thread printed to your stderr. One quick thing to check is whether you have enough disk space to complete the optimize. It looks like that