Re: IO exception during merge/optimize

A couple more data points: RamSizeIndex(min)Optimize(min)Peak mem 1.9G2455G 800M2454G 400M25 53.5G 100M2553G 50M 26 43G Peter On Thu, Oct 29, 2009 at 8:4

Re: IO exception during merge/optimize

Thanks a lot Peter! Really appreciate it. Peter Keegan wrote: > Mark, > > With 1.9G, I had to increase the JVM heap significantly (to 8G) to avoid > paging and GC hits. Here is a table comparing indexing times, optimizing > times and peak memory usage as a function of the RAMBufferSize. This was

Re: IO exception during merge/optimize

Mark, With 1.9G, I had to increase the JVM heap significantly (to 8G) to avoid paging and GC hits. Here is a table comparing indexing times, optimizing times and peak memory usage as a function of the RAMBufferSize. This was run on a 64-bit server with 32GB RAM: RamSizeIndex(min)Opt

Re: IO exception during merge/optimize

Any chance I could get you to try that again with a buffer of like 800MB to a gig and do a comparison? I've been investigating the returns you get with a larger buffer size. It appears to be pretty diminishing returns over 100MB or so - at higher than that, I've gotten both slower speeds for some

Re: IO exception during merge/optimize

I'm glad we finally got to the bottom of this :) This fix will be in 2.9.1. This is a nice fast indexing result, too... Mike On Thu, Oct 29, 2009 at 3:55 PM, Peter Keegan wrote: > Btw, this 2.9 indexer is fast! I indexed 4Gb (1.07 million docs) with > optimization in just under 30 min. > I use

Re: IO exception during merge/optimize

Btw, this 2.9 indexer is fast! I indexed 4Gb (1.07 million docs) with optimization in just under 30 min. I used setRAMBufferSizeMB=1.9G Peter On Thu, Oct 29, 2009 at 3:46 PM, Peter Keegan wrote: > A handful of the source documents did contain the U+ character. The > patch from *LUCENE-2016

Re: IO exception during merge/optimize

A handful of the source documents did contain the U+ character. The patch from *LUCENE-2016 *fixed the problem. Thanks Mike! Peter On Wed, Oct 28, 2009 at 1:29 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > Hmm, only a few aff

Re: IO exception during merge/optimize

Hmm, only a few affected terms, and all this particular "literals:cfid196$" term, with optional suffixes. Really strange. One things that's odd is the exact term "literals:cfid196$" is printed twice, which should never happen (every unique term should be stored only once, in the terms dict). And

Re: IO exception during merge/optimize

Just to be safe, I ran with the official jar file from one of the mirrors and reproduced the problem. The debug session is not showing any characters = '\u' (checking this in Tokenizer). The output from the modified CheckIndex follows. There are only a few terms with the inconsistency. They are

RE: IO exception during merge/optimize

less [mailto:luc...@mikemccandless.com] > Sent: Wednesday, October 28, 2009 4:25 PM > To: java-user@lucene.apache.org > Subject: Re: IO exception during merge/optimize > > Right, I would expect Lucene would silently truncate the term at the > U+, and not lead to this odd exception. >

Re: IO exception during merge/optimize

On Wed, Oct 28, 2009 at 10:58 AM, Peter Keegan wrote: > The only change I made to the source code was the patch for PayloadNearQuery > (LUCENE-1986). That patch certainly shouldn't lead to this. > It's possible that our content contains U+. I will run in debugger and > see. OK may as well c

Re: IO exception during merge/optimize

thats exactly the result I saw FWIW On Wed, Oct 28, 2009 at 11:25 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > Right, I would expect Lucene would silently truncate the term at the > U+, and not lead to this odd exception. > > Mike > > On Wed, Oct 28, 2009 at 11:23 AM, Robert M

Re: IO exception during merge/optimize

Right, I would expect Lucene would silently truncate the term at the U+, and not lead to this odd exception. Mike On Wed, Oct 28, 2009 at 11:23 AM, Robert Muir wrote: > i might be wrong about this, but recently I intentionally tried to create > index with terms with U+ to see if it would

Re: IO exception during merge/optimize

i might be wrong about this, but recently I intentionally tried to create index with terms with U+ to see if it would cause a problem :) the U+ seemed to be discarded completely (maybe at UTF-8 encode time)... then again I was using RAMDirectory. On Wed, Oct 28, 2009 at 10:58 AM, Peter Ke

Re: IO exception during merge/optimize

The only change I made to the source code was the patch for PayloadNearQuery (LUCENE-1986). It's possible that our content contains U+. I will run in debugger and see. The data is 'sensitive', so I may not be able to provide a bad segment, unfortunately. Peter On Wed, Oct 28, 2009 at 10:43 AM

Re: IO exception during merge/optimize

OK... when you exported the sources & built yourself, you didn't make any changes, right? It's really odd how many of the errors are due to the term "literals:cfid196$", or some variation (one time with "on" appended, another time with "microsoft"). Do you know what documents typically contain th

RE: IO exception during merge/optimize

> >Also, what does Lucene version "2.9 exported - 2009-10-27 15:31:52" mean? > This appears to be something added by the ant build, since I built Lucene > from the source code. This is because it was build from a source artifact with no SVN revision information. At this place, normally the svn rev

Re: IO exception during merge/optimize

My last post got truncated - probably exceeded max msg size. Let me know if you want to see more of the IndexWriter log. Peter

Re: IO exception during merge/optimize

The unit tests do test multi-segment indexes (though we could always use deeper testing, here), but, don't test big-ish indexes, like this, very well. Are you also using JDK 1.6.0_16 when running CheckIndex? If you run CheckIndex on the same index several times in a row, does it report precisely

Re: IO exception during merge/optimize

Without the optimize, it looks like there are errors on all segments except the first: Opening index @ D:\mnsavs\lresumes1\lresumes1.luc\lresumes1.search.main.2 Segments file=segments_2 numSegments=3 version=FORMAT_DIAGNOSTICS [Lucene 2.9] 1 of 3: name=_0 docCount=413557 compound=false

Re: IO exception during merge/optimize

It's reproducible with a large no. of docs (>1 million), but not with 100K docs. I got same error with jvm 1.6.0_16. The index was optimized after all docs are added. I'll try removing the optimize. Peter On Tue, Oct 27, 2009 at 2:57 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > T

Re: IO exception during merge/optimize

This is odd -- is it reproducible? Can you narrow it down to a small set of docs that when indexed produce a corrupted index? If you attempt to optimize the index, does it fail? Mike On Tue, Oct 27, 2009 at 1:40 PM, Peter Keegan wrote: > It seems the index is corrupted immediately after the in

Re: IO exception during merge/optimize

It seems the index is corrupted immediately after the initial build (ample disk space was provided): Output from CheckIndex: Opening index @ D:\mnsavs\lresumes1\lresumes1.luc\lresumes1.search.main.2 Segments file=segments_3 numSegments=1 version=FORMAT_DIAGNOSTICS [Lucene 2.9] 1 of 1: name=_7

Re: IO exception during merge/optimize

On Tue, Oct 27, 2009 at 10:37 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > OK that exception looks more reasonable, for a disk full event. > > But, I can't tell from your followon emails: did this lead to index > corruption? > Yes, but this may be caused by the application ignorin

Re: IO exception during merge/optimize

OK that exception looks more reasonable, for a disk full event. But, I can't tell from your followon emails: did this lead to index corruption? Also, I noticed you're using a rather old 1.6.0 JRE (1.6.0_03) -- you really should upgrade that to the latest 1.6.0 -- there's at least one known proble

Re: IO exception during merge/optimize

Clarification: this CheckIndex is on the index from which the merge/optimize failed. Peter On Tue, Oct 27, 2009 at 10:07 AM, Peter Keegan wrote: > Running CheckIndex after the IOException did produce an error in a term > frequency: > > Opening index @ D:\mnsavs\lresumes3\lresumes3.luc\lresumes3.s

Re: IO exception during merge/optimize

Running CheckIndex after the IOException did produce an error in a term frequency: Opening index @ D:\mnsavs\lresumes3\lresumes3.luc\lresumes3.search.main.3 Segments file=segments_4 numSegments=2 version=FORMAT_DIAGNOSTICS [Lucene 2.9] 1 of 2: name=_7 docCount=1075533 compound=false has

Re: IO exception during merge/optimize

After rebuilding the corrupted indexes, the low disk space exception is now occurring as expected. Sorry for the distraction. fyi, here are the details: java.io.IOException: There is not enough space on the disk at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAcces

Re: IO exception during merge/optimize

On Mon, Oct 26, 2009 at 3:00 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > On Mon, Oct 26, 2009 at 2:55 PM, Peter Keegan > wrote: > > On Mon, Oct 26, 2009 at 2:50 PM, Michael McCandless < > > luc...@mikemccandless.com> wrote: > > > >> On Mon, Oct 26, 2009 at 10:44 AM, Peter Keegan

Re: IO exception during merge/optimize

On Mon, Oct 26, 2009 at 2:55 PM, Peter Keegan wrote: > On Mon, Oct 26, 2009 at 2:50 PM, Michael McCandless < > luc...@mikemccandless.com> wrote: > >> On Mon, Oct 26, 2009 at 10:44 AM, Peter Keegan >> wrote: >> > Even running in console mode, the exception is difficult to interpret. >> > Here's an

Re: IO exception during merge/optimize

On Mon, Oct 26, 2009 at 2:50 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > On Mon, Oct 26, 2009 at 10:44 AM, Peter Keegan > wrote: > > Even running in console mode, the exception is difficult to interpret. > > Here's an exception that I think occurred during an add document, commit

Re: IO exception during merge/optimize

On Mon, Oct 26, 2009 at 10:44 AM, Peter Keegan wrote: > Even running in console mode, the exception is difficult to interpret. > Here's an exception that I think occurred during an add document, commit or > close: > doc counts differ for segment _g: field Reader shows 137 but segmentInfo > shows 5

Re: IO exception during merge/optimize

Even running in console mode, the exception is difficult to interpret. Here's an exception that I think occurred during an add document, commit or close: doc counts differ for segment _g: field Reader shows 137 but segmentInfo shows 5777 I ensured that the disk space was low before updating the ind

Re: IO exception during merge/optimize

The environment involves a lot of I/O from merge/optimize operations on multiple indexes (shards) on one server. I will try running the indexers in console mode, where I would expect to see all errors and exceptions. Peter On Sun, Oct 25, 2009 at 8:40 PM, Michael McCandless < luc...@mikemccandle

Re: IO exception during merge/optimize

Hmm, if you got no exception whatsoever, something more fundamental seems to be wrong w/ the error reporting when running as a windows service. Maybe make a simple Java test program that throws an exception and try to get that working? Mike On Sun, Oct 25, 2009 at 7:35 PM, Peter Keegan wrote: >

Re: IO exception during merge/optimize

>Did you get any traceback printed at all? no, only what I reported. >Did you see any BG thread exceptions on wherever your System.err is directed to? The jvm was running as a windows service, so output to System.err may have gone to the bit bucket. That's an interesting point, though. Peter On

Re: IO exception during merge/optimize

Hmm... Lucene tries to catch the original cause (from the BG thread doing the merge) and forward it to the main thread waiting for optimize to complete. Did you get any traceback printed at all? It should include one traceback into Lucene's optimized method, and then another (under "caused by") s

Re: IO exception during merge/optimize

btw, this is with Lucene 2.9 On Sat, Oct 24, 2009 at 5:20 PM, Peter Keegan wrote: > I'm sometimes seeing the following exception from an operation that does a > merge and optimize: > java.io.IOException: background merge hit exception: _0:C1082866 _1:C79 > into _2 [optimize] [mergeDocStores] > I

IO exception during merge/optimize

I'm sometimes seeing the following exception from an operation that does a merge and optimize: java.io.IOException: background merge hit exception: _0:C1082866 _1:C79 into _2 [optimize] [mergeDocStores] I'm pretty sure that it's caused by a temporary low disk space condition, but I'd like to be ab