Hi Jose,
Thank you for very informative response.
I have commented out the line of codes that does the forceMerge(50) and
commit() while the indexing is happening. Also increase the ram buffer size
iwc.setRAMBufferSizeMB(512.0);
and after index is done, then only forceMerge and commit but this
Hi Jason,
No, the StrackTrace shows clearly the cause of the errror occurred during
the merge into a single index file segment(forgeMerge parameter defines the
number of desired segments at end).
During the indexing of a document, Lucene might decide to create a new
segment of the information ext
Hello Jose,
Thank you for your response, I took a closer look. Below are my responses:
> Seems that you want to force a max number of segments to 1,
// you're done adding documents to it):
//
writer.forceMerge(1);
writer.close();
Yes, the line of code is uncommented be
Seems that you want to force a max number of segments to 1,
On a previous thread someone answered that the number of segments will
affect the Index Size, and is not related with Index Integrity (like size
of index may vary according with number of segments).
on version 4.6 there is a small issue o
I've got some follow-up from the user.
> Is it possible disk filled up? Though I'd expect an IOE during write
> or close in that case.
>
> In this case nothing should be lost in the index: the merge simply
> refused to commit itself, since it detected something went wrong. But
> I believe we al
On Sat, Sep 10, 2011 at 12:58 PM, Michael McCandless
wrote:
> On Fri, Sep 9, 2011 at 6:50 PM, Koji Sekiguchi wrote:
Also: what java version are they running? We added this check
originally as a workaround for a JRE bug... but usually when that bug
strikes the file size is very clos
On Fri, Sep 9, 2011 at 6:50 PM, Koji Sekiguchi wrote:
>>> Also: what java version are they running? We added this check
>>> originally as a workaround for a JRE bug... but usually when that bug
>>> strikes the file size is very close (like off by just 1 byte or 8
>>> bytes or something).
>>
>> I t
Also: what java version are they running? We added this check
originally as a workaround for a JRE bug... but usually when that bug
strikes the file size is very close (like off by just 1 byte or 8
bytes or something).
I think they are using 1.6, but I should ask the minor number.
Could you show
Hi Mike,
This wouldn't be caused by the "NFS happily deletes open files"
problem (= Stale NFS file handle error).
But this could in theory be caused by the NFS client somehow being
wrong about the file's metadata (file length). It's sort of odd
because I would expect since the client wrote the
Interesting...
This wouldn't be caused by the "NFS happily deletes open files"
problem (= Stale NFS file handle error).
But this could in theory be caused by the NFS client somehow being
wrong about the file's metadata (file length). It's sort of odd
because I would expect since the client wrote
Also: what sorts of stored fields are you storing? Binary?
Compressed? Text with unicode characters? Roughly how many stored
fields per document?
Mike
vivek sar wrote:
Hi,
We ran into the same issue (corrupted index) using Lucene 2.4.0.
There was no outage or system reboot - not su
The exception you hit during optimize indicates some corruption in the
stored fields file (_X.fdt). Then, the exception hit during
CheckIndex was different -- the postings entry (_r0y.frq) for term
"desthost:wir" was supposed to have 6 entries but had 0.
Did "CheckIndex -fix" allow you to optim
Hi,
We ran into the same issue (corrupted index) using Lucene 2.4.0.
There was no outage or system reboot - not sure how could it get
corrupted. Here is the exception,
Caused by: java.io.IOException: background merge hit exception:
_io5:c66777491 _nh9:c10656736 _taq:c2021563 _s8m:c1421051
_uh5:
>
>
> It's very strange that CheckIndex -fix did not resolve the issue. After
> fixing it, if you re-run CheckIndex on the index do you still see that
> original one broken segment present? CheckIndex should have removed
> reference to that one segment.
>
I just ran it again, and it detected the
It's remotely possible you hit a JVM bug in the past and that caused the
corruption.
EG there is at least one JVM bug lurking that can affect Lucene (though
apparently with an OS-level fault):
https://issues.apache.org/jira/browse/LUCENE-1342
I don't know much about Amazon's elastic block stor
So my apologies for the duplicate comments, I went to go get proof of
duplicates and was confused as we apparently have duplicates across
different shards now in our distributed setup (a bug on our end.) I assumed
when I saw duplicates that it was the same problem as last time. Still
doesn't help m
So you have a segment (_tej) with 22201 docs, all but 30 of which are
deleted, and somehow one of the posting lists in _tej.frq is referencing an
out-of-bound docID 34950. Odd...
Are you sure the IO system doesn't have any consistency issues? What
environment are you running on (machine, OS, fil
Also, this (Solr server going down during an add) should not be able to
cause this kind of corruption.
Mike
Yonik Seeley wrote:
> On Fri, Jan 2, 2009 at 3:47 PM, Brian Whitman wrote:
> > I will but I bet I can guess what happened -- this index has many
> duplicates
> > in it as well (same uniqu
On Fri, Jan 2, 2009 at 3:47 PM, Brian Whitman wrote:
> I will but I bet I can guess what happened -- this index has many duplicates
> in it as well (same uniqueKey id multiple times) - this happened to us once
> before and it was because the solr server went down during an add.
That should no lon
Here's checkindex:
NOTE: testing will be more thorough if you run java with
'-ea:org.apache.lucene', so assertions are enabled
Opening index @ /vol/solr/data/index/
Segments file=segments_vxx numSegments=8 version=FORMAT_HAS_PROX [Lucene
2.4]
1 of 8: name=_ks4 docCount=2504982
compound=fal
I will but I bet I can guess what happened -- this index has many duplicates
in it as well (same uniqueKey id multiple times) - this happened to us once
before and it was because the solr server went down during an add. We may
have to re-index, but I will run checkIndex now. Thanks
(Thread for dupe
It looks like your index has some kind of corruption. Were there any other
exceptions prior to this one, or, any previous problems with the OS/IO
system?
Can you run CheckIndex (java org.apache.lucene.index.CheckIndex to see
usage) and post the output?
Mike
Brian Whitman wrote:
> I am getting
OK I found one path whereby optimize would detect that the
ConcurrentMergeScheduler had hit an exception while merging in a BG
thread, and correctly throw an IOException back to its caller, but
fail to set the root cause in that exception. I just committed it, so
it should be fixed in 2.
vivek sar wrote:
Thanks Mike for the insight. I did check the stdout log and found it
was complaining of not having enough disk space. I thought we need
only x2 of the index size. Our index size is 10G (max) and we had 45G
left on that parition - should it still complain of the space?
Is ther
Thanks Mike for the insight. I did check the stdout log and found it
was complaining of not having enough disk space. I thought we need
only x2 of the index size. Our index size is 10G (max) and we had 45G
left on that parition - should it still complain of the space?
Some comments/questions on o
Lucene tries to carry forward the root cause exception from the merge,
into that IOException that optimize throws. But it doesn't always
succeed in doing so; I'll open a Jira issue and try to figure out why
this is the case.
All the exception "means" is that the optimize didn't finish -- you
sti
Did you see a "caused by" in your stack trace? Or, a separate
unhandled exception in a thread? Is this reproducible?
Also, did you have a reader open on the index when you ran the
optimize? If so, you'd need 2X the index size (eg 20 GB with the
example below) free.
Mike
sandyg wrote
Hi,
thx for the reply.
But i had enough space in my desk .
Michael McCandless-2 wrote:
>
>
> Normally when optimize throws this exception, either it includes a
> "caused by" in the exception, or, you should have seen a stack trace
> from a merge thread printed to your stderr.
>
> One quic
Normally when optimize throws this exception, either it includes a
"caused by" in the exception, or, you should have seen a stack trace
from a merge thread printed to your stderr.
One quick thing to check is whether you have enough disk space to
complete the optimize. It looks like that
29 matches
Mail list logo
