OK it does sound like you're primarily protecting against an
untrustworthy storage system (or, maybe, Lucene bugs ;).
Probably the best option is to do this fully externally, ie, compute
digest yourself, store it away in a separate Lucene field, then test
the digest on loading the field?
Mike
Jason Rutherglen wrote:
I'm thinking more in terms of CRC32 checks performed on database
pages. Is there a way to incorporate this technique in a way that
does not affect performance too much in Lucene? The question is, when
is the CRC32 check is performed, and to which files is it applied if
any?
On Fri, Sep 26, 2008 at 12:13 PM, Michael McCandless
<[EMAIL PROTECTED]> wrote:
Corrupted data in what sense?
EG if you don't trust your IO system to store data properly?
Mike
Jason Rutherglen wrote:
Mike,
As part of my goal of trying to use Lucene as primary storage
mechanism (perhaps not the best idea), what do you think is the best
way to handle storing data in Lucene and preventing corrupted data
the
way something like an SQL database handles corrupted data? Or is
there simply no good way to do this?
Jason
On Fri, Sep 26, 2008 at 10:30 AM, Michael McCandless
<[EMAIL PROTECTED]> wrote:
It's perfectly fine to have a reader open on an index, while an
IndexWriter
runs optimize.
Which version of Lucene are you using? And which OS & filesystem?
Mike
Chaula Ganatra wrote:
It was the Reader on same index, which I did not close so gave
exception
in writer.optimise()
Chaula
-----Original Message-----
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 7:17 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
Can you post the full stack trace in both cases?
Mike
Chaula Ganatra wrote:
I found one case when such multiple files are remained, when we
call
writer.optimise() it throws exception and multiple files
remained in
index dir.
After such multiple files, when we add document in index by
calling
writer.addDocument it throws java.lang.NegativeArraySizeException
Regards,
Chaula
-----Original Message-----
From: Grant Ingersoll [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 6:02 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
There is the CheckIndex tool included in the distribution for
checking/
fixing bad indexes, but it can't solve everything.
The bigger question is why it is happening to begin with. Can
you
describe your indexing process? How do you know the index is
actually
corrupted? Are you seeing exceptions when opening it?
-Grant
On Sep 26, 2008, at 6:49 AM, Chaula Ganatra wrote:
We have an application in which index will be updated
frequently.
During development time, found that index files gets
corrupted, i.e.
more than one cfs files,some other extension files e.g. frq,
fnm,
nrm
Remains there in index directory.
Is there any way that such issue does not occur at all or if it
happens
we can recover the index data again?
It would be a great help, if some one can.
Regards,
Chaula
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
