We have observed Journal Corruption while replaying the journal log.
I could be missing something but I don't see we have clear indication of
where
journal file ends.
In the Journal.java:scanJournal() we have multiple breaks but mostly
we assume either the journal file ends in the right place or the length is
zero.
scanJournal() {
...
while(true) {
// entry start offset
long offset = recLog.fc.position();
// start reading entry
lenBuff.clear();
fullRead(recLog, lenBuff);
if (lenBuff.remaining() != 0) {
break;
}
lenBuff.flip();
int len = lenBuff.getInt();
if (len == 0) {
break;
}
}
====
So there is an implicit assumption here that the journal file ends
or the length is zero.
But... JournalChannel.java:
void preAllocIfNeeded(long size) throws IOException {
if (bc.position() + size > nextPrealloc) {
nextPrealloc += preAllocSize;
zeros.clear();
fc.write(zeros, nextPrealloc - journalAlignSize);
}
}
We extend the file withe prealloc size at the journalAlignSize.
Based on our tunables (in our case preAllocSize:4MB journalAlignSize:512)
We could extend the file and the fc.write and we are writing only the
last journalAlignSize of zeros. i.e 4MB-512 can be garbage in the file.
When the journal is replayed there is a potential for garbage and journal
corruption indication.
Being said that we haven't see this so far, so the bug may not be
this obvious.. anyway just trying to see what I am missing.
Here is the stack:
20191010181408.832000 [INFO ] [main] [JournalChannel.java:154] -
Opening journal /sfs/sfsdata/journal/current/16cdf99de69.txn
Exception in thread "main" java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:827)
at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:1047)
at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:1009)
at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1070)
at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:141)
at
org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:58)
at
org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:78)
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408)
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113)
at
org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:80)
at org.apache.bookkeeper.server.Main.doMain(Main.java:228)
at org.apache.bookkeeper.server.Main.main(Main.java:202)
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:296)
--
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi
