Hey,
In GarbageCollectorThread.doCompactEntryLogs {
...
try {
compactEntryLog(scannerFactory, meta);
scannerFactory.flush(); <--------------- this
will eventually call entrylogger.flushCurrentLog and it force writes the
content of the BufferedChannel but not the metadata of the file
LOG.info("Removing entry log {} after compaction",
meta.getEntryLogId());
removeEntryLog(meta.getEntryLogId()); <----------- this
will delete the compacted entrylog
}
...
in doCompactEntryLogs, we first write the non-deleted entries of the
entryLog which is getting compacted to the currentLog in entryLogger, then
we flush the entrylogger before deleting the compacted entrylog. But when
currentLog is flushed by the entrylogger, it flushes only the content of
the file but not the metadata. After flush is completed the compacted
entrylog is deleted. It is not ok to not to force flush the metadata of the
currentLog for the persisted (checkpointed) data. The filesystem behaviour
is unexpected in this case and there is possibility of data loss if the
Bookie crashes before closing that logchannel.
Thanks,
Charan
