@Mohit:
When flume dies unexpectedly the .tmp file remains. When it restarts
there is some logic in HDFS sink to recover it(and continue writing
from there). I'm not actually sure of the specifics. You may want to
try and just kill -9 a running flume process on a test machine and
then start it up, look at the logs and see what happens with the output.
Does it also work when there is a long delay before flume gets
started? We are bucketing by the hr so if start occurs in the next
hour but flume actually died in previous hr and had .tmp then does it
still cleanup on restart
I'm not sure. I think your best bet here is to simulate this on a test
server. Start flume, after a bit kill 9 the process, wait until the
bucket becomes invalid, and restart.
My gut feeling is that it will recover if you have events with the
timestamp belonging to that bucket still incoming (in your persistent
channelor read in after recovery). If that path doesn't get touched
again though, it will probably remain as a .tmp file? *This could be
blatantly wrong, so I suggest you test it*
