Take a look at log.index.size.max.bytes described in http://kafka.apache.org/documentation.html#brokerconfigs
Thanks, Jun On Sun, Nov 3, 2013 at 4:55 PM, lee yeonok <iyeo...@gmail.com> wrote: > Hi, all. > I am not quite understood how *.index files are used or created. > > I started a broker1, and set the property "log.segment.bytes" to "100". > Here's a file tree in log directory what I've got after that broker1 > received some messages. > > kafka-logs-1/ > |-- preview-0 > | |-- 00000000000000000000.index > | |-- 00000000000000000000.log > | |-- 00000000000000000031.index > | |-- 00000000000000000031.log > | |-- 00000000000000000033.index > | |-- 00000000000000000033.log > | |-- 00000000000000000035.index > | |-- 00000000000000000035.log > | |-- 00000000000000000037.index > | |-- 00000000000000000037.log > | |-- 00000000000000000039.index > | `-- 00000000000000000039.log > > > At the point the segment size reached at 100 bytes, a new file is created > as I expected, however, I am not sure why the old *.index files are being > truncated. > In my log directory, 00000000000000000000.index, > 00000000000000000031.index, 00000000000000000033.index, > 00000000000000000035.index, > 00000000000000000037.index are zero size file. Is that all right? > > I thought that in this case the consumers are not able to get the messages > from beginning since the old index files are being truncated, however I > suspect that there's no problem to consume the messages from beginning. > > I removed all the index files, and found that when the broker get new > message the latest index file is recovered. At this moment the old index > files doesn't exists though, consuming all messages from beginning is not > problem. > > So my conclusion is this Kafka only has the latest index, and in the case > the old message requested Kafka re-parse every log files(not refer index > file). > Is it right? If then, having a zero size index files is unnecessary.... > Why does Kafka recreate all the missing index files when the broker shut > down? > I shut down a broker1, and the removed index files are recreated. > > > -- > yeonok lee >
