Hi, all. I am not quite understood how *.index files are used or created. I started a broker1, and set the property "log.segment.bytes" to "100". Here's a file tree in log directory what I've got after that broker1 received some messages.
kafka-logs-1/ |-- preview-0 | |-- 00000000000000000000.index | |-- 00000000000000000000.log | |-- 00000000000000000031.index | |-- 00000000000000000031.log | |-- 00000000000000000033.index | |-- 00000000000000000033.log | |-- 00000000000000000035.index | |-- 00000000000000000035.log | |-- 00000000000000000037.index | |-- 00000000000000000037.log | |-- 00000000000000000039.index | `-- 00000000000000000039.log At the point the segment size reached at 100 bytes, a new file is created as I expected, however, I am not sure why the old *.index files are being truncated. In my log directory, 00000000000000000000.index, 00000000000000000031.index, 00000000000000000033.index, 00000000000000000035.index, 00000000000000000037.index are zero size file. Is that all right? I thought that in this case the consumers are not able to get the messages from beginning since the old index files are being truncated, however I suspect that there's no problem to consume the messages from beginning. I removed all the index files, and found that when the broker get new message the latest index file is recovered. At this moment the old index files doesn't exists though, consuming all messages from beginning is not problem. So my conclusion is this Kafka only has the latest index, and in the case the old message requested Kafka re-parse every log files(not refer index file). Is it right? If then, having a zero size index files is unnecessary.... Why does Kafka recreate all the missing index files when the broker shut down? I shut down a broker1, and the removed index files are recreated. -- yeonok lee
