Hi all,
we just had a case with Kafka 0.9 where an index rebuild for ~200M
segments took on average 45 seconds. All indexes of a partition were
corrupt. There are 13 segments and the rebuild took 10 minutes.
After the rebuild, these are representative sizes:
% ll -h /data/xyz-0
-rw-r--r-- 1 solr solr 45K Aug 16 10:44 00000000000000096346.index
-rw-r--r-- 1 solr solr 191M Aug 16 10:44 00000000000000096346.log
I now wonder whether this is an expected. Here is a log-excerpt showing
the long run time for this segment, this one more than one minute.
[2016-08-16 10:44:20,831] WARN Found a corrupted index file,
/data/xyz-0/00000000000000096346.index, deleting and rebuilding index...
(kafka.log.Log)
[2016-08-16 10:45:46,305] WARN Found a corrupted index file,
/data/xyz-0/00000000000000011722.index, deleting and rebuilding index...
(kafka.log.Log)
These runtime seem excessive on normal desktop hardware. Or am I
underestimating the necessary effort to rebuild an index file?
Harald.
