We happened to run into a disk space usage issue with Kafka 0.10.0.1
(the version we are using) on one of our production setups this morning.
Turns out (log4j) logging from Kafka ended up using 81G and more of disk
space. Looking at the files, I see the controller.log itself is 30G and
more (for a single day). Looking at the default log4j.properties that's
shipped in Kafka, it uses the DailyRollingFileAppender which is one of
the things that contributes to this issue. I see that there's already a
patch and JIRA to fix this
https://issues.apache.org/jira/browse/KAFKA-2394. It's been marked for
0.11 because there wasn't a clear decision when to ship it.
Given that we have been going through 0.10.x releases these days and the
0.11 release looking a bit away, is there any chance, this specific JIRA
can make it to 0.10.x? I personally don't see any compatibility issues
that it will introduce when it comes to *functionality/features* of
Kafka itself, so I am not sure if it's that big a change to wait all the
way till 0.11. Furthermore, since the default shipped setting can cause
issues like the one I noted, I think it probably would be fine to
include it in one of the 0.10.x releases. Of course, we ourselves can
setup the logging config on our setup to use a size based rolling file
config instead of the one shipped by default, but if this is something
that can make it to 0.10.x, we would like to avoid doing this
customization ourselves.
That's one part of the issue. The other is, I see this in the default
shipped log4j.properties:
log4j.logger.kafka.controller=*TRACE,* controllerAppender
log4j.additivity.kafka.controller=false
log4j.logger.state.change.logger=*TRACE*, stateChangeAppender
log4j.additivity.state.change.logger=false
Is it intentional to have this at TRACE level for the default shipped
config instead of having something like INFO or maybe DEBUG?
-Jaikiran
