Hi Jaikiran, Thanks for raising this. The compatibility issue is that the file name pattern for the rolled files are different. Because there's a simple workaround, it was deemed that it could wait until 0.11.0. This is actually not far off if you consider that we have 4 active KIPs that are proposing message format changes (a message format change implies a major version bump).
The log config settings for the controller and state change logger have been that way since they were introduced. They're generally useful when investigating issues with the controller. Looks like this is too noisy in some scenarios though. It may be worth filing a JIRA with specifics of your use case to see if something can be done to improve that. Ismael On Tue, Dec 13, 2016 at 9:32 AM, Jaikiran Pai <jai.forums2...@gmail.com> wrote: > We happened to run into a disk space usage issue with Kafka 0.10.0.1 (the > version we are using) on one of our production setups this morning. Turns > out (log4j) logging from Kafka ended up using 81G and more of disk space. > Looking at the files, I see the controller.log itself is 30G and more (for > a single day). Looking at the default log4j.properties that's shipped in > Kafka, it uses the DailyRollingFileAppender which is one of the things that > contributes to this issue. I see that there's already a patch and JIRA to > fix this https://issues.apache.org/jira/browse/KAFKA-2394. It's been > marked for 0.11 because there wasn't a clear decision when to ship it. > > Given that we have been going through 0.10.x releases these days and the > 0.11 release looking a bit away, is there any chance, this specific JIRA > can make it to 0.10.x? I personally don't see any compatibility issues that > it will introduce when it comes to *functionality/features* of Kafka > itself, so I am not sure if it's that big a change to wait all the way till > 0.11. Furthermore, since the default shipped setting can cause issues like > the one I noted, I think it probably would be fine to include it in one of > the 0.10.x releases. Of course, we ourselves can setup the logging config > on our setup to use a size based rolling file config instead of the one > shipped by default, but if this is something that can make it to 0.10.x, we > would like to avoid doing this customization ourselves. > > That's one part of the issue. The other is, I see this in the default > shipped log4j.properties: > > > log4j.logger.kafka.controller=*TRACE,* controllerAppender > log4j.additivity.kafka.controller=false > > log4j.logger.state.change.logger=*TRACE*, stateChangeAppender > log4j.additivity.state.change.logger=false > > > Is it intentional to have this at TRACE level for the default shipped > config instead of having something like INFO or maybe DEBUG? > > > -Jaikiran >
