I think if you just log to the console, kubernetes will manage the log
rotation for you.
https://kubernetes.io/docs/concepts/cluster-administration/logging/
You can use "kubectl logs" command to fetch the logs or use some logging
agent to move the logs centrally.
On Tue, Aug 4, 2020 at 8:15 PM Srinivas Seema <srinivassee...@gmail.com>
wrote:
> Hi All,
>
> We have Kafka cluster deployed in Kubernetes and running with docker image
> (solsson/kafka:2.4.0)
>
> I have below logging configuration: config/log4j.properties
>
> # Unspecified loggers and loggers with additivity=true output to server.log
> and stdout
> # Note that INFO only applies to unspecified loggers, the log level of the
> child logger is used otherwise
> log4j.rootLogger=INFO, stdout, kafkaAppender
>
> log4j.appender.stdout=org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
>
> log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log
> log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
>
>
> log4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.stateChangeAppender.File=${kafka.logs.dir}/state-change.log
> log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m
> (%c)%n
>
> log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.requestAppender.File=${kafka.logs.dir}/kafka-request.log
> log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
>
> log4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log
> log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
>
> log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH
> log4j.appender.controllerAppender.File=${kafka.logs.dir}/controller.log
> log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m
> (%c)%n
>
> log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH
>
> log4j.appender.authorizerAppender.File=${kafka.logs.dir}/kafka-authorizer.log
> log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m
> (%c)%n
>
> # Change the line below to adjust ZK client logging
> log4j.logger.org.apache.zookeeper=INFO
>
> # Change the two lines below to adjust the general broker logging level
> (output to server.log and stdout)
> log4j.logger.kafka=INFO
> log4j.logger.org.apache.kafka=INFO
>
> # Change to DEBUG or TRACE to enable request logging
> log4j.logger.kafka.request.logger=WARN, requestAppender
> log4j.additivity.kafka.request.logger=false
>
> # Uncomment the lines below and change
> log4j.logger.kafka.network.RequestChannel$ to TRACE for additional output
> # related to the handling of requests
> #log4j.logger.kafka.network.Processor=TRACE, requestAppender
> #log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender
> #log4j.additivity.kafka.server.KafkaApis=false
> log4j.logger.kafka.network.RequestChannel$=WARN, requestAppender
> log4j.additivity.kafka.network.RequestChannel$=false
>
> log4j.logger.kafka.controller=TRACE, controllerAppender
> log4j.additivity.kafka.controller=false
>
> log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender
> log4j.additivity.kafka.log.LogCleaner=false
>
> log4j.logger.state.change.logger=TRACE, stateChangeAppender
> log4j.additivity.state.change.logger=false
>
> # Access denials are logged at INFO level, change to DEBUG to also log
> allowed accesses
> log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender
> log4j.additivity.kafka.authorizer.logger=false
>
> Problem Statement:
> logs are getting piled up and using more disk and pods are going to evicted
> state.
>
> I would like to know the best approach to rotate the logs and archive
> (kafka component wise & provided size e.g: 10 MB), so that I can analyze
> logs and save the disk space.
>
> ~Srinivas
>
