Hi, TBH, I don't have much experience with logging, but you might want to consider using Side Outputs [1] to route invalid records into a separate stream. The stream can then separately handled, be written to files or Kafka or wherever.
Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/side_output.html 2018-03-20 10:36 GMT+01:00 Puneet Kinra <puneet.ki...@customercentria.com>: > Hi > > I have a use case in which i want to log bad records in the log file. I > have configured the log4j > property file is getting generated as well but it also going to flink logs > as well i want to detach > it from flink logs want to write to log file. > > .Here is configuration > *(Note :AMSSource is the custom written adaptor here)* > > # This affects logging for both user code and Flink > log4j.rootLogger=INFO, file > log4j.logger.amssource=DEBUG, amssourceAppender > > # Uncomment this if you want to _only_ change Flink's logging > #log4j.logger.org.apache.flink=INFO > > # The following lines keep the log level of common libraries/connectors on > # log level INFO. The root logger does not override this. You have to > manually > # change the log levels here. > log4j.logger.akka=INFO > log4j.logger.org.apache.kafka=INFO > log4j.logger.org.apache.hadoop=INFO > log4j.logger.org.apache.zookeeper=INFO > > # Log all infos in the given file > log4j.appender.file=org.apache.log4j.FileAppender > log4j.appender.file.file=D:\\logs\\flink-log > log4j.appender.file.append=false > log4j.appender.file.layout=org.apache.log4j.PatternLayout > log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} > %-5p %-60c %x - %m%n > > # Suppress the irrelevant (wrong) warnings from the Netty channel handler > log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.ch > annel.DefaultChannelPipeline=ERROR, file > > > #BonusPointAppender > log4j.appender.bonuspointAppender=org.apache.log4j.RollingFileAppender > log4j.appender.bonuspointAppender.MaxFileSize=1024MB > log4j.appender.bonuspointAppender.MaxBackupIndex=10 > log4j.appender.bonuspointAppender.Append=true > log4j.appender.bonuspointAppender.File=D:\\logs\\flink-bpuser-bonus.logs > #log4j.appender.bonuspointAppender.DatePattern='.'yyyy-MM-dd > log4j.appender.bonuspointAppender.layout=org.apache.log4j.PatternLayout > log4j.appender.bonuspointAppender.layout.ConversionPattern=%d [%t] %-5p > (%C %M:%L) %x - %m%n > > #AMSSourceAppender > log4j.appender.amssourceAppender=org.apache.log4j.RollingFileAppender > log4j.appender.amssourceAppender.MaxFileSize=1024MB > log4j.appender.amssourceAppender.MaxBackupIndex=10 > log4j.appender.amssourceAppender.Append=true > log4j.appender.amssourceAppender.File=D:\\logs\\flink- > bpuser-bonus-amssource.logs > #log4j.appender.amssourceAppender.DatePattern='.'yyyy-MM-dd > log4j.appender.amssourceAppender.layout=org.apache.log4j.PatternLayout > log4j.appender.amssourceAppender.layout.ConversionPattern=%d [%t] %-5p > (%C %M:%L) %x - %m%n > > > > > -- > *Cheers * > > *Puneet Kinra* > > *Mobile:+918800167808 <+91%2088001%2067808> | Skype : > puneet.ki...@customercentria.com <puneet.ki...@customercentria.com>* > > *e-mail :puneet.ki...@customercentria.com > <puneet.ki...@customercentria.com>* > > >
