Thanks! I got your point. Will try it out. From: Chesnay Schepler <ches...@apache.org> Date: Tuesday, October 19, 2021 at 01:44 To: Fuyao Li <fuyao...@oracle.com>, user <user@flink.apache.org> Cc: Rohit Gupta <rohit.c.gu...@oracle.com> Subject: Re: [External] : Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes 1) Adding it as a dependency to the Flink application does not work with an actual Flink cluster, because said dependency must be available when the cluster is started. It works in the IDE because there everything is put onto the same classpath.
2) folder structure shouldn't be relevant. So long as it is in somewhere in the lib directory of the Flink distribution it should be picked up. 3) Put the jar into the lib directory, do not add the config map, start the cluster and check the logs. The classpath will be logged very early in the startup procedure. On 18/10/2021 22:52, Fuyao Li wrote: Hi Chesnay, Thanks for the reply. 1. The internal logging framework is built upon slf4j/log4j2 (The same one Flink uses, but it comes with an additional POM dependency). I have added such dependency in the Flink application POM file. But it seems only to work locally in IDE. When it is in the Flink cluster environment, it can’t work. 2. I tried to only add the configmap and put a single jar into lib/ folder, and it seems it still can’t find the classpath. How should I organize the folder structure? /lib/internal-logging/xxx.jar or this jar file must be directly under /lib, something like /lib/xxx.jar? 3. I got you point, I guess it is stilling using Flink default logging classpath and that causes the issue of not recognizing the internal framework? How to check the classpath of the Flink logging? Could you share me some blogs..? I am not familiar with this. Best, Fuyao From: Chesnay Schepler <ches...@apache.org><mailto:ches...@apache.org> Date: Tuesday, September 28, 2021 at 07:06 To: Fuyao Li <fuyao...@oracle.com><mailto:fuyao...@oracle.com>, user <user@flink.apache.org><mailto:user@flink.apache.org> Cc: Rohit Gupta <rohit.c.gu...@oracle.com><mailto:rohit.c.gu...@oracle.com> Subject: [External] : Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes Could you clarify whether this internal framework uses a custom slfj4/log4j2 version, or is it just using what Flink comes with? Did you only add the configmap and put a single jar into lib, or did you make other changes in Flink? Can you remove just the configmap, start the cluster, and provide us with the classpath that Flink is logging? On 25/09/2021 01:57, Fuyao Li wrote: Hi Flink Community, I am trying enable a company internal logging framework built upon SLF4J and log4j. This logging framework has another separate jar and specific logging configurations. After debugging, I am able to make Flink application running correctly in the local IDE with the internal logging framework after adding related SLF4J, log4j dependencies, and logging framework dependencies. However, I still run into errors when I deploy this into the Kubernetes environment. I tried to add the logging framework jar to /opt/flink/lib/ folder, but it doesn’t help much. I am not sure which part I am missing here. I have attached relevant information below. Thanks for your help. This is the log4j2-console.properties I proposed, I have injected this as a configmap (mounted to /opt/flink/conf inside the pod using a Flink native Kubernetes Operator I build). Such configuration will run correctly in Local IDE and generate logs in the internal logging framework expected shape. (I have rename it to log4j2.properties and put it into resources/ folder during local debug.) packages = oracle.spectra.logging.base status = WARN monitorInterval = 30 shutdownHook = disable rootLogger.level = ${sys:spectra-log-level:-INFO} rootLogger.appenderRef.asyncC.ref = AsyncCAppender rootLogger.appenderRef.asyncF.ref = AsyncFAppender appender.asyncC.name = AsyncCAppender appender.asyncC.type = Async appender.asyncC.bufferSize = 256 appender.asyncC.appenderRef.type = AppenderRef appender.asyncC.appenderRef.ref = JSONLogConsoleAppender # Log all infos to the console appender.console.name = JSONLogConsoleAppender appender.console.target = SYSTEM_OUT appender.console.type = Console appender.console.layout.type = SpectraJsonLayout appender.console.layout.compact = true appender.console.layout.eventEol = true appender.asyncF.name = AsyncFAppender appender.asyncF.type = Async appender.asyncF.bufferSize = 256 appender.asyncF.appenderRef.type = AppenderRef appender.asyncF.appenderRef.ref = RollingFileAppender # Log all infos in the given rolling file appender.rolling.type = RollingFile appender.rolling.name = RollingFileAppender appender.rolling.fileName = ${sys:log.file} appender.rolling.filePattern = ${sys:log.file}.%i appender.rolling.layout.type = SpectraJsonLayout appender.rolling.layout.compact = false appender.rolling.layout.eventEol = true appender.rolling.policies.type = Policies appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size=100MB appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 10 # Uncomment this if you want to _only_ change Flink's logging #logger.flink.name = org.apache.flink #logger.flink.level = INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. logger.akka.name = akka logger.akka.level = INFO logger.kafka.name= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper logger.zookeeper.level = INFO # Suppress the irrelevant (wrong) warnings from the Netty channel handler logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF This is the error I got from the Job Manager pod in the Kubernetes. sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only file system sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only file system /docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only file system /docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml: Read-only file system /docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system Starting kubernetes-application as a console application on host faw-poc-demo-67864b696b-9zbbc. 2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for SpectraJsonLayout 2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for SpectraJsonLayout 2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for SpectraJsonLayout 2021-09-24 23:42:53,689 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.NullPointerException java.lang.NullPointerException at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104) at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108) 2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108) 2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for SpectraJsonLayout 2021-09-24 23:42:53,758 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104) at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108) 2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108) 2021-09-24 23:42:53,766 main ERROR Null object returned for RollingFile in Appenders. 2021-09-24 23:42:53,766 main ERROR Null object returned for Console in Appenders. 2021-09-24 23:42:53,771 main ERROR No appender named RollingFileAppender was configured Exception in thread "main" java.lang.ExceptionInInitializerError Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No appenders are available for AsyncAppender AsyncFAppender at org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108) As you can see, the SpectraJsonLayout that is defined in the logging framework jar is somehow not recognized. I am not sure how to solve this? Could anyone share some information. Really appreciate your help! Best regards, Fuyao
