Hi Flink Community,
I am trying enable a company internal logging framework built upon SLF4J and
log4j. This logging framework has another separate jar and specific logging
configurations. After debugging, I am able to make Flink application running
correctly in the local IDE with the internal logging framework after adding
related SLF4J, log4j dependencies, and logging framework dependencies.
However, I still run into errors when I deploy this into the Kubernetes
environment. I tried to add the logging framework jar to /opt/flink/lib/
folder, but it doesn’t help much. I am not sure which part I am missing here. I
have attached relevant information below. Thanks for your help.
This is the log4j2-console.properties I proposed, I have injected this as a
configmap (mounted to /opt/flink/conf inside the pod using a Flink native
Kubernetes Operator I build).
Such configuration will run correctly in Local IDE and generate logs in the
internal logging framework expected shape. (I have rename it to
log4j2.properties and put it into resources/ folder during local debug.)
packages = oracle.spectra.logging.base
status = WARN
monitorInterval = 30
shutdownHook = disable
rootLogger.level = ${sys:spectra-log-level:-INFO}
rootLogger.appenderRef.asyncC.ref = AsyncCAppender
rootLogger.appenderRef.asyncF.ref = AsyncFAppender
appender.asyncC.name = AsyncCAppender
appender.asyncC.type = Async
appender.asyncC.bufferSize = 256
appender.asyncC.appenderRef.type = AppenderRef
appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
# Log all infos to the console
appender.console.name = JSONLogConsoleAppender
appender.console.target = SYSTEM_OUT
appender.console.type = Console
appender.console.layout.type = SpectraJsonLayout
appender.console.layout.compact = true
appender.console.layout.eventEol = true
appender.asyncF.name = AsyncFAppender
appender.asyncF.type = Async
appender.asyncF.bufferSize = 256
appender.asyncF.appenderRef.type = AppenderRef
appender.asyncF.appenderRef.ref = RollingFileAppender
# Log all infos in the given rolling file
appender.rolling.type = RollingFile
appender.rolling.name = RollingFileAppender
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = SpectraJsonLayout
appender.rolling.layout.compact = false
appender.rolling.layout.eventEol = true
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name =
org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
This is the error I got from the Job Manager pod in the Kubernetes.
sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only file
system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only file
system
/docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml: Read-only file
system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
Starting kubernetes-application as a console application on host
faw-poc-demo-67864b696b-9zbbc.
2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for
SpectraJsonLayout
2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for
SpectraJsonLayout
2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,689 main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.RollingFileAppender for element
RollingFile: java.lang.NullPointerException java.lang.NullPointerException
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in class
org.apache.logging.log4j.core.appender.RollingFileAppender for element
RollingFile: java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.RollingFileAppender
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.RollingFileAppender
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,758 main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.ConsoleAppender for element Console:
java.lang.NullPointerException java.lang.NullPointerException
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in class
org.apache.logging.log4j.core.appender.ConsoleAppender for element Console:
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.ConsoleAppender
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.ConsoleAppender
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,766 main ERROR Null object returned for RollingFile in
Appenders.
2021-09-24 23:42:53,766 main ERROR Null object returned for Console in
Appenders.
2021-09-24 23:42:53,771 main ERROR No appender named RollingFileAppender was
configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No
appenders are available for AsyncAppender AsyncFAppender
at
org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
As you can see, the SpectraJsonLayout that is defined in the logging framework
jar is somehow not recognized. I am not sure how to solve this? Could anyone
share some information. Really appreciate your help!
Best regards,
Fuyao
