Hey, We are deploying our Flink Cluster on a standalone Kubernetes with the longrunning job written in scala.
We recently upgraded our Flink cluster from 1.12 to 1.14.3 - after which we started seeing a few problems related to logging which I have been struggling to fix for the past days). Related is also an attempt to add, we are also attempting to add a Sentry integration for our error logs. PROBLEM 1 - Error logs not being sent to Sentry. We are bundling our code and dependencies into a FAT jar, which includes a log4j2.xml specifying the Sentry Appender. But if I understand the documentation <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/advanced/logging/#configuring-log4j-2> correctly our log4j2.xml won't be picked up by Flink as it already defines a set of default logging configurations files (e.g. log4j and logback). Q: How does Flink resolve logging configurations to use? I can see the following JVM override params provided when running in our dockerized version locally. -Dlog.file=/opt/flink/log/flink--taskexecutor-0-thoros-taskmanager-6 b9785d4df-c28n4.log 2022-02-15 10:01:59,826 INFO org.apache.flink.runtime.taskexecutor. TaskManagerRunner [] - -Dlog4j .configuration=file:/opt/flink/conf/log4j-console.properties 2022-02-15 10:01:59,827 INFO org.apache.flink.runtime.taskexecutor. TaskManagerRunner [] - -Dlog4j .configurationFile=file:/opt/flink/conf/log4j-console.properties 2022-02-15 10:01:59,830 INFO org.apache.flink.runtime.taskexecutor. TaskManagerRunner [] - -Dlogback .configurationFile=file:/opt/flink/conf/logback-console.xml Content of the log4j2.xml (path: src/main/resources): <?xml version="1.0" encoding="UTF-8"?> <Configuration status="warn" packages="org.apache.logging.log4j.core,io.sentry.log4j2"> <Appenders> <Console name="Console" target="SYSTEM_OUT"> <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/> </Console> <Sentry name="Sentry" dsn="<DSN>" minimumEventLevel="ERROR" /> </Appenders> <Loggers> <Root level="info"> <AppenderRef ref="Sentry"/> <AppenderRef ref="Console"/> </Root> </Loggers> </Configuration> For our kubernetes deployment we have followed the reference example here https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions . My assumption is that I would need to also provide the Sentry-related configuration to the "log4-console.properties" for it to be picked up by the Taskmanager and JobManager? PROBLEM 2: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console I am not sure what's going on here. Following dependencies are bundled with the FAT jar "com.typesafe.scala-logging" %% "scala-logging" % scalaLoggingVersion, "org.slf4j" % "slf4j-api" % "1.7.33", "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.17.0", "org.apache.logging.log4j" % "log4j-core" % "2.17.0", "org.apache.logging.log4j" %% "log4j-api-scala" % "12.0", "io.sentry" % "sentry-log4j2" % "5.6.0", Confused about what is going on here, possible this might not be Flink related matter but I am not sure..any tips on how to best debug this would be much appreciated. -- *Thanks,* *Jonas*
