Hey,

We are deploying our Flink Cluster on a standalone Kubernetes with the
longrunning job written in scala.

We recently upgraded our Flink cluster from 1.12 to 1.14.3 - after which we
started seeing a few problems related to logging which I have been
struggling to fix for the past days).
Related is also an attempt to add, we are also attempting to add a Sentry
integration for our error logs.

PROBLEM 1 - Error logs not being sent to Sentry.
We are bundling our code and dependencies into a FAT jar, which includes a
log4j2.xml specifying the Sentry Appender. But if I understand the
documentation
<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/advanced/logging/#configuring-log4j-2>
correctly our log4j2.xml won't be picked up by Flink as it already defines
a set of default logging configurations files (e.g. log4j and logback).

Q: How does Flink resolve logging configurations to use?

I can see the following JVM override params provided when running in our
dockerized version locally.

-Dlog.file=/opt/flink/log/flink--taskexecutor-0-thoros-taskmanager-6
b9785d4df-c28n4.log
2022-02-15 10:01:59,826 INFO org.apache.flink.runtime.taskexecutor.
TaskManagerRunner [] - -Dlog4j
.configuration=file:/opt/flink/conf/log4j-console.properties
2022-02-15 10:01:59,827 INFO org.apache.flink.runtime.taskexecutor.
TaskManagerRunner [] - -Dlog4j
.configurationFile=file:/opt/flink/conf/log4j-console.properties
2022-02-15 10:01:59,830 INFO org.apache.flink.runtime.taskexecutor.
TaskManagerRunner [] - -Dlogback
.configurationFile=file:/opt/flink/conf/logback-console.xml

Content of the log4j2.xml (path: src/main/resources):

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn"
packages="org.apache.logging.log4j.core,io.sentry.log4j2">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level
%logger{36} - %msg%n"/>
        </Console>
        <Sentry name="Sentry"
                dsn="<DSN>"
                minimumEventLevel="ERROR"
        />
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Sentry"/>
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>


For our kubernetes deployment we have followed the reference example here
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions
.
My assumption is that I would need to also provide the Sentry-related
configuration to the "log4-console.properties" for it to be picked up by
the Taskmanager and JobManager?


PROBLEM 2:
ERROR StatusLogger Log4j2 could not find a logging implementation.
Please add log4j-core to the classpath. Using SimpleLogger to log to the
console

I am not sure what's going on here. Following dependencies are bundled with
the FAT jar

"com.typesafe.scala-logging" %% "scala-logging" % scalaLoggingVersion,
"org.slf4j" % "slf4j-api" % "1.7.33",
"org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.17.0",
"org.apache.logging.log4j" % "log4j-core" % "2.17.0",
"org.apache.logging.log4j" %% "log4j-api-scala" % "12.0",
"io.sentry" % "sentry-log4j2" % "5.6.0",

Confused about what is going on here, possible this might not be Flink
related matter but I am not sure..any tips on how to best debug this would
be much appreciated.
-- 
*Thanks,*
*Jonas*

Reply via email to