Hi all,
Hope you can shed some light on this matter since I've been trying to
workaround the issue for some time already but I can't make it click.
I've been trying to go from Spark 3.3.0 up to 3.5.1 and logs are currently
a mess. Using log4j2 fails when loading the file and still using the
log4j.properties shows weird characters (logback?) when deployed in
Kubernetes, not in local tho, so it seems like a packaging issue.
As info prior reading the output logs:* log4j2.properties and
log4j.properties are found within the project fat jar, under the /resources
directory*. I only fit one within the folder at a time so they don't clash.
- *Running Spark via spark-operator fails loading log4j2.properties file
with the following trace when running via spark-operator and locally via
IDE*:
ERROR StatusConsoleListener Unable to locate plugin type for
JsonTemplateLayout
ERROR StatusConsoleListener Unable to locate plugin for JsonTemplateLayout
ERROR StatusConsoleListener Could not create plugin of type class
org.apache.logging.log4j.core.appender.ConsoleAppender for element Console:
java.lang.NullPointerException
ERROR StatusConsoleListener Unable to invoke factory method in class
org.apache.logging.log4j.core.appender.ConsoleAppender for element Console
ERROR StatusConsoleListener Null object returned for Console in Appenders.
ERROR StatusConsoleListener Unable to locate appender "console" for logger
config "root"
- Preserving the old log4j.properties file till now works running locally
and logs are OK as previous with the old Spark 3.3.0*, but when launched
via spark-operator, weird logs appear (I think it may be falling back to
logback?) since the following chars are found*:
[]wEx%.S.S.S.BlockManagerMasterEndpointlr
%.%.%.%.BlockManagerMasterEndpointlr %.%.%.%.BlockManagerMasterEndpointlr
-.-.-.-.BlockManagerMasterEndpointlr
[.a5.s5.s5.BlockManagerMasterEndpointlr
org.apache.spark.storage.BlockManagerMasterEndpointlr%.apac0.}.}.BlockManagerMasterEndpointlr
:.:.:.:.BlockManagerMasterEndpointlr Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology
information
Some info about dependencies and their versions in case it may be a
conflict of versions:
org.apache.spark %% spark-core % 3.5.1 % "provided"
org.apache.spark %% spark-sql % 3.5.1 % "provided"
org.apache.spark %% spark-mllib % 3.5.1 % "provided"
org.apache.hadoop % hadoop-common % 3.3.4 % "provided"
org.apache.hadoop % hadoop-aws % 3.3.4
com.fasterxml.jackson.core % jackson-core % 2.15.2
com.fasterxml.jackson.core % jackson-databind % 2.15.2
com.fasterxml.jackson.core % jackson-annotations % 2.15.2
org.apache.logging.log4j % log4j-1.2-api % 2.20.0
org.apache.logging.log4j % log4j-api % 2.20.0
org.apache.logging.log4j % log4j-core % 2.20.0
org.apache.logging.log4j % log4j-slf4j2-impl % 2.20.0
org.slf4j % jcl-over-slf4j % 2.0.13
org.slf4j % jul-to-slf4j % 2.0.13
org.slf4j % slf4j-api % 2.0.13
I've also got Spring Boot within the module due to some stuff but the
logging packages are excluded so they don't interfere.
org.springframework.boot % spring-boot-starter % 3.3.0
exclude("org.springframework.boot", "spring-boot-starter-logging")
And now, how the newest resources/log4j2.properties file looks like:
rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console
appender.console.type = Consoleappender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = JsonTemplateLayout
appender.console.layout.eventTemplateUri =
classpath:org/apache/spark/SparkLayout.json
# Set the default spark-shell/spark-sql log level to WARN. When running the
# spark-shell/spark-sql, the log level for these classes is used to overwrite
# the root logger's log level, so that the user can have different defaults
# for the shell and regular Spark apps.logger.repl.name =
org.apache.spark.repl.Main
logger.repl.level = warn
logger.thriftserver.name =
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
logger.thriftserver.level = warn
# Settings to quiet third party logs that are too
verboselogger.jetty1.name = org.sparkproject.jetty
logger.jetty1.level = warnlogger.jetty2.name =
org.sparkproject.jetty.util.component.AbstractLifeCycle
logger.jetty2.level = errorlogger.replexprTyper.name =
org.apache.spark.repl.SparkIMain$exprTyper
logger.replexprTyper.level = infologger.replSparkILoopInterpreter.name
= org.apache.spark.repl.SparkILoop$SparkILoopInterpreter
logger.replSparkILoopInterpreter.level = infologger.parquet1.name =
org.apache.parquet
logger.parquet1.level = errorlogger.parquet2.name = parquet
logger.parquet2.level = error
# SPARK-9183: Settings to avoid annoying messages when looking up
nonexistent UDFs in SparkSQL with Hive
supportlogger.RetryingHMSHandler.name =
org.apache.hadoop.hive.metastore.RetryingHMSHandler
logger.RetryingHMSHandler.level = fatallogger.FunctionRegistry.name =
org.apache.hadoop.hive.ql.exec.FunctionRegistry
logger.FunctionRegistry.level = error
# For deploying Spark ThriftServer
# SPARK-34128: Suppress undesirable TTransportException warnings
involved in THRIFT-4805
appender.console.filter.1.type = RegexFilter
appender.console.filter.1.regex = .*Thrift error occurred during
processing of message.*
appender.console.filter.1.onMatch = deny
appender.console.filter.1.onMismatch = neutral
