Greetings All !
I needed some help in obtaining the application logs but I am really
confused where it's currently located. Please allow me to explain my
problem:
1. I am running the Spark application (written in Java) in a Hortonworks
Data Platform Hadoop cluster
2. My spark-submit command is the following:
spark-submit --class com.ibm.spark.sparkkafkamapper.ServiceMapperMain
--master yarn --packages
org.apache.kafka:kafka-clients:0.10.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.0
--deploy-mode client --files
job-configuration.properties,log4j.properties,/usr/hdp/current/spark2-client/conf/hive-site.xml
./spark-job.jar --settings=job-configuration.properties
log4j-spark.properties --conf
"spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
--conf
"spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
3. Content of the log4.properties is the following:
# Set everything to be logged to the console
log4j.rootCategory=ALL, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{1}: %m%n
# Set the default spark-shell log level to WARN. When running the
spark-shell, the
# log level for this class is used to overwrite the root logger's log
level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ALL
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up
nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
#Any custom class debug
log4j.logger.com.vmeg.code=DEBUG
4. Inside the spark application, I am having plenty of logger functions and
one example is like this:
logger.info("No dataset recieved for this batch");
Question: My question is where is the application log files being written ?
I have checked in the yarn logs but couldn't find the messages I have
written in the java file.
Request your help please as I am little confused and know that there is
something very silly which I am missing.
Thanks in advance !
Debu
