Hi Ted & Nguyen, @Ted , I was under the belief that if the log4j.properties file would be taken from the application classpath if file path is not specified. Please correct me if I am wrong. I tried your approach as well still I couldn't find the logs.
@nguyen I am running it on a Yarn cluster , so Spark UI is redirecting me to Yarn UI. I couldn't see the logs there as well. I checked the logs on both Master and worker. I am running a cluster with one master and one worker. Even I tired yarn logs there also its not turning up. Does yarn logs include executor logs as well ? Request your help to identify the issue . On Fri, Apr 29, 2016 at 7:32 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Please use the following syntax: > > --conf > > "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///local/file/log4j.properties" > > FYI > > On Fri, Apr 29, 2016 at 6:03 AM, dev loper <spark...@gmail.com> wrote: > >> Hi Spark Team, >> >> I have asked the same question on stack overflow , no luck yet. >> >> >> http://stackoverflow.com/questions/36923949/where-to-find-logs-within-spark-rdd-processing-function-yarn-cluster-mode?noredirect=1#comment61419406_36923949 >> >> I am running my Spark Application on Yarn Cluster. No matter what I do, I >> am not able to get the logs within the RDD function printed . Below you can >> find the sample snippet which I have written for the RDD processing >> function . I have simplified the code to illustrate the syntax I have used >> to write the function. When I am running it locally I am able to see the >> logs but not in cluster mode. Neither System.err.println nor the logger >> seems to be working. But I could see all my driver logs. I even tried to >> log using the Root logger , but it was not working at all within the RDD >> processing function .I was desperate to see the log messages so finally I >> found a guide to use logger as transient ( >> https://www.mapr.com/blog/how-log-apache-spark) ,but event that didn't >> help >> >> class SampleFlatMapFunction implements PairFlatMapFunction >> <Tuple2<String,String>,String,String>{ >> >> private static final long serialVersionUID = 6565656322667L; >> transient Logger executorLogger = LogManager.getLogger("sparkExecutor"); >> >> >> private void readObject(java.io.ObjectInputStream in) >> throws IOException, ClassNotFoundException { >> in.defaultReadObject(); >> executorLogger = LogManager.getLogger("sparkExecutor"); >> } >> @Override >> public Iterable<Tuple2<String,String>> call(Tuple2<String, String> >> tuple) throws Exception { >> >> executorLogger.info(" log testing from executorLogger ::"); >> System.err.println(" log testing from executorLogger system error >> stream "); >> >> >> List<Tuple2<String, String>> updates = new ArrayList<>(); >> //process Tuple , expand and add it to list. >> return updates; >> >> } >> }; >> >> My Log4j Configuration is given below >> >> log4j.appender.console=org.apache.log4j.ConsoleAppender >> log4j.appender.console.target=System.err >> log4j.appender.console.layout=org.apache.log4j.PatternLayout >> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p >> %c{1}: %m%n >> >> log4j.appender.stdout=org.apache.log4j.ConsoleAppender >> log4j.appender.stdout.target=System.out >> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout >> log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p >> %c{1}: %m%n >> >> log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender >> log4j.appender.RollingAppender.File=/var/log/spark/spark.log >> log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd >> log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout >> log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - >> %m%n >> >> log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender >> >> log4j.appender.RollingAppenderU.File=${spark.yarn.app.container.log.dir}/spark-app.log >> log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd >> log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout >> log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - >> %m%n >> >> >> # By default, everything goes to console and file >> log4j.rootLogger=INFO, RollingAppender, console >> >> # My custom logging goes to another file >> log4j.logger.sparkExecutor=INFO, stdout, RollingAppenderU >> >> >> i have tried yarn logs, Spark UI Logs nowhere I could see the log >> statements from RDD processing functions . I tried below Approaches but it >> didn't work >> >> yarn logs -applicationId >> >> I checked even below HDFS path also >> >> /tmp/logs/ >> >> >> I am running my spark-submit command by passing below arguments, Even >> then its not working >> >> --master yarn --deploy-mode cluster --conf >> "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" >> --conf >> "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" >> >> Can somebody guide me on logging within spark RDD and map functions ? >> What am I missing in the above steps ? >> >> Thanks >> >> Dev >> > >
