Your logs are getting archived in your logs bucket in S3. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html
Regards Sab On Mon, Feb 22, 2016 at 12:14 PM, HARSH TAKKAR <takkarha...@gmail.com> wrote: > Hi > > In am using an EMR cluster for running my spark jobs, but after the job > finishes logs disappear, > > I have added a log4j.properties in my jar, but all the logs still > redirects to EMR resource manager which vanishes after jobs completes, is > there a way i could redirect the logs to a location in file syatem, I am > working on price points and its very critical for me to maintain logs. > > Just to add i get following error when my application starts. > > java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No such file > or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:146) > at java.io.FileInputStream.<init>(FileInputStream.java:101) > at > sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) > at > sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) > at > org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) > at > org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) > at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) > at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122) > at > org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107) > at org.apache.spark.Logging$class.log(Logging.scala:51) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621) > at > org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) > > > -- Architect - Big Data Ph: +91 99805 99458 Manthan Systems | *Company of the year - Analytics (2014 Frost and Sullivan India ICT)* +++
