Hi Marcelo, I found the problem from http://mail-archives.apache.org/mod_mbox/spark-user/201409.mbox/%3cCAL+LEBfzzjugOoB2iFFdz_=9TQsH=DaiKY=cvydfydg3ac5...@mail.gmail.com%3e this link. The problem is the application I am running, is not generating "APPLICATION_COMPLETE" file. If I add this file manually it is showing application in the UI. So the problem is with application which is not calling Stop method on the spark context.
Thank you and Todd for helping. Hopefully I will be able to apply these on the actual cluster. Regards, Srini. On Wed, Mar 4, 2015 at 10:20 AM, Srini Karri <skarri....@gmail.com> wrote: > Yes. I do see files, actually I missed copying the other settings: > > spark.master spark:// > skarri-lt05.redmond.corp.microsoft.com:7077 > spark.eventLog.enabled true > spark.rdd.compress true > spark.storage.memoryFraction 1 > spark.core.connection.ack.wait.timeout 6000 > spark.akka.frameSize 50 > spark.executor.extraClassPath > D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes > spark.eventLog.dir > D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events > spark.history.fs.logDirectory > D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events > > > On Wed, Mar 4, 2015 at 10:15 AM, Marcelo Vanzin <van...@cloudera.com> > wrote: > >> On Wed, Mar 4, 2015 at 10:08 AM, Srini Karri <skarri....@gmail.com> >> wrote: >> > spark.executor.extraClassPath >> > >> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes >> > spark.eventLog.dir >> > >> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events >> > spark.history.fs.logDirectory >> > >> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events >> >> Do you see any files in that directory? >> >> spark.eventLog.dir won't do anything unless you also have >> spark.eventLog.enabled=true somewhere. And these are application >> configs, so make sure they're set when running your application (not >> when starting the history server). >> >> -- >> Marcelo >> > >
