I did not change driver program. I just shutdown the context and again started.
BTW, I see this ticket already open in unassigned state - SPARK-6892 <https://issues.apache.org/jira/browse/SPARK-6892> that talks about this issue. Is this a known issue? Also, any workarounds? On Wed, Jun 10, 2015 at 9:18 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Delete the checkpoint directory, you might have modified your driver > program. > > Thanks > Best Regards > > On Wed, Jun 10, 2015 at 9:44 PM, Ashish Nigam <ashnigamt...@gmail.com> > wrote: > >> Hi, >> If checkpoint data is already present in HDFS, driver fails to load as it >> is performing lookup on previous application directory. As that folder >> already exists, it fails to start context. >> Failed job's application id was application_1432284018452_0635 and job >> was performing lookup on application_1432284018452_0633 folder. >> >> Here's snippet of exception stack trace- >> >> 15/06/10 05:28:36 INFO yarn.ApplicationMaster: Final app status: FAILED, >> exitCode: 15, (reason: User class threw exception: Log directory >> hdfs://x.x.x.x:8020/user/myuser/spark/applicationHistory/application_1432284018452_0633 >> already exists!) >> >> Exception in thread "Driver" java.io.IOException: Log directory >> hdfs://x.x.x.x >> :8020/user/myuswer/spark/applicationHistory/application_1432284018452_0633 >> <http://172.16.201.171:8020/user/shn/spark/applicationHistory/application_1432284018452_0633> >> already exists! >> >> at >> org.apache.spark.util.FileLogger.createLogDir(FileLogger.scala:129) >> >> at org.apache.spark.util.FileLogger.start(FileLogger.scala:115) >> >> at >> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:74) >> >> at org.apache.spark.SparkContext.<init>(SparkContext.scala:368) >> >> at >> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:118) >> >> at >> org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:561) >> >> at >> org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:561) >> >> at scala.Option.map(Option.scala:145) >> >> at >> org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:561) >> >> at >> >> >> >> Any idea on how to fix this issue? >> >> >> Thanks >> >> Ashish >> > >
