It was not intended to be experimental as this improves general performance. We tested the feature since 0.9, and didnt see any problems. We need to investigate the cause of this. Can you give us the logs showing this error so that we can analyze this.
TD On Tue, Jun 3, 2014 at 10:08 AM, Michael Chang <[email protected]> wrote: > Thanks Tathagata, > > Thanks for all your hard work! In the future, is it possible to mark > "experimental" features as such on the online documentation? > > Thanks, > Michael > > > On Mon, Jun 2, 2014 at 6:12 PM, Tathagata Das <[email protected] > > wrote: > >> Spark.streaming.unpersist was an experimental feature introduced with >> Spark 0.9 (but kept disabled), which actively clears off RDDs that are not >> useful any more. in Spark 1.0 that has been enabled by default. It is >> possible that this is an unintended side-effect of that. If >> spark.cleaner.ttl works then that should be used. >> >> TD >> >> >> On Mon, Jun 2, 2014 at 9:42 AM, Michael Chang <[email protected]> wrote: >> >>> Hey Mayur, >>> >>> Thanks for the suggestion, I didn't realize that was configurable. I >>> don't think I'm running out of memory, though it does seem like these >>> errors go away when i turn off the spark.streaming.unpersist configuration >>> and use spark.cleaner.ttl instead. Do you know if there are known issues >>> with the unpersist option? >>> >>> >>> On Sat, May 31, 2014 at 12:17 AM, Mayur Rustagi <[email protected] >>> > wrote: >>> >>>> You can increase your akka timeout, should give you some more life.. >>>> are you running out of memory by any chance? >>>> >>>> >>>> Mayur Rustagi >>>> Ph: +1 (760) 203 3257 >>>> http://www.sigmoidanalytics.com >>>> @mayur_rustagi <https://twitter.com/mayur_rustagi> >>>> >>>> >>>> >>>> On Sat, May 31, 2014 at 6:52 AM, Michael Chang <[email protected]> >>>> wrote: >>>> >>>>> I'm running a some kafka streaming spark contexts (on 0.9.1), and they >>>>> seem to be dying after 10 or so minutes with a lot of these errors. I >>>>> can't really tell what's going on here, except that maybe the driver is >>>>> unresponsive somehow? Has anyone seen this before? >>>>> >>>>> 14/05/31 01:13:30 ERROR BlockManagerMaster: Failed to remove RDD 12635 >>>>> >>>>> akka.pattern.AskTimeoutException: Timed out >>>>> >>>>> at >>>>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334) >>>>> >>>>> at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118) >>>>> >>>>> at >>>>> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:691) >>>>> >>>>> at >>>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:688) >>>>> >>>>> at >>>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455) >>>>> >>>>> at >>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407) >>>>> >>>>> at >>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411) >>>>> >>>>> at >>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363) >>>>> >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> >>>>> Thanks, >>>>> >>>>> Mike >>>>> >>>>> >>>>> >>>> >>> >> >
