Spark Streaming RDD Cleanup too slow

I have a Spark Streaming job which takes too long to delete temp RDD's. I collect about 4MM telemetry metrics per minute and do minor aggregations in the Streaming Job. I am using Amazon R4 instances. The Driver RPC call although Async,i believe, is slow getting the handle for future object at "

Re: RDD Cleanup

Hi Prem, I am experiencing the same problem on Spark 1.0.2 and Job Server 0.4.0 Did you find a solution for this problem? Thank you, Hung -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cleanup-tp9182p16843.html Sent from the Apache Spark User List

Re: RDD Cleanup

Hi premdass, Where did you set spark.cleaner.referenceTracking = true/false? Was this in your job-server conf? Cheers. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cleanup-tp9182p10939.html Sent from the Apache Spark User List mailing list

Re: RDD Cleanup

w this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cleanup-tp9182p9191.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: RDD Cleanup

. Thanks, Prem -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cleanup-tp9182p9191.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: RDD Cleanup

sed under a context, event > though the job using that had been finished (so subsequent jobs can use > that > RDD). > > > Thanks, > Prem > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cleanup-tp9182.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

RDD Cleanup

though the job using that had been finished (so subsequent jobs can use that RDD). Thanks, Prem -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cleanup-tp9182.html Sent from the Apache Spark User List mailing list archive at Nabble.com.