Hi, Spark streaming does not seem to clear MapPartitionsRDD and ShuffledRDD that are persisted after the use of updateStateByKey and reduceByKeyAndWindow with inverse functions even after checkpointing the data. Any idea as to why thing happens? Is there a way that I can set a time out to clear the persisted data after a while? It seems to be not clearing the cached MapPartitionsRDD and ShuffledRDD even after I explicitly call unpersist and also do the checkpointing.
Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-does-not-seem-to-clear-MapPartitionsRDD-and-ShuffledRDD-that-are-persisted-after-thea-tp28850.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
