Each handle to the RDD holds its lineage information, which means it knows how it was computed starting from data in a reliable storage or from other RDDs. RDDs hence can be reconstructed when the node fails.
Best Regards, Sonal Nube Technologies <http://www.nubetech.co> <http://in.linkedin.com/in/sonalgoyal> On Fri, Mar 28, 2014 at 3:55 PM, colt_colt <weihong9...@hotmail.com> wrote: > I am curious about Spark fail over scenario, if some executor down, that > means the JVM crashed. AM will restart the executor, but how about the RDD > data in JVM? if I didn't persist RDD, does Spark will recompute lost RDD > or > just let it lose? there is some description in Spark site: "Each RDD > remembers the lineage of deterministic operations that were used on a > fault-tolerant input dataset to create it." > > thanks in advance > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-does-Spark-handle-executor-down-RDD-in-this-executor-will-be-recomputed-automatically-tp3422.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
