Byte code, no. It's sufficient to store the information that the RDD represents, which can include serialized function closures, but that's not quite storing byte code.
On Wed, Aug 24, 2016 at 2:00 AM, kant kodali <kanth...@gmail.com> wrote: > Hi Guys, > > I have this question for a very long time and after diving into the source > code(specifically from the links below) I have a feeling that the lineage > of an RDD (the transformations) are converted into byte code and stored in > memory or disk. or if I were to ask another question on a similar note do > we ever store JVM byte code or python byte code in memory or disk? This > make sense to me because if we were to construct an RDD after a node > failure we need to go through the lineage and execute the respective > transformations so storing their byte codes does make sense however many > people seem to disagree with me so it would be great if someone can clarify. > > https://github.com/apache/spark/blob/6ee40d2cc5f467c78be662c1639fc3 > d5b7f796cf/python/pyspark/rdd.py#L1452 > > https://github.com/apache/spark/blob/6ee40d2cc5f467c78be662c1639fc3 > d5b7f796cf/python/pyspark/rdd.py#L1471 > > https://github.com/apache/spark/blob/6ee40d2cc5f467c78be662c1639fc3 > d5b7f796cf/python/pyspark/rdd.py#L229 > > https://github.com/apache/spark/blob/master/python/ > pyspark/cloudpickle.py#L241 >
