Fernando Pereira created SPARK-20580:
----------------------------------------
Summary: Allow RDD cache with unserializable objects
Key: SPARK-20580
URL: https://issues.apache.org/jira/browse/SPARK-20580
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.3.0
Reporter: Fernando Pereira
Priority: Minor
In my current scenario we load complex Python objects in the worker nodes that
are not completely serializable. We then apply map certain operations to the
RDD which at some point we collect. In this basic usage all works well.
However, if we cache() the RDD (which defaults to memory) suddenly it fails to
execute the transformations after the caching step. Apparently caching
serializes the RDD data and deserializes it whenever more transformations are
required.
It would be nice to avoid serialization of the objects if they are to be cached
to memory, and keep the original object
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
