Thanks Sabarish, I thought of same. will try that.
Hi Ted, good question. I guess one way is to have an api like
`rdd.persist(storageLevel,
compress)` where 'compress' can be true or false.
On Tue, Mar 15, 2016 at 5:18 PM, Sabarish Sasidharan wrote:
> It will compress only rdds with serializati
It will compress only rdds with serialization enabled in the persistence
mode. So you could skip _SER modes for your other rdds. Not perfect but
something.
On 15-Mar-2016 4:33 pm, "Nirav Patel" wrote:
> Hi,
>
> I see that there's following spark config to compress an RDD. My guess is
> it will c
Looks like there is no such capability yet.
How would you specify which rdd's to compress ?
Thanks
> On Mar 15, 2016, at 4:03 AM, Nirav Patel wrote:
>
> Hi,
>
> I see that there's following spark config to compress an RDD. My guess is it
> will compress all RDDs of a given SparkContext, ri
