Update:
You can now answer this on stackoverflow for 100 bounty:
http://stackoverflow.com/questions/33704073/how-to-send-transformed-data-from-partitions-to-s3
On Fri, Nov 13, 2015 at 4:56 PM, Walrus theCat
wrote:
> Hi,
>
> I have an RDD which crashes the driver when being collected. I want t
Maybe you want to be using rdd.saveAsTextFile() ?
> On Nov 13, 2015, at 4:56 PM, Walrus theCat wrote:
>
> Hi,
>
> I have an RDD which crashes the driver when being collected. I want to send
> the data on its partitions out to S3 without bringing it back to the driver.
> I try calling rdd.for
It would be easier if you could show some code.
Regards
Sab
On 14-Nov-2015 6:26 am, "Walrus theCat" wrote:
> Hi,
>
> I have an RDD which crashes the driver when being collected. I want to
> send the data on its partitions out to S3 without bringing it back to the
> driver. I try calling rdd.for
Hi Walrus,
Try caching the results just before calling the rdd.count.
Regards,
Ajay
> On Nov 13, 2015, at 7:56 PM, Walrus theCat wrote:
>
> Hi,
>
> I have an RDD which crashes the driver when being collected. I want to send
> the data on its partitions out to S3 without bringing it back to
Hi,
I have an RDD which crashes the driver when being collected. I want to
send the data on its partitions out to S3 without bringing it back to the
driver. I try calling rdd.foreachPartition, but the data that gets sent has
not gone through the chain of transformations that I need. It's the dat
