Re: [Discuss][Spark staging dir] way to disable spark writing to _temporary

2017-04-08 Thread Steve Loughran
On 8 Apr 2017, at 10:36, Yash Sharma mailto:yash...@gmail.com>> wrote: Very interesting. I will give it a try. Thanks for pointing this. Also, are you planning to contribute it to spark, and could it be a good default option for spark S3 copies ? It's going into Hadoop core itself, HADOOP-137

Re: [Discuss][Spark staging dir] way to disable spark writing to _temporary

2017-04-08 Thread Yash Sharma
Very interesting. I will give it a try. Thanks for pointing this. Also, are you planning to contribute it to spark, and could it be a good default option for spark S3 copies ? Have you got any bench marking that could show the improvements in the job. Thanks, Yash On Sat, 8 Apr 2017 at 02:38 Ryan