Found this thread: http://search-hadoop.com/m/JW1q5HMrge2
Cheers On Fri, Mar 6, 2015 at 6:42 AM, Sean Owen <so...@cloudera.com> wrote: > This was discussed in the past and viewed as dangerous to enable. The > biggest problem, by far, comes when you have a job that output M > partitions, 'overwriting' a directory of data containing N > M old > partitions. You suddenly have a mix of new and old data. > > It doesn't match Hadoop's semantics either, which won't let you do > this. You can of course simply remove the output directory. > > On Fri, Mar 6, 2015 at 2:20 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > Adding support for overwrite flag would make saveAsXXFile more user > friendly. > > > > Cheers > > > > > > > >> On Mar 6, 2015, at 2:14 AM, Jeff Zhang <zjf...@gmail.com> wrote: > >> > >> Hi folks, > >> > >> I found that RDD:saveXXFile has no overwrite flag which I think is very > helpful. Is there any reason for this ? > >> > >> > >> > >> -- > >> Best Regards > >> > >> Jeff Zhang > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > >
