Hi Anastasios, Thanks for your reply. If I just increase the numPartitions to be twice larger, how coalesce(numPartitions: Int, shuffle: Boolean = false) keeps the data locality? Do I need to define my own Partitioner?
Thanks, Fei On Sun, Jan 15, 2017 at 3:58 AM, Anastasios Zouzias <zouz...@gmail.com> wrote: > Hi Fei, > > How you tried coalesce(numPartitions: Int, shuffle: Boolean = false) ? > > https://github.com/apache/spark/blob/branch-1.6/core/ > src/main/scala/org/apache/spark/rdd/RDD.scala#L395 > > coalesce is mostly used for reducing the number of partitions before > writing to HDFS, but it might still be a narrow dependency (satisfying your > requirements) if you increase the # of partitions. > > Best, > Anastasios > > On Sun, Jan 15, 2017 at 12:58 AM, Fei Hu <hufe...@gmail.com> wrote: > >> Dear all, >> >> I want to equally divide a RDD partition into two partitions. That means, >> the first half of elements in the partition will create a new partition, >> and the second half of elements in the partition will generate another new >> partition. But the two new partitions are required to be at the same node >> with their parent partition, which can help get high data locality. >> >> Is there anyone who knows how to implement it or any hints for it? >> >> Thanks in advance, >> Fei >> >> > > > -- > -- Anastasios Zouzias > <a...@zurich.ibm.com> >
