Hey Rishitesh,
Thats perfect thanks so much! Dont know why i didnt think of using
mapPartitions like this
Thanks,
Jem
On Fri, Aug 28, 2015 at 10:35 AM Rishitesh Mishra <rishi80.mis...@gmail.com>
wrote:
> Hi Jem,
> A simple way to get this is to use MapPartitionedRDD. Please see the below
> code. For this you need to know your parent RDD's partition numbers that
> you want to exclude. One drawback here is the new RDD will also invoke
> similar number of tasks as parent RDDs as both the RDDs have same number of
> partitions. We only be excluding the results from certain partitions. If
> you can live with that , then its OK.
>
> val ones = sc.makeRDD(1 to 100, 10).map(x => x) // base RDD
>
> // Reduced RDD
> val reduced = ones.mapPartitions { iter => {
>
> new Iterator[Int](){
> override def hasNext: Boolean = {
> if(Seq(0,1,2).contains(TaskContext.get().partitionId)) {
> false
> } else{
> iter.hasNext
> }
> }
>
> override def next():Int = iter.next()
> }
>
> }
> }.collect().foreach(println)
>
>
>
>
> On Fri, Aug 28, 2015 at 12:33 PM, Jem Tucker <jem.tuc...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to create an RDD from a selected number of its parents
>> partitions. My current approach is to create my own SelectedPartitionRDD
>> and implement compute and numPartitions myself, problem is the compute
>> method is marked as @developerApi, and hence unsuitable for me to be using
>> in my application. Are there any alternative methods that will only use the
>> stable parts of the spark API?
>>
>> Thanks,
>>
>> Jem
>>
>
>
