Doesn't iter still need to fit entirely into memory?
On Wed, Feb 11, 2015 at 5:55 PM, Mark Hamstra <m...@clearstorydata.com>
wrote:
> rdd.mapPartitions { iter =>
> val grouped = iter.grouped(batchSize)
> for (group <- grouped) { ... }
> }
>
> On Wed, Feb 11, 2015 at 2:44 PM, Corey Nolet <cjno...@gmail.com> wrote:
>
>> I think the word "partition" here is a tad different than the term
>> "partition" that we use in Spark. Basically, I want something similar to
>> Guava's Iterables.partition [1], that is, If I have an RDD[People] and I
>> want to run an algorithm that can be optimized by working on 30 people at a
>> time, I'd like to be able to say:
>>
>> val rdd: RDD[People] = .....
>> val partitioned: RDD[Seq[People]] = rdd.partition(30)....
>>
>> I also don't want any shuffling- everything can still be processed
>> locally.
>>
>>
>> [1]
>> http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/Iterables.html#partition(java.lang.Iterable,%20int)
>>
>
>
