No, but you can write a couple lines of code that do this. It's not
optimized of course. This is actually a long and interesting side
discussion, but I'm not sure how much it could be given that the
computation is "pull" rather than "push"; there is no concept of one
pass over the data resulting in many RDDs. However you can cache /
persist the source RDD to at least make sure it is not recomputed.

I don't think groupByKey is quite a solution since it means one RDD
for which all values for one key must fit in memory, and because the
desired output is an RDD, I am not sure that is suitable.

On Wed, Jun 10, 2015 at 1:56 PM, dgoldenberg <[email protected]> wrote:
> Hi,
>
> I'm gathering that the typical approach for splitting an RDD is to apply
> several filters to it.
>
> rdd1 = rdd.filter(func1);
> rdd2 = rdd.filter(func2);
> ...
>
> Is there/should there be a way to create 'buckets' like these in one go?
>
> List<RDD> rddList = rdd.filter(func1, func2, ..., funcN)
>
> Another angle here is, when applying a filter(func), is there a way to get
> two RDD's back, one for which func returned true for all elements of the
> original RDD (the one being filtered), and the other one for which func
> returned false for all the elements?
>
> Pair<RDD> pair = rdd.filterTrueFalse(func);
>
> Right now I'm doing
>
> RDD x = rdd.filter(func);
> RDD y = rdd.filter(reverseOfFunc);
>
> This seems a bit tautological to me, though Spark must be optimizing this
> out (?)
>
> Thanks.
>
>
>
>
>
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/Split-RDD-based-on-criteria-tp23254.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to