Hi DB Tsai,
Thanks very much for your kind reply!
Sorry that for one more issue, as tested it seems that filter could only return
JavaRDD<Boolean> but not any JavaRDD<T> , is it ?Then it is not much convenient
to do general filter for RDD, mapPartitions could work some, but if some
partition will left and return none element after filter by mapPartitions, some
problemwill be there.
Best Wishes!Zhiliang
On Saturday, December 5, 2015 3:00 PM, DB Tsai <[email protected]> wrote:
This is tricky. You need to shuffle the ending and beginning elements
using mapPartitionWithIndex.
Sincerely,
DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0xAF08DF8D
On Fri, Dec 4, 2015 at 10:30 PM, Zhiliang Zhu <[email protected]> wrote:
> Hi All,
>
> I would like to compare any two adjacent elements in one given rdd, just as
> the single machine code part:
>
> int a[N] = {...};
> for (int i=0; i < N - 1; ++i) {
> compareFun(a[i], a[i+1]);
> }
> ...
>
> mapPartitions may work for some situations, however, it could not compare
> elements in different partitions.
> foreach also seems not work.
>
> Thanks,
> Zhiliang
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
