I solved the problem.
It was caused by using spark-core_2.11 mvn repository.
When I compiled with spark-core_2.10, the problem doesn't show up again.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/problem-with-using-mapPartitions-tp12514p12523.ht
Thank you for your reply.
But the typo is not reason for the problem.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/problem-with-using-mapPartitions-tp12514p12520.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Hi,
I try to aggregate the value in each partition internally.
For example,
Before:
worker 1:worker 2:
1, 2, 1 2, 1, 2
After:
worker 1: worker 2:
(1->2), (2->1) (1->1), (2->2)
I try to use mappartitions,
object MyTest {
