Map partitions works as follows : 1) For each partition of your RDD, it provides an iterator over the values within that partition 2) You then define a function that operates on that iterator
Thus if you do the following: val parallel = sc.parallelize(1 to 10, 3) parallel.mapPartitions( x => x.map(s => s + 1)).collect You would get: res3: Array[Int] = Array(2, 3, 4, 5, 6, 7, 8, 9, 10, 11) In your example, x is not a pointer that traverses the iterator (e.g. With .next()) , it¹s literally the Iterable object itself. On 3/18/15, 10:19 AM, "ashish.usoni" <[email protected]> wrote: >I am trying to understand about mapPartitions but i am still not sure how >it >works > >in the below example it create three partition >val parallel = sc.parallelize(1 to 10, 3) > >and when we do below >parallel.mapPartitions( x => List(x.next).iterator).collect > >it prints value >Array[Int] = Array(1, 4, 7) > >Can some one please explain why it prints 1,4,7 only > >Thanks, > > > > >-- >View this message in context: >http://apache-spark-user-list.1001560.n3.nabble.com/mapPartitions-How-Does >-it-Works-tp22123.html >Sent from the Apache Spark User List mailing list archive at Nabble.com. > >--------------------------------------------------------------------- >To unsubscribe, e-mail: [email protected] >For additional commands, e-mail: [email protected] > ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
