This is a bug. Could you make a JIRA? -Xiangrui On Sat, Nov 15, 2014 at 3:27 AM, lev <kat...@gmail.com> wrote: > Hi, > > I'm having trouble using both zipWithIndex and repartition. When I use them > both, the following action will get stuck and won't return. > I'm using spark 1.1.0. > > > Those 2 lines work as expected: > > scala> sc.parallelize(1 to 10).repartition(10).count() > res0: Long = 10 > > scala> sc.parallelize(1 to 10).zipWithIndex.count() > res1: Long = 10 > > > But this statement get stuck and doesn't return: > > scala> sc.parallelize(1 to 10).zipWithIndex.repartition(10).count() > 14/11/15 03:18:55 INFO spark.SparkContext: Starting job: apply at > Option.scala:120 > 14/11/15 03:18:55 INFO scheduler.DAGScheduler: Got job 3 (apply at > Option.scala:120) with 3 output partitions (allowLocal=false) > 14/11/15 03:18:55 INFO scheduler.DAGScheduler: Final stage: Stage 4(apply at > Option.scala:120) > 14/11/15 03:18:55 INFO scheduler.DAGScheduler: Parents of final stage: > List() > 14/11/15 03:18:55 INFO scheduler.DAGScheduler: Missing parents: List() > 14/11/15 03:18:55 INFO scheduler.DAGScheduler: Submitting Stage 4 > (ParallelCollectionRDD[7] at parallelize at <console>:13), which has no > missing parents > 14/11/15 03:18:55 INFO storage.MemoryStore: ensureFreeSpace(1096) called > with curMem=7616, maxMem=138938941 > 14/11/15 03:18:55 INFO storage.MemoryStore: Block broadcast_4 stored as > values in memory (estimated size 1096.0 B, free 132.5 MB) > > > Am I doing something wrong here or is it a bug? > Is there some work around? > > Thanks, > Lev. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/repartition-combined-with-zipWithIndex-get-stuck-tp18999.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
