In my code I have something like val rdd: (WrapWithComparable[(Array[Byte], Array[Byte], Array[Byte])], Externalizer[KeyValue]) = ... val rdd_coalesced = rdd.coalesce(Math.min(1000, rdd.partitions.length))
My purpose is to limit the number of partitions (later sortByKey always reported "too many open files" error). However, it won't compile, scala compiler complains "erroneous and inaccessible type". What's the problem? BTW, I found coalesce requires an implicit Ordering, why does it need that? I'm currently using repartition, it compiles fine, the doc says it always shuffles and recommends using coalesce for reducing partitions. Anyone can help me here? -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
