I'm not sure about .union(), but at least in the case of .join(), as long as you have hash partitioned the original RDDs and persisted them, calls to .join() take advantage of already knowing which partition the keys are on, and will not repartition rdd1.
val rdd1 = log.partitionBy(new HashPartitioner(10)).persist() val rdd3 = rdd1.join(rdd2) I suspect you want to use one of the key aware operations anyways, rather than .union() I know other operations are also partitioner aware like this, though I don't know which ones. Perhaps use the partitioner property in order to test your operation? cheers, ds -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Insert-new-data-into-specific-partition-of-an-RDD-tp20156p20291.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
