Re: choose the number of partition according to the number of nodes

Thank you Nicholas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/choose-the-number-of-partition-according-to-the-number-of-nodes-tp4362p4364.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: choose the number of partition according to the number of nodes

>From the Spark tuning guide : In general, we recommend 2-3 tasks per CPU core in your cluster. I think you can only get one task per partition to run concurrently for a given RDD. So if your RDD has 10 partitions, then 10 tasks at most can operat