Syed, Thanks for the tip. I'm not sure if coalesce is doing what I'm intending to do, which is, in effect, to subdivide the RDD into N parts (by calling coalesce and doing operations on the partitions.) It sounds like, however, this won't bottleneck my processing power. If this sets off any alarms for anyone, feel free to chime in.
On Mon, Mar 24, 2014 at 2:50 PM, Syed A. Hashmi <shas...@cloudera.com>wrote: > RDD.coalesce should be fine for rebalancing data across all RDD > partitions. Coalesce is pretty handy in situations where you have sparse > data and want to compact it (e.g. data after applying a strict filter) OR > you know the magic number of partitions according to your cluster which > will be optimal. > > One point to watch out though is that if N is greater than your current > partitions, you need to pass shuffle=true to coalesce. If N is less than > your current partitions (i.e. you are shrinking partitions, do not set > shuffle=true, otherwise it will cause additional unnecessary shuffle > overhead. > > > On Mon, Mar 24, 2014 at 2:32 PM, Walrus theCat <walrusthe...@gmail.com>wrote: > >> For instance, I need to work with an RDD in terms of N parts. Will >> calling RDD.coalesce(N) possibly cause processing bottlenecks? >> >> >> On Mon, Mar 24, 2014 at 1:28 PM, Walrus theCat <walrusthe...@gmail.com>wrote: >> >>> Hi, >>> >>> Quick question about partitions. If my RDD is partitioned into 5 >>> partitions, does that mean that I am constraining it to exist on at most 5 >>> machines? >>> >>> Thanks >>> >> >> >
