Aureliano, Just to answer your second question (unrelated to Spark), arrays in java and scala can't be larger than the maximum value of an Integer (Integer.MAX_VALUE), which means that arrays are limited to about 2.2 billion elements.
--j On Wed, Mar 12, 2014 at 1:08 PM, Aureliano Buendia <buendia...@gmail.com>wrote: > Hi, > > I asked a similar question a while ago, didn't get any answers. > > I'd like to share a 10 gb double array between 50 to 100 workers. The > physical memory of workers is over 40 gb, so it can fit in each memory. The > reason I'm sharing this array is that a cartesian operation is applied to > this array, and I want to avoid network shuffling. > > 1. Is Spark broadcast built for pushing variables of gb size? Does it need > special configurations (eg akka config, etc) to work under this condition? > > 2. (Not directly related to spark) Is the an upper limit for scala/java > arrays other than the physical memory? Do they stop working when the array > elements count exceeds a certain number? >
