Running a custom "setup task" on all workers

he real work. This seems a bit "hacky" to me - any clever ideas on how to do this differently? Much obliged, *Noam Barcay* Developer // *Kenshoo* *Office* +972 3 746-6500 *427 // *Mobile* +972 54 475-3142 __ *www.Kenshoo.com* <http://kenshoo.com/&g

Re: sparkcontext.objectFile return thousands of partitions

maybe each of the file parts has many blocks? did you try SparkContext.coalesce to reduce the number of partitions? can be done w/ or w/o data-shuffle. *Noam Barcay* Developer // *Kenshoo* *Office* +972 3 746-6500 *427 // *Mobile* +972 54 475-3142

possible memory leak when re-creating SparkContext's in the same JVM

id see examples of Java applications re-creating SparkContex's - for example, Ooyala's spark-jobserver - so I would imagine this is possible, no? https://github.com/ooyala/spark-jobserver/blob/master/job-server/src/spark.jobserver/JobManagerActor.scala#L104 Thanks, *Noam Barcay* Deve