Hi Demi, Thanks for sharing.
What we usually do is let the driver read the configuration for the job and pass the config object to the actual job as a serializable object. That way avoids the need of a centralized config sharing point that needs to be accessed from the workers. as you have defined in your solution. We use chef to write the configuration for the job in the environment it belongs to (dev, prod,...) when the job gets deployed to a host node. That config file is used to instantiate the job. We can maintain any number of different environments in that way. kr, Gerard. On Fri, Dec 12, 2014 at 6:38 PM, Demi Ben-Ari <demi.ben...@gmail.com> wrote: > > Hi to all, > > Our problem was passing configuration from Spark Driver to the Slaves. > After a lot of time spent figuring out how things work, this is the > solution I came up with. > Hope this will be helpful for others as well. > > > You can read about it in my Blog Post > <http://progexc.blogspot.co.il/2014/12/spark-configuration-mess-solved.html> > > -- > Enjoy, > Demi Ben-Ari > Senior Software Engineer > Windward LTD. >
