Re: spark config params conventions

2014-03-14 Thread Chester Chen
Based on typesafe config maintainer's response, with latest version of typeconfig, the double quote is no longer needed for key like spark.speculation, so you don't need code to strip the quotes Chester Alpine data labs Sent from my iPhone On Mar 12, 2014, at 2:50 PM, Aaron Davidson wrote:

Re: spark config params conventions

2014-03-12 Thread Aaron Davidson
One solution for typesafe config is to use "spark.speculation" = true Typesafe will recognize the key as a string rather than a path, so the name will actually be "\"spark.speculation\"", so you need to handle this contingency when passing the config operations to spark (stripping the quotes from

Re: spark config params conventions

2014-03-12 Thread Mark Hamstra
That's the whole reason why some of the intended configuration changes were backed out just before the 0.9.0 release. It's a well-known issue, even if a completely satisfactory solution isn't as well-known and is probably something which should do another iteration on. On Wed, Mar 12, 2014 at 9: