Re: spark config params conventions

2014-03-14 Thread Chester Chen
Based on typesafe config maintainer's response, with latest version of typeconfig, the double quote is no longer needed for key like spark.speculation, so you don't need code to strip the quotes Chester Alpine data labs Sent from my iPhone On Mar 12, 2014, at 2:50 PM, Aaron Davidson wrote:

Re: spark config params conventions

2014-03-12 Thread Aaron Davidson
One solution for typesafe config is to use "spark.speculation" = true Typesafe will recognize the key as a string rather than a path, so the name will actually be "\"spark.speculation\"", so you need to handle this contingency when passing the config operations to spark (stripping the quotes from

Re: spark config params conventions

2014-03-12 Thread yao
+1. I agree to keep the old ones only for backward compatibility purpose. On Wed, Mar 12, 2014 at 12:38 PM, Evan Chan wrote: > +1. > > Not just for Typesafe Config, but if we want to consider hierarchical > configs like JSON rather than flat key mappings, it is necessary. It > is also clearer.

Re: spark config params conventions

2014-03-12 Thread Evan Chan
+1. Not just for Typesafe Config, but if we want to consider hierarchical configs like JSON rather than flat key mappings, it is necessary. It is also clearer. On Wed, Mar 12, 2014 at 9:58 AM, Aaron Davidson wrote: > Should we try to deprecate these types of configs for 1.0.0? We can start > by

Re: spark config params conventions

2014-03-12 Thread Aaron Davidson
Should we try to deprecate these types of configs for 1.0.0? We can start by accepting both and giving a warning if you use the old one, and then actually remove them in the next minor release. I think "spark.speculation.enabled=true" is better than "spark.speculation=true", and if we decide to use