This is the SPARK JIRA which introduced the warning: [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties in spark-shell and spark-submit
On Fri, Jun 12, 2015 at 4:34 PM, Peng Cheng <[email protected]> wrote: > Hi Andrew, > > Thanks a lot! Indeed, it doesn't start with spark, the following > properties are read by implementation of the driver rather than spark conf: > > --conf spooky.root=s3n://spooky- \ > --conf spooky.checkpoint=s3://spooky-checkpoint \ > > This used to work from Spark 1.0.0 to 1.3.1. Do you know the new way to > set the same properties? > > Yours Peng > > On 12 June 2015 at 14:20, Andrew Or <[email protected]> wrote: > >> Hi Peng, >> >> Setting properties through --conf should still work in Spark 1.4. From >> the warning it looks like the config you are trying to set does not start >> with the prefix "spark.". What is the config that you are trying to set? >> >> -Andrew >> >> 2015-06-12 11:17 GMT-07:00 Peng Cheng <[email protected]>: >> >>> In Spark <1.3.x, the system property of the driver can be set by --conf >>> option, shared between setting spark properties and system properties. >>> >>> In Spark 1.4.0 this feature is removed, the driver instead log the >>> following >>> warning: >>> >>> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv >>> >>> How do set driver's system property in 1.4.0? Is there a reason it is >>> removed without a deprecation warning? >>> >>> Thanks a lot for your advices. >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >
