Ran into an issue setting a property on the SparkConf that wasn't made
available on the worker. After some digging[1] I noticed that only
properties that start with "spark." are sent by the schedular. I'm not
sure if this was intended behavior or not.
Using Spark Streaming 1.4.1 running on Java
Thanks for the response.
In this particular case we passed a url that would be leveraged when
configuring some serialization support for Kryo. We are using a schema
registry and leveraging it to efficiently serialize avro objects without the
need to register specific records or schemas up front.
That works.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/possible-bug-user-SparkConf-properties-not-copied-to-worker-process-tp13665p13689.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
-
