Assume in my executor I say
SparkConf sparkConf = new SparkConf();
sparkConf.set("spark.kryo.registrator",
"com.lordjoe.distributed.hydra.HydraKryoSerializer");
sparkConf.set("mysparc.data", "Some user Data");
sparkConf.setAppName("Some App");
Now
1) Are there default values set in some system file which are populated
if I call new SparkConf - if not how do I get those? _ I think i see
defaults foe the master, the Serializer...
2) If I set a property in SparkConf for my SparkContext will I see that
property in a Slave machine?
3) If I set a property anf then call showSparkProperties() do I see
that property set and if not how can I see the property set - say in
another thread as in
if in some other thread on the executor say as in
showSparkPropertiesInAnotherThread();
4) How can a slave machine access properties set on the executor
I an really interested in sparkConf.set("spark.kryo.registrator",
"com.lordjoe.distributed.hydra.HydraKryoSerializer");
which needs to be used by the Slave
/**
* dump all spark properties to System.err
*/
public static void showSparkProperties()
{
SparkConf sparkConf = new SparkConf();
Tuple2<String, String>[] all = sparkConf.getAll();
for (Tuple2<String, String> prp : all) {
System.err.println(prp._1().toString() + "=" + prp._2());
}
}
public static void showSparkPropertiesInAnotherThread()
{
new Thread(new Runnable() {
@Override
public void run() {
showSparkProperties();
}
}).start();
}