Hi Users,
I am setting spark configuration in below way;
val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()
spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")
import spark.implicits._
and submitting spark job with spark - submit. but none of the above
configuration is
getting reflected to the job, I have checked at Spark-UI.
I know setting up like this while creation of spark object, it's working well.
val spark = SparkSession.builder().appName(APP_NAME)
.config("spark.network.timeout", "1500s")
.config("spark.broadcast.compress", "true")
.config("spark.sql.broadcastTimeout", "36000")
.getOrCreate()
import spark.implicits._
Can someone please throw light?
