Spark migration from 2.3 to 3.0.1

[Shrikant Prasad]

Hi,

I am trying to migrate one spark application from Spark 2.3 to 3.0.1.

The issue can be reproduced using below sample code:

object TestMain {

val session =
SparkSession.builder().appName("test").enableHiveSupport().getOrCreate()

def main(args: Array[String]): Unit = {

import session.implicits._
val a = *session.*sparkContext.parallelize(*Array*(("A",1),("B",2))).toDF("_
c1","_c2").*rdd*.map(x=> x(0).toString).collect()
*println*(a.mkString("|"))

}
}

It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below
exception:

Caused by: org.apache.spark.SparkException: A master URL must be set in
your configuration

                at
org.apache.spark.SparkContext.<init>(SparkContext.scala:394)

                at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)

                at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)

                at scala.Option.getOrElse(Option.scala:189)

                at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)

                at TestMain$.<init>(TestMain.scala:7)

                at TestMain$.<clinit>(TestMain.scala)


>From the exception it appears that it tries to create spark session on
executor also in Spark 3 whereas its not created again on executor in Spark
2.3.

Can anyone help in identfying why there is this change in behavior?

Thanks and Regards,

Shrikant

-- 
Regards,
Shrikant Prasad