When processing data, I create an instance of RDD[Iterable[MyCaseClass]] and I
want to convert it to RDD[MyCaseClass] so that it can be further converted to
dataset or dataframe with toDS() function. But I encounter a problem that
SparkContext can not be instantiated within SparkSession.map function because
it already exists, even with allowMultipleContexts set to true.
val sc = new SparkConf()
sc.set("spark.driver.allowMultipleContexts", "true")
new SparkContext(sc).parallelize(seq)
How can I fix this?
Thanks.
