java.lang.NullPointerException

[Mina Aslani]

Hi,

I get
java.lang.NullPointerException at
org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:128)

When I try to createDataFrame using the sparkSession, see below:

SparkConf conf = new SparkConf().setMaster(....).setAppName("test");
conf.set("spark.driver.allowMultipleContexts", "true");

//Read messages in batch of duration seconds
JavaStreamingContext jssc = new JavaStreamingContext(conf,
Durations.seconds(1));

SparkSession spark = SparkSession.builder().config(conf).getOrCreate();

List<Row> data = Arrays.asList(
        RowFactory.create(val1, val2, val3)
);
StructType schema = new StructType(new StructField[]{
        createStructField("field1", StringType, false),
        createStructField("field2", StringType, false),
        createStructField("field3", StringType, false)
});


Dataset<Row> df = spark.createDataFrame(data, schema);


Any idea? I am using JAVA therefore,I cannot convert my RDD to DF(). T

Therefore, I try to get specific field/values and createDataFrame manually.


Regards,

Mina