Re: Recreate Dataset from list of Row in spark streaming application.

2018-10-06 Thread Kuttaiah Robin
Thanks Ryan. Yes. That is ForeachWriter. Can someone help me on how to solve this?. Basically the output of flatMapGroupsWithState function is Dataset sessionUpdates ; InsightEventUpdate class contains list of Spark Row which I need to convert back to Dataset. Something like Dataset corr

Re: Recreate Dataset from list of Row in spark streaming application.

2018-10-05 Thread Shixiong(Ryan) Zhu
oracle.insight.spark.identifier_processor.ForeachFederatedEventProcessor is a ForeachWriter. Right? You can not use SparkSession in its process method as it will run in executors. Best Regards, Ryan On Fri, Oct 5, 2018 at 6:54 AM Kuttaiah Robin wrote: > Hello, > > I have a spark streaming appl

Recreate Dataset from list of Row in spark streaming application.

2018-10-05 Thread Kuttaiah Robin
Hello, I have a spark streaming application which reads from Kafka based on the given schema. Dataset m_oKafkaEvents = getSparkSession().readStream().format("kafka") .option("kafka.bootstrap.servers", strKafkaAddress) .option("assign", strSubscription) .option