Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Liana Napalkova
: Re: How to properly execute `foreachPartition` in Spark 2.2 Spark Dataset / Dataframe has foreachPartition() as well. Its implementation is much more efficient than RDD's. There is ton of code snippets, say https://github.com/hdinsight/spark-streaming-data-persistence-examples/blob/maste

Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Cody Koeninger
fka, then > writeStream to Kafka? > > > > > > *From: *Liana Napalkova > *Date: *Monday, December 18, 2017 at 10:07 AM > *To: *Silvio Fiorito , " > user@spark.apache.org" > > *Subject: *Re: How to properly execute `foreachPartition` in Spark 2.2 > > > &

Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Liana Napalkova
If there is no other way, then I will follow this recommendation. From: Silvio Fiorito Sent: 18 December 2017 16:20:03 To: Liana Napalkova; user@spark.apache.org Subject: Re: How to properly execute `foreachPartition` in Spark 2.2 Couldn’t you readStream from

Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Silvio Fiorito
Subject: Re: How to properly execute `foreachPartition` in Spark 2.2 I need to firstly read from Kafka queue into a DataFrame. Then I should perform some transformations with the data. Finally, for each row in the DataFrame I should conditionally apply KafkaProducer in order to send some dat

Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Timur Shenkao
g-queries > > > > > > > > *From: *Liana Napalkova > *Date: *Monday, December 18, 2017 at 9:45 AM > *To: *"user@spark.apache.org" > *Subject: *How to properly execute `foreachPartition` in Spark 2.2 > > > > Hi, > > > > I wonder how

Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Liana Napalkova
. From: Silvio Fiorito Sent: 18 December 2017 16:00:39 To: Liana Napalkova; user@spark.apache.org Subject: Re: How to properly execute `foreachPartition` in Spark 2.2 Why don’t you just use the Kafka sink for Spark 2.2? https://spark.apache.org/docs/2.2.0

Re: How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Silvio Fiorito
ct: How to properly execute `foreachPartition` in Spark 2.2 Hi, I wonder how to properly execute `foreachPartition` in Spark 2.2. Below I explain the problem is details. I appreciate any help. In Spark 1.6 I was doing something similar to this: DstreamFromKafka.foreachR

How to properly execute `foreachPartition` in Spark 2.2

2017-12-18 Thread Liana Napalkova
Hi, I wonder how to properly execute `foreachPartition` in Spark 2.2. Below I explain the problem is details. I appreciate any help. In Spark 1.6 I was doing something similar to this: DstreamFromKafka.foreachRDD(session => { session.foreachPartition { partitionOfReco