: Re: How to properly execute `foreachPartition` in Spark 2.2
Spark Dataset / Dataframe has foreachPartition() as well. Its implementation is
much more efficient than RDD's.
There is ton of code snippets, say
https://github.com/hdinsight/spark-streaming-data-persistence-examples/blob/maste
fka, then
> writeStream to Kafka?
>
>
>
>
>
> *From: *Liana Napalkova
> *Date: *Monday, December 18, 2017 at 10:07 AM
> *To: *Silvio Fiorito , "
> user@spark.apache.org"
>
> *Subject: *Re: How to properly execute `foreachPartition` in Spark 2.2
>
>
>
&
If there is no other way, then I will follow this recommendation.
From: Silvio Fiorito
Sent: 18 December 2017 16:20:03
To: Liana Napalkova; user@spark.apache.org
Subject: Re: How to properly execute `foreachPartition` in Spark 2.2
Couldn’t you readStream from
Subject: Re: How to properly execute `foreachPartition` in Spark 2.2
I need to firstly read from Kafka queue into a DataFrame. Then I should perform
some transformations with the data. Finally, for each row in the DataFrame I
should conditionally apply KafkaProducer in order to send some dat
@spark.apache.org
> *Subject:* Re: How to properly execute `foreachPartition` in Spark 2.2
>
>
> Why don’t you just use the Kafka sink for Spark 2.2?
>
>
>
> https://spark.apache.org/docs/2.2.0/structured-streaming-
> kafka-integration.html#creating-a-kafka-sink-for-streamin
.
From: Silvio Fiorito
Sent: 18 December 2017 16:00:39
To: Liana Napalkova; user@spark.apache.org
Subject: Re: How to properly execute `foreachPartition` in Spark 2.2
Why don’t you just use the Kafka sink for Spark 2.2?
https://spark.apache.org/docs/2.2.0
Why don’t you just use the Kafka sink for Spark 2.2?
https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html#creating-a-kafka-sink-for-streaming-queries
From: Liana Napalkova
Date: Monday, December 18, 2017 at 9:45 AM
To: "user@spark.apache.org"
Subject: How to properl
