In an event driven architecture you have to perform an action to stop the
event in a controlled manner.
That ( say) streamingContext.stop() or for that matter
streamingContext.stop(stopGracedully=True) won't cause any stop by itself. *You
need to build in a mechanism *to tell spark streaming to pr
What is it that stop(stopGraceFully=True) does then?
https://spark.apache.org/docs/latest/api/python/_modules/pyspark/streaming/context.html#StreamingContext.stop
ons. 8. feb. 2023 kl. 19:22 skrev Brian Wylie :
> It's been a few years (so this approach might be out of date) but here's
> what I us
It's been a few years (so this approach might be out of date) but here's
what I used for PySpark as part of this SO (
https://stackoverflow.com/questions/45717433/stop-structured-streaming-query-gracefully/65708677
)
```
# Helper method to stop a streaming query
def stop_stream_query(query, wait_
That is a valid question and I am not aware of any new addition to Spark
Structured Streaming (SSS) in newer releases for this graceful shutdown.
Going back to my earlier explanation, there are occasions that you may want
to stop the Spark program gracefully. Gracefully meaning that Spark
applicat
What are some other "newer" methodologies?
Really interested to understand what is possible here as this is a topic
came up in this forum time and again.
On Thu, 6 May 2021 at 5:13 pm, Gourav Sengupta <
gourav.sengupta.develo...@gmail.com> wrote:
> Hi Mich,
>
> thanks a ton for your kind respons
Hi Mich,
thanks a ton for your kind response, looks like we are still using the
earlier methodologies for stopping a spark streaming program gracefully.
Regards,
Gourav Sengupta
On Wed, May 5, 2021 at 6:04 PM Mich Talebzadeh
wrote:
>
> Hi,
>
>
> I believe I discussed this in this forum. I sen
Hi,
I believe I discussed this in this forum. I sent the following to spark-dev
forum as an add-on to Spark functionality. This is the gist of it.
Spark Structured Streaming AKA SSS is a very useful tool in dealing with
Event Driven Architecture. In an Event Driven Architecture, there is
genera
