Thanks for the suggestion.
1. Heartbeat:
As a matter of fact, the heartbeat solution is what I thought of as well.
However that needs to be outside spark-streaming.
Furthermore, it cannot be generalized to all spark applications. For, e.g.
I am doing several filtering operations before I reach th
There is a heartbeat stream pattern that you can use: Create a service
(perhaps a thread in your driver) that pushes a heartbeat event to a
different stream every N seconds. Consume that stream as well in your
streaming application, and perform an action on every heartbeat.
This has worked well in
Hi,
In general in spark stream one can do transformations ( filter, map etc.)
or output operations (collect, forEach) etc. in an event-driven pardigm...
i.e. the action happens only if a message is received.
Is it possible to do actions every few seconds in a polling based fashion,
regardless if a
