Currently the only way to pause it is to stop it. The way I would do this is use the Direct Kafka API to access the Kafka offsets, and save them to a data store as batches finish. If you see a batch job failing because downstream is down, stop the context. When it comes back up, start a new streaming context using the saved offsets as starting points.
TD On Wed, Jul 8, 2015 at 11:52 AM, Shushant Arora <shushantaror...@gmail.com> wrote: > Is it possible to pause and resume a streaming app? > > I have a streaming app which reads events from kafka and post to some > external source. I want to pause the app when external source is down and > resume it automatically when it comes back ? > > Is it possible to pause the app and is it possible to pause only > processing part let receiver work and brong data in memory+disk, once > external source is up process whole data in one go. > > Thanks > Shushant >
