Re: Stopping a job

2020-06-09 Thread Kostas Kloudas
; *From: *Kostas Kloudas > *Date: *Monday, June 8, 2020 at 2:35 AM > *To: *Arvid Heise > *Cc: *M Singh , User-Flink > *Subject: *Re: Stopping a job > > > > What Arvid said is correct. > > The only thing I have to add is that "stop" allows also exactly-once sink

Re: Stopping a job

2020-06-08 Thread M Singh
Cc: M Singh , User-Flink Subject: Re: Stopping a job   What Arvid said is correct.  The only thing I have to add is that "stop" allows also exactly-once sinks to push out their buffered data to their final destination (e.g. Filesystem). In other words, it takes into account si

Re: Stopping a job

2020-06-08 Thread Senthil Kumar
: Kostas Kloudas Date: Monday, June 8, 2020 at 2:35 AM To: Arvid Heise Cc: M Singh , User-Flink Subject: Re: Stopping a job What Arvid said is correct. The only thing I have to add is that "stop" allows also exactly-once sinks to push out their buffered data to their final destin

Re: Stopping a job

2020-06-08 Thread Kostas Kloudas
What Arvid said is correct. The only thing I have to add is that "stop" allows also exactly-once sinks to push out their buffered data to their final destination (e.g. Filesystem). In other words, it takes into account side-effects, so it guarantees exactly-once end-to-end, assuming that you are us

Re: Stopping a job

2020-06-08 Thread Arvid Heise
It was before I joined the dev team, so the following are kind of speculative: The concept of stoppable functions never really took off as it was a bit of a clumsy approach. There is no fundamental difference between stopping and cancelling on (sub)task level. Indeed if you look in the twitter sou

Re: Stopping a job

2020-06-06 Thread M Singh
Hi Arvid:    Thanks for the links.   A few questions: 1. Is there any particular interface in 1.9+ that identifies the source as stoppable ?2. Is there any distinction b/w stop and cancel  in 1.9+ ?3. Is there any list of sources which are documented as stoppable besides the one listed in your

Re: Stopping a job

2020-06-06 Thread Arvid Heise
Yes, it seems as if FlinkKinesisConsumer does not implement it. Here are the links to the respective javadoc [1] and code [2]. Note that in later releases (1.9+) this interface has been removed. Stop is now implemented through a cancel() on source level. In general, I don't think that in a Kinesi

Re: Stopping a job

2020-06-06 Thread M Singh
Hi Arvid: I check the link and it indicates that only Storm SpoutSource, TwitterSource and NifiSource support stop.    Does this mean that FlinkKinesisConsumer is not stoppable ? Also, can you please point me to the Stoppable interface mentioned in the link ?  I found the following but am not su

Re: Stopping a job

2020-06-05 Thread Arvid Heise
Hi, could you check if this SO thread [1] helps you already? [1] https://stackoverflow.com/questions/53735318/flink-how-to-solve-error-this-job-is-not-stoppable On Thu, Jun 4, 2020 at 7:43 PM M Singh wrote: > Hi: > > I am running a job which consumes data from Kinesis and send data to > anothe

Stopping a job

2020-06-04 Thread M Singh
Hi: I am running a job which consumes data from Kinesis and send data to another Kinesis queue.  I am using an older version of Flink (1.6), and when I try to stop the job I get an exception  Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.rest.util.RestClientExc