Great, thanks! On Tue, Mar 30, 2021 at 11:00 AM Till Rohrmann <trohrm...@apache.org> wrote:
> This is a good idea. I will add it to the section here [1]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#terminating-a-job > > Cheers, > Till > > On Tue, Mar 30, 2021 at 2:46 PM Vishal Santoshi <vishal.santo...@gmail.com> > wrote: > >> Got it. Is it possible to add this very important note to the >> documentation. Our case is the former as in this is an infinite pipeline >> and we were establishing the CiCD release process when non breaking >> changes ( DAG compatible changes are made ) on a running pipe. >> >> Regards >> >> On Tue, Mar 30, 2021 at 8:14 AM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >>> Hi Vishal, >>> >>> The difference between stop-with-savepoint and >>> stop-with-savepoint-with-drain is that the latter emits a max watermark >>> before taking the snapshot. The idea is to trigger all pending timers and >>> flush the content of some buffering operations like windowing. >>> Semantically, you should use the first option if you want to stop the job >>> and resume it at a later point in time. Stop-with-savepoint-with-drain >>> should only be used if you want to terminate your job and don't intend to >>> resume it because the max watermark destroys the correctness of results >>> which are generated after the job is resumed. >>> >>> For the concrete problem at hand it is difficult to say why it does not >>> stop. It would be helpful if you could provide us with the debug logs of >>> such a run. I am also pulling Arvid who works on Flink's connector >>> ecosystem. >>> >>> Cheers, >>> Till >>> >>> On Mon, Mar 29, 2021 at 11:08 PM Vishal Santoshi < >>> vishal.santo...@gmail.com> wrote: >>> >>>> More interested whether a StreamingFileSink without a drain >>>> negatively affects it's exactly-once semantics , given that I state on SP >>>> would have the offsets from kafka + the valid lengths of the part files at >>>> SP. To be honest not sure whether the flushed buffers on sink are included >>>> in the length, or this is not an issue with StreamingFileSink. If it is the >>>> former then I would assume we should be documented and then have to look >>>> why this hang happens. >>>> >>>> On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi < >>>> vishal.santo...@gmail.com> wrote: >>>> >>>>> Is this a known issue. We do a stop + savepoint with drain. I see no >>>>> back pressure on our operators. It essentially takes a SP and then the >>>>> SInk >>>>> ( StreamingFileSink to S3 ) just stays in the RUNNING state. >>>>> >>>>> Without drain i stop + savepoint works fine. I would imagine drain is >>>>> important ( flush the buffers etc ) but why this hang ( I did it 3 times >>>>> and waited 15 minutes each time ). >>>>> >>>>> Regards. >>>>> >>>>
