Re: Flink rolling upgrade support

2017-07-19 Thread Moiz Jinia
gt;> Thanks! >> Moiz >> >> >> >> -- >> View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-rolling-upgrade-support-tp10674p14313.html >> Sent from the Apache Flink User Mailing List archive. mailing list archive >> at Nabble.com. > > > > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-rolling-upgrade-support-tp10674p14337.html > To unsubscribe from Flink rolling upgrade support, click here. > NAML > > View this message in context: Re: Flink rolling upgrade support > > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Flink rolling upgrade support

2017-07-19 Thread Ted Yu
n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_inst

Re: Flink rolling upgrade support

2017-07-19 Thread Moiz Jinia
Yup! Thanks. Moiz — sent from phone On 19-Jul-2017, at 9:21 PM, Aljoscha Krettek [via Apache Flink User Mailing List archive.] wrote: This was now answered in your other Thread, right? Best, Aljoscha >> On 18. Jul 2017, at 11:37, Moiz Jinia <[hidden email]> wrote: >> >> Aljoscha Krettek wr

Re: Flink rolling upgrade support

2017-07-19 Thread Aljoscha Krettek
This was now answered in your other Thread, right? Best, Aljoscha > On 18. Jul 2017, at 11:37, Moiz Jinia wrote: > > Aljoscha Krettek wrote >> Hi, >> zero-downtime updates are currently not supported. What is supported in >> Flink right now is a savepoint-shutdown-restore cycle. With this, you

Re: Flink rolling upgrade support

2017-07-18 Thread Moiz Jinia
Aljoscha Krettek wrote > Hi, > zero-downtime updates are currently not supported. What is supported in > Flink right now is a savepoint-shutdown-restore cycle. With this, you > first > draw a savepoint (which is essentially a checkpoint with some meta data), > then you cancel your job, then you do

Re: Flink rolling upgrade support

2016-12-23 Thread Gyula Fóra
Hi! I think in many cases it is more convenient to have a savepoint-and-stop operation to use for upgrading the cluster/job but it should not be required. If the output of your job needs to be exactly once and you don't have an external deduplication mechanism than even the current fault-tolerance

Re: Flink rolling upgrade support

2016-12-23 Thread Aljoscha Krettek
Hi Greg, yes certainly, there are more requirements to this than the quick sketch I gave above and that seems to be one of them. Cheers, Aljoscha On Thu, 22 Dec 2016 at 17:54 Greg Hogan wrote: > Aljoscha, > > For the second, possible solution is there also a requirement that the > data sinks ha

Re: Flink rolling upgrade support

2016-12-22 Thread Greg Hogan
Aljoscha, For the second, possible solution is there also a requirement that the data sinks handle out-of-order writes? If the new job outpaces the old job which is then terminated, the final write from the old job could have overwritten "newer" writes from the new job. Greg On Tue, Dec 20, 2016

Re: Flink rolling upgrade support

2016-12-22 Thread Ron Crocker
Hi Stephan - I agree that the savepoint-shutdown-restart model is nominally the same as the rolling restart with one notable exception - a lack of atomicity. There is a gap between invoking the savepoint command and the shutdown command. My problem isn’t fortunate enough to have idempotent oper

Re: Flink rolling upgrade support

2016-12-20 Thread Stephan Ewen
Hi Andrew! Would be great to know if what Aljoscha described works for you. Ideally, this costs no more than a failure/recovery cycle, which one typically also gets with rolling upgrades. Best, Stephan On Tue, Dec 20, 2016 at 6:27 PM, Aljoscha Krettek wrote: > Hi, > zero-downtime updates are

Re: Flink rolling upgrade support

2016-12-20 Thread Aljoscha Krettek
Hi, zero-downtime updates are currently not supported. What is supported in Flink right now is a savepoint-shutdown-restore cycle. With this, you first draw a savepoint (which is essentially a checkpoint with some meta data), then you cancel your job, then you do whatever you need to do (update mac