We are seeing below logs in production sometime ago, after that we stopped
HA. Do you people think HA is enabled properly from the below logs?
Regards
Bhaskar
2019-09-24 17:40:17,675 INFO
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService -
Starting ZooKeeperLeaderElectionService
ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2019-09-24 17:40:17,675 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
- Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2019-09-24 17:40:20,975 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
2019-09-24 17:40:20,976 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system [akka.tcp:
//fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123] has
failed, address is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123]]
Caused by: [No route to host]
2019-09-24 17:40:23,976 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
2019-09-24 17:40:23,977 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system [akka.tcp:
//fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123] has
failed, address is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123]]
Caused by: [No route to host]
2019-09-24 17:40:26,982 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
2019-09-24 17:40:26,983 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system [akka.tcp:
//fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123] has
failed, address is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123]]
Caused by: [No route to host]
2019-09-24 17:40:29,988 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
2019-09-24 17:40:29,988 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system [akka.tcp:
//fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123] has
failed, address is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123]]
Caused by: [No route to host]
2019-09-24 17:40:32,994 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
2019-09-24 17:40:32,995 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system [akka.tcp:
//fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123] has
failed, address is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123]]
Caused by: [No route to host]
2019-09-24 17:40:36,000 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
2019-09-24 17:40:36,001 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system [akka.tcp:
//fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123] has
failed, address is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://fl...@service-flink-jobmanager-1.cloudsecure.svc.cluster.local:6123]]
Caused by: [No route to host]
2019-09-24 17:40:39,006 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.NoRouteToHostException: No route to host
On Fri, Oct 11, 2019 at 9:39 AM Yun Tang <myas...@live.com> wrote:
> Hi Hao
>
> It seems that I misunderstood the background of usage for your cases. High
> availability configuration targets for fault tolerance not for general
> development evolution. If you want to change your job topology, just follow
> the general rule to restore from savepoint/checkpoint, do not rely on HA to
> do job migration things.
>
> Best
> Yun Tang
> ------------------------------
> *From:* Hao Sun <ha...@zendesk.com>
> *Sent:* Friday, October 11, 2019 8:33
> *To:* Yun Tang <myas...@live.com>
> *Cc:* Vijay Bhaskar <bhaskar.eba...@gmail.com>; Yang Wang <
> danrtsey...@gmail.com>; Sean Hester <sean.hes...@bettercloud.com>;
> Aleksandar Mastilovic <amastilo...@sightmachine.com>; Yuval Itzchakov <
> yuva...@gmail.com>; user <user@flink.apache.org>
> *Subject:* Re: Challenges Deploying Flink With Savepoints On Kubernetes
>
> Yep I know that option. That's where get me confused as well. In a HA
> setup, where do I supply this option (allowNonRestoredState)?
> This option requires a savepoint path when I start a flink job I remember.
> And HA does not require the path
>
> Hao Sun
>
>
> On Thu, Oct 10, 2019 at 11:16 AM Yun Tang <myas...@live.com> wrote:
>
> Just a minor supplement @Hao Sun <ha...@zendesk.com>, if you decided to
> drop a operator, don't forget to add --allowNonRestoredState (short: -n)
> option [1]
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#allowing-non-restored-state
>
> Best
> Yun Tang
>
> ------------------------------
> *From:* Vijay Bhaskar <bhaskar.eba...@gmail.com>
> *Sent:* Thursday, October 10, 2019 19:24
> *To:* Yang Wang <danrtsey...@gmail.com>
> *Cc:* Sean Hester <sean.hes...@bettercloud.com>; Aleksandar Mastilovic <
> amastilo...@sightmachine.com>; Yun Tang <myas...@live.com>; Hao Sun <
> ha...@zendesk.com>; Yuval Itzchakov <yuva...@gmail.com>; user <
> user@flink.apache.org>
> *Subject:* Re: Challenges Deploying Flink With Savepoints On Kubernetes
>
> Thanks Yang. We will try and let you know if any issues arise
>
> Regards
> Bhaskar
>
> On Thu, Oct 10, 2019 at 1:53 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
> @ Hao Sun,
> I have made a confirmation that even we change parallelism and/or modify
> operators, add new operators,
> the flink cluster could also recover from latest checkpoint.
>
> @ Vijay
> a) Some individual jobmanager/taskmanager crashed exceptionally(someother
> jobmanagers
> and taskmanagers are alive), it could recover from the latest checkpoint.
> b) All jobmanagers and taskmanagers fails, it could still recover from the
> latest checkpoint if the cluster-id
> is not changed.
>
> When we enable the HA, The meta of jobgraph and checkpoint is saved on
> zookeeper and the real files are save
> on high-availability storage(HDFS). So when the flink application is
> submitted again with same cluster-id, it could
> recover jobs and checkpoint from zookeeper. I think it has been supported
> for a long time. Maybe you could have a
> try with flink-1.8 or 1.9.
>
> Best,
> Yang
>
>
> Vijay Bhaskar <bhaskar.eba...@gmail.com> 于2019年10月10日周四 下午2:26写道：
>
> Thanks Yang and Sean. I have couple of questions:
>
> 1) Suppose the scenario of , bringing back entire cluster,
> a) In that case, at least one job manager out of HA group should be
> up and running right? or
> b) All the job managers fails, then also this works? In that case
> please let me know the procedure/share the documentation?
> How to start from previous check point?
> What Flink version onwards this feature is stable?
>
> Regards
> Bhaskar
>
>
> On Wed, Oct 9, 2019 at 8:51 AM Yang Wang <danrtsey...@gmail.com> wrote:
>
> Hi Vijay,
>
> If you are using HA solution, i think you do not need to specify the
> savepoint. Instead the checkpoint is used.
> The checkpoint is done automatically and periodically based on your
> configuration.When the
> jobmanager/taskmanager fails or the whole cluster crashes, it could always
> recover from the latest
> checkpoint. Does this meed your requirement?
>
> Best,
> Yang
>
> Sean Hester <sean.hes...@bettercloud.com> 于2019年10月1日周二 上午1:47写道：
>
> Vijay,
>
> That is my understanding as well: the HA solution only solves the problem
> up to the point all job managers fail/restart at the same time. That's
> where my original concern was.
>
> But to Aleksandar and Yun's point, running in HA with 2 or 3 Job Managers
> per cluster--as long as they are all deployed to separate GKE nodes--would
> provide a very high uptime/low failure rate, at least on paper. It's a
> promising enough option that we're going to run in HA for a month or two
> and monitor results before we put in any extra work to customize the
> savepoint start-up behavior.
>
> On Fri, Sep 27, 2019 at 2:24 AM Vijay Bhaskar <bhaskar.eba...@gmail.com>
> wrote:
>
> I don't think HA will help to recover from cluster crash, for that we
> should take periodic savepoint right? Please correct me in case i am wrong
>
> Regards
> Bhaskar
>
> On Fri, Sep 27, 2019 at 11:48 AM Vijay Bhaskar <bhaskar.eba...@gmail.com>
> wrote:
>
> Suppose my cluster got crashed and need to bring up the entire cluster
> back? Does HA still helps to run the cluster from latest save point?
>
> Regards
> Bhaskar
>
> On Thu, Sep 26, 2019 at 7:44 PM Sean Hester <sean.hes...@bettercloud.com>
> wrote:
>
> thanks to everyone for all the replies.
>
> i think the original concern here with "just" relying on the HA option is
> that there are some disaster recovery and data center migration use cases
> where the continuity of the job managers is difficult to preserve. but
> those are admittedly very edgy use cases. i think it's definitely worth
> reviewing the SLAs with our site reliability engineers to see how likely it
> would be to completely lose all job managers under an HA configuration.
> that small a risk might be acceptable/preferable to a one-off solution.
>
> @Aleksander, would love to learn more about Zookeeper-less HA. i think i
> spotted a thread somewhere between Till and someone (perhaps you) about
> that. feel free to DM me.
>
> thanks again to everyone!
>
> On Thu, Sep 26, 2019 at 7:32 AM Yang Wang <danrtsey...@gmail.com> wrote:
>
> Hi, Aleksandar
>
> Savepoint option in standalone job cluster is optional. If you want to
> always recover
> from the latest checkpoint, just as Aleksandar and Yun Tang said you could
> use the
> high-availability configuration. Make sure the cluster-id is not changed,
> i think the job
> could recover both at exceptionally crash and restart by expectation.
>
> @Aleksandar Mastilovic <amastilo...@sightmachine.com>, we are also have
> an zookeeper-less high-availability implementation[1].
> Maybe we could have some discussion and contribute this useful feature to
> the community.
>
> [1].
> https://docs.google.com/document/d/1Z-VdJlPPEQoWT1WLm5woM4y0bFep4FrgdJ9ipQuRv8g/edit
>
> Best,
> Yang
>
> Aleksandar Mastilovic <amastilo...@sightmachine.com> 于2019年9月26日周四
> 上午4:11写道：
>
> Would you guys (Flink devs) be interested in our solution for
> zookeeper-less HA? I could ask the managers how they feel about
> open-sourcing the improvement.
>
> On Sep 25, 2019, at 11:49 AM, Yun Tang <myas...@live.com> wrote:
>
> As Aleksandar said, k8s with HA configuration could solve your problem.
> There already have some discussion about how to implement such HA in k8s if
> we don't have a zookeeper service: FLINK-11105 [1] and FLINK-12884 [2].
> Currently, you might only have to choose zookeeper as high-availability
> service.
>
> [1] https://issues.apache.org/jira/browse/FLINK-11105
> [2] https://issues.apache.org/jira/browse/FLINK-12884
>
> Best
> Yun Tang
> ------------------------------
> *From:* Aleksandar Mastilovic <amastilo...@sightmachine.com>
> *Sent:* Thursday, September 26, 2019 1:57
> *To:* Sean Hester <sean.hes...@bettercloud.com>
> *Cc:* Hao Sun <ha...@zendesk.com>; Yuval Itzchakov <yuva...@gmail.com>;
> user <user@flink.apache.org>
> *Subject:* Re: Challenges Deploying Flink With Savepoints On Kubernetes
>
> Can’t you simply use JobManager in HA mode? It would pick up where it left
> off if you don’t provide a Savepoint.
>
> On Sep 25, 2019, at 6:07 AM, Sean Hester <sean.hes...@bettercloud.com>
> wrote:
>
> thanks for all replies! i'll definitely take a look at the Flink k8s
> Operator project.
>
> i'll try to restate the issue to clarify. this issue is specific to
> starting a job from a savepoint in job-cluster mode. in these cases the Job
> Manager container is configured to run a single Flink job at start-up. the
> savepoint needs to be provided as an argument to the entrypoint. the Flink
> documentation for this approach is here:
>
>
> https://github.com/apache/flink/tree/master/flink-container/kubernetes#resuming-from-a-savepoint
>
> the issue is that taking this approach means that the job will *always* start
> from the savepoint provided as the start argument in the Kubernetes YAML.
> this includes unplanned restarts of the job manager, but we'd really prefer
> any *unplanned* restarts resume for the most recent checkpoint instead of
> restarting from the configured savepoint. so in a sense we want the
> savepoint argument to be transient, only being used during the initial
> deployment, but this runs counter to the design of Kubernetes which always
> wants to restore a deployment to the "goal state" as defined in the YAML.
>
> i hope this helps. if you want more details please let me know, and thanks
> again for your time.
>
>
> On Tue, Sep 24, 2019 at 1:09 PM Hao Sun <ha...@zendesk.com> wrote:
>
> I think I overlooked it. Good point. I am using Redis to save the path to
> my savepoint, I might be able to set a TTL to avoid such issue.
>
> Hao Sun
>
>
> On Tue, Sep 24, 2019 at 9:54 AM Yuval Itzchakov <yuva...@gmail.com> wrote:
>
> Hi Hao,
>
> I think he's exactly talking about the usecase where the JM/TM restart and
> they come back up from the latest savepoint which might be stale by that
> time.
>
> On Tue, 24 Sep 2019, 19:24 Hao Sun, <ha...@zendesk.com> wrote:
>
> We always make a savepoint before we shutdown the job-cluster. So the
> savepoint is always the latest. When we fix a bug or change the job graph,
> it can resume well.
> We only use checkpoints for unplanned downtime, e.g. K8S killed JM/TM,
> uncaught exception, etc.
>
> Maybe I do not understand your use case well, I do not see a need to start
> from checkpoint after a bug fix.
> From what I know, currently you can use checkpoint as a savepoint as well
>
> Hao Sun
>
>
> On Tue, Sep 24, 2019 at 7:48 AM Yuval Itzchakov <yuva...@gmail.com> wrote:
>
> AFAIK there's currently nothing implemented to solve this problem, but
> working on a possible fix can be implemented on top of
> https://github.com/lyft/flinkk8soperator which already has a pretty fancy
> state machine for rolling upgrades. I'd love to be involved as this is an
> issue I've been thinking about as well.
>
> Yuval
>
> On Tue, Sep 24, 2019 at 5:02 PM Sean Hester <sean.hes...@bettercloud.com>
> wrote:
>
> hi all--we've run into a gap (knowledge? design? tbd?) for our use cases
> when deploying Flink jobs to start from savepoints using the job-cluster
> mode in Kubernetes.
>
> we're running a ~15 different jobs, all in job-cluster mode, using a mix
> of Flink 1.8.1 and 1.9.0, under GKE (Google Kubernetes Engine). these are
> all long-running streaming jobs, all essentially acting as microservices.
> we're using Helm charts to configure all of our deployments.
>
> we have a number of use cases where we want to restart jobs from a
> savepoint to replay recent events, i.e. when we've enhanced the job logic
> or fixed a bug. but after the deployment we want to have the job resume
> it's "long-running" behavior, where any unplanned restarts resume from the
> latest checkpoint.
>
> the issue we run into is that any obvious/standard/idiomatic Kubernetes
> deployment includes the savepoint argument in the configuration. if the Job
> Manager container(s) have an unplanned restart, when they come back up they
> will start from the savepoint instead of resuming from the latest
> checkpoint. everything is working as configured, but that's not exactly
> what we want. we want the savepoint argument to be transient somehow (only
> used during the initial deployment), but Kubernetes doesn't really support
> the concept of transient configuration.
>
> i can see a couple of potential solutions that either involve custom code
> in the jobs or custom logic in the container (i.e. a custom entrypoint
> script that records that the configured savepoint has already been used in
> a file on a persistent volume or GCS, and potentially when/why/by which
> deployment). but these seem like unexpected and hacky solutions. before we
> head down that road i wanted to ask:
>
> - is this is already a solved problem that i've missed?
> - is this issue already on the community's radar?
>
> thanks in advance!
>
> --
> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865
> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305
> <http://www.bettercloud.com/> <http://www.bettercloud.com/>
> *Altitude 2019 in San Francisco | Sept. 23 - 25*
> It’s not just an IT conference, it’s “a complete learning and networking
> experience”
> <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude>
>
>
>
> --
> Best Regards,
> Yuval Itzchakov.
>
>
>
> --
> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865
> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305
> <http://www.bettercloud.com/> <http://www.bettercloud.com/>
> *Altitude 2019 in San Francisco | Sept. 23 - 25*
> It’s not just an IT conference, it’s “a complete learning and networking
> experience”
> <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude>
>
>
>
>
> --
> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865
> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305
> <http://www.bettercloud.com> <http://www.bettercloud.com>
> *Altitude 2019 in San Francisco | Sept. 23 - 25*
> It’s not just an IT conference, it’s “a complete learning and networking
> experience”
> <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude>
>
>
>
> --
> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865
> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305
> <http://www.bettercloud.com> <http://www.bettercloud.com>
> *Introducing the BetterCloud Integration Center *
> Automate actions across every app and own SaaSOps
> <https://www.bettercloud.com/integrations-webinar/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-integration-center>
>
>
