Question about Reactive mode support

Hello, We were going through FlIP-159 and FLIP-160 and found this feature of interest to us for auto-scaling purposes. The limitations

Re: Question about Reactive mode support

it's hard to say what will work best for us. We just want to see if reactive mode will be available in the future so that we can leverage it when we have more data. Thanks, Sonam ________ From: Robert Metzger Sent: Thursday, March 11, 2021 5:28 AM To: Sonam Mandal

Failure detection in Flink

Hello, I'm looking for some resources around failure detection in Flink between the various components such as Task Manager, Job Manager, Resource Manager, etc. For example, how does the Job Manager detect that a Task Manager is down (long GC pause or it just crashed)? There is some indication

Re: Failure detection in Flink

reference on how Flink reacts to such failure scenarios? If not, can you provide some details on how this works? Thanks, Sonam Get Outlook for iOS<https://aka.ms/o0ukef> From: Till Rohrmann Sent: Tuesday, March 30, 2021 5:02:43 AM To: Sonam Mandal Cc

Re: Failure detection in Flink

nks, Sonam From: Till Rohrmann Sent: Tuesday, March 30, 2021 8:33 AM To: Sonam Mandal Cc: user@flink.apache.org Subject: Re: Failure detection in Flink Well, the FLIP-6 documentation is probably the best resource albeit being a bit outdated. The components react a bit differently: JobMaster loses

Question about setting up Task-local recovery with a RocksDB state backend

Hello, I've been going through the documentation for task-local recovery and came across this section which discusses that with incremental check

How to know if task-local recovery kicked in for some nodes?

Hello, We are experimenting with task local recovery and I wanted to know whether there is a way to validate that some tasks of the job recovered from the local state rather than the remote state. We've currently set this up to have 2 Task Managers with 2 slots each, and we run a job with para

Re: How to know if task-local recovery kicked in for some nodes?

minimum state size we should experiment with to check recovery time differences between the two modes? Thanks, Sonam From: dhanesh arole Sent: Wednesday, April 7, 2021 3:43:11 AM To: Till Rohrmann Cc: Sonam Mandal ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject

Re: How to know if task-local recovery kicked in for some nodes?

Hi Till, Got it, that definitely makes sense, was just looking for some ballpark number to start with. Appreciate your help! Thanks, Sonam From: Till Rohrmann Sent: Monday, April 12, 2021 1:00 AM To: Sonam Mandal Cc: dhanesh arole ; Tzu-Li (Gordon) Tai ; user

Task Local Recovery with mountable disks in the cloud

Hello, We've been experimenting with Task-local recovery using Kubernetes. We have a way to specify mounting the same disk across Task Manager restarts/deletions for when the pods get recreated. In this scenario, we noticed that task local recovery does not kick in (as expected based on the doc

Re: Task Local Recovery with mountable disks in the cloud

0:24 AM To: dev Cc: user@flink.apache.org ; Sonam Mandal Subject: Re: Task Local Recovery with mountable disks in the cloud Hi Sonam, sorry for the late reply. We were a bit caught in the midst of the feature freeze for the next major Flink release. In general, I think it is a very good idea

Re: Task Local Recovery with mountable disks in the cloud

Hi Till, Sure, that sounds good. I'll open a FLIP for this when we start working on it. Thanks for the insights! Regards, Sonam From: Till Rohrmann Sent: Monday, May 10, 2021 2:26 AM To: Sonam Mandal Cc: dev ; user@flink.apache.org Subject: Re: Task

Regarding FLIP-91's status

Hello, I was curious about the progress on FLIP-91. Is this actively being developed? I believe the code is in development at https://github.com/ververica/flink-sql-gateway, is this the right REPO? I haven

Re: Regarding FLIP-91's status

Friday, May 28, 2021 5:18 AM To: Sonam Mandal Cc: user@flink.apache.org ; Jark Wu Subject: Re: Regarding FLIP-91's status Hi Sonam, It looks like it has been stale for some time. You might be able to restart the discussion replying to the respective thread in the dev mailing list [1]. You

Recommendation for dealing with Job Graph incompatibility across varying Flink versions

Hello, We are exploring running multiple Flink clusters within a Kubernetes cluster such that each Flink cluster can run with a specified Flink image version. Since the Flink Job Graph needs to be compatible with the Flink version running in the Flink cluster, this brings a challenge in how we

Re: Recommendation for dealing with Job Graph incompatibility across varying Flink versions

From: Paul K Moore Sent: Friday, June 18, 2021 2:25:52 AM To: Sonam Mandal Cc: user@flink.apache.org ; Srinivasulu Punuru Subject: Re: Recommendation for dealing with Job Graph incompatibility across varying Flink versions Hi Sonam, I am not a long-standing Flink user (3 months only) so

Recommended way to submit a SQL job via code without getting tied to a Flink version?

Hello, We've written a simple tool which takes SQL statements as input and uses a StreamTableEnvironment​ to eventually submit this to the Flink cluster. We've noticed that the Flink library versions we depend on must match the Flink version running in our Kubernetes cluster for the job submiss

Re: Recommended way to submit a SQL job via code without getting tied to a Flink version?

way to handle this? Is this safe and does it work in all scenarios? Thanks, Sonam From: Matthias Pohl Sent: Tuesday, June 29, 2021 02:29 To: Sonam Mandal Cc: user@flink.apache.org; Jark Wu; Timo Walther Subject: Re: Recommended way to submit a SQL job via code

Re: Recommended way to submit a SQL job via code without getting tied to a Flink version?

Hi Stephan, Thanks for the detailed explanation! This really helps understand all this better. Appreciate your help! Regards, Sonam From: Stephan Ewen Sent: Wednesday, June 30, 2021 3:56:22 AM To: Sonam Mandal ; user@flink.apache.org Cc: matth