Job cluster and HA

Hi, I was experimenting with HA lately and see that it recovers successfully job, in the case of jobmanager restarts. Now my question is whether it will work for the job cluster. Based on the instructions https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md

Deploying and managing multiple jobs on EMR

Hi, I'm planing to deploy multiple jobs on a EMR Flink cluster and I'd like to understand what the community is using to manage job deployments, updates and failures? Is there any strong Open Source project or each organization is building their custom manager?

Re: How many task managers to launch for a job?

Hi Song, You said "In that way, the total slots (or number of TaskManagers if you config on slot for each TaskManager)", do you imply that one taskmanager contains one slot? Do you have some experience on how many slots to spawn for one TaskManager? I read the Flip-6, it says "For the sake of simpl

Re: Generic return type on a user-defined scalar function

Hi JingsongLee and Timo, Thanks for taking a look and for the feedback! All the best, Morrisa Morrisa Brenner Software Engineer ━━ 225 Franklin St, Boston, MA 02110 klaviyo.com > On May 21, 2019, at 12:10 AM, JingsongLee wrote: > > Hi Morrisa: > > It seems that

Re: Job crashing cluster

Hi, Could you share error stack trace? Thanks & Regards, Sushant Sawant On Fri, 24 May 2019, 19:18 Timothy Victor, wrote: > If a flink job crashes during startup (throws exception) the entire > cluster goes down. This is even on a simple bare metal host. > > > I have tried catching the excep

Job crashing cluster

If a flink job crashes during startup (throws exception) the entire cluster goes down. This is even on a simple bare metal host. I have tried catching the exception, but even that didnt prevent the JM and cluster from crashing. Has anyone run into this problem? I'm on Flink 1.7.1 Thanks Tim

[DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

Hi All, As mentioned in our speak[1] given in FlinkForwardChina2018, we have improved HeapKeyedStateBackend to support disk spilling and put it in production here in Alibaba for last year's Singles' Day. Now we're ready to upstream our work and the design doc is up for review[2]. Please let us kno

Re: Re: How can i just implement a crontab function using flink?

Thanks. got it wangl...@geekplus.com.cn From: Puneet Kinra Date: 2019-05-24 17:02 To: wangl...@geekplus.com.cn CC: user Subject: Re: How can i just implement a crontab function using flink? There is concept of periodic watermarker , you can use that if you are working on eventtime. On Fri,

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

This issue is another case where we have problems figuring out the boundaries and responsibilities between the ExecutionEnvironments and the ClusterClient. I believe we should figure this out first, and decide whether the ClusterClient (or anything based on it) should be made public to accomo

Re: How many task managers to launch for a job?

Yes true. I am trying to figure out how the TaskManagers are distributed across physical machines by Mesos and YARN. Maybe I shoud start a new thread for help. Thank you Song Best, Pengcheng On Fri, May 24, 2019 at 10:52 AM Xintong Song wrote: > As far as I know, Flink does not have any requirem

What policies does YARN/Mesos use to distribute TaskManagers across physical machines?

Hi all, I had a question when I was reading Flip-6. When submitting a job, YARN/Mesos ResourceManager needs to spawn, let's say, 5 TaskManagers for the job. Suppose we have 4 physical machines, how would ResourceManager place the 5 TaskManagers across the 4 machines? Does it just follow a Round-Rob

Re: How to generate a sequential watermark which increases by one unit each time

Hi Averell, IMHO, a simple approach would be adding a rich map that holds the sequence value (backed by states) and attach it to the records before the assigner operator. Best, Paul Lam > 在 2019年5月21日，20:37，Averell 写道： > > Hi everyone, > > I have a stream of files, each file has multiple re

Re: How many task managers to launch for a job?

As far as I know, Flink does not have any requirements on how the TaskManagers are distributed across physical machines. So I think it really depends on the scheduling policy of the Mesos cluster. I'm not an expert on Mesos, so correct me if I was wrong. Thank you~ Xintong Song On Fri, May 24,

Re: Re: How can i just implement a crontab function using flink?

Thanks, it's a alternative solution. wangl...@geekplus.com.cn From: Jörn Franke Date: 2019-05-24 16:31 To: wangl...@geekplus.com.cn CC: user Subject: Re: How can i just implement a crontab function using flink? Just sent a dummy event from the source system every minute Am 24.05.2019 um 10:2

Re: How can i just implement a crontab function using flink?

There is concept of periodic watermarker , you can use that if you are working on eventtime. On Fri, May 24, 2019 at 1:51 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > > I want to do something every one minute. > > Using TumblingWindow, the function will not be triigged if ther

Re: How can i just implement a crontab function using flink?

Just sent a dummy event from the source system every minute > Am 24.05.2019 um 10:20 schrieb "wangl...@geekplus.com.cn" > : > > > I want to do something every one minute. > > Using TumblingWindow, the function will not be triigged if there's no message > received during this minute. But i st

How can i just implement a crontab function using flink?

I want to do something every one minute. Using TumblingWindow, the function will not be triigged if there's no message received during this minute. But i still need to execute the funtion. How can i implement it ? wangl...@geekplus.com.cn

Re: How many task managers to launch for a job?

Hi Song, Thank you for the clarification. Now I know TaskManagers are automatically allocated. Yet, I am still not very clear how the TMs are allocated. I'm guessing the allocation process would be: On the job side, I have a job with each operator parallelism=5. Since one TaskManager has one slot.

Re: Upgrading from 1.4 to 1.8, losing Kafka consumer state

Hi Nik, Could you check outt the taskmanagers’ logs? When restored from a savepoint/checkpoint, FlinkKafkaConsumer would log the starting offset of Kafka partitions. WRT `auto.offset.rest` in Kafka configuration, it’s of a relatively low priority, and would only be used when there’s no restore