Re: JobMaster does not register with ResourceManager in high availability setup

2020-03-22 Thread tison
Hi, It seems the leader info has been published but since you don't turn on DEBUG log on org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService still we can only *guess* the retrieval service in JobMaster doesn't get notified and even I don't see a INFO level log Starting ZooK

backpressure and memory

2020-03-22 Thread seeksst
Hi, everyone: I’m a flink sql user, and the version is 1.8.2. Recently I confuse about memory and backpressure. I have two job on yarn, due to memory over, it’s frequently killed by yarn. One job,I have 3 taskmanagers and 6 parallelism, each one has 8G memory.It read from kafka, one minute

[ANNOUNCE] Weekly Community Update 2020/12

2020-03-22 Thread Konstantin Knauf
Dear community, happy to share this week's community digest featuring "Flink Forward Virtual Conference 2020", a small update on Flink 1.10.1, a better Filesystem connector for the Table API & SQL, new source/sink interfaces for the Table API and a bit more. Flink Development == * [r

Re: Windows on SinkFunctions

2020-03-22 Thread tison
Hi Sidney, For the case, you can exactly write stream. ... .window() .apply() .addSink() Operator chain will chain these operators into one so that you don't have to worry about the efficiency. Best, tison. Sidney Feiner 于2020年3月22日周日 下午10:03写道: > Hey, > I wanted to know if it's pos

Re: [DISCUSS] FLIP-111: Docker image unification

2020-03-22 Thread Andrey Zagrebin
Thanks for summarising the discussion points, Till. # Configuration ## Env variables Agree, this looks like an independent effort. ## dynamic program arguments Indeed, jobmanager.sh needs small extension. It can be addressed independently but I think it has chance to be addressed in this release

Windows on SinkFunctions

2020-03-22 Thread Sidney Feiner
Hey, I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well. For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink. Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in

Re: Issue with single job yarn flink cluster HA

2020-03-22 Thread Dinesh J
Attaching the job manager log for reference. 2020-03-22 11:39:02,693 WARN org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever - Error while retrieving the leader gateway. Retrying to connect to akka.tcp://flink@host1:28681/user/dispatcher. 2020-03-22 11:39:02,724 WARN akka.re

Issue with single job yarn flink cluster HA

2020-03-22 Thread Dinesh J
Hi all, We have single job yarn flink cluster setup with High Availability. Sometimes job manager failure successfully restarts next attempt from current checkpoint. But occasionally we are getting below error. {"errors":["Service temporarily unavailable due to an ongoing leader election. Please r