Re: Errors in QueryableState sample code?

Hi Ken, About the first question, the way of fixing is right. About the second question, you are right, the "response.get()" missed 'ValueState<>'. Can you create a JIRA issue or fix it if you want. Thanks, vino. Ken Krugler 于2018年9月20日周四 上午6:27写道： > Hi all, > > I was looking at the Example

Re: Which mode to choose flink on Yarn.

That mainly depends on how much parallelism of your job. The main bottleneck of job manager usually because it is busy to handle rpc requests and gc. At most time you can set larger jm memory to address it by pass `-jm 4096` to `yarn-session.sh start`. Best, tison. weilongxing 于2018年9月20日周四 下午

Re: Which mode to choose flink on Yarn.

> In addition, per job mode calculate resource that the job required, while > session mode require you config a static config of that persistent session. I tested using flink 1.5.2 and found that session mode can also support dynamic resource. You don’t have to config static config. I am wond

Re: Which mode to choose flink on Yarn.

Thanks. I am wondering whether the job manager will be the bottleneck and how many jobs could a job manager support in session mode. I tried to find the bottleneck in test environment but failed. > 在 2018年9月20日，下午2:16，vino yang 写道： > > Hi weilong, > > As you said, there are advantages and

Re: Which mode to choose flink on Yarn.

Hi weilong, As vino said, the main advantage of per job mode is that it provides job-level isolation, and that of session mode is that it set up a persistent session and accept job, which means the overhead of resource request/setup would loose. In addition, per job mode calculate resource that th

Re: Which mode to choose flink on Yarn.

Hi weilong, As you said, there are advantages and disadvantages to each of the two approaches. However, I hope you know that the "single job" mode has a huge advantage over the "YARN flink session" mode in that it provides job-level isolation (whether JM or TM). This will allow the Job to be more

Re: WriteTimeoutException in Cassandra sink kill the job

https://issues.apache.org/jira/browse/FLINK-10310?focusedCommentId=16618447&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16618447 On Tue, Sep 11, 2018 at 10:15 AM Jayant Ameta wrote: > Hi Till, > I've opened a JIRA issue: > https://issues.apache.org/jira/browse/

Which mode to choose flink on Yarn.

There are two methods to deploy flink applications on yarn. The first one is use yarn-session and all flink applications are deployed in the session. The second method is each flink application deploy on yarn as a yarn application. My question is what's the difference between these two methods?

Re: Accessing Global State when processing KeyedStreams

Hi Rong, Thanks for your suggestion, I'll give that a go. I just found a great article on this as well that explains the functionality https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink Regards, Scott -- Sent from: http://apache-flink-user-mailing-list-archi

Re: Utilising EMR's master node

Hi Gary, Thanks for your help. Regarding TM configurations, in term of performance, when my 2 servers have 16 vcores each, should I have 2 TMs with 16GB mem, 16 task slots each, or 8 TMs with 4GB mem and 4 task slots each? Thanks and regards, Averell -- Sent from: http://apache-flink-user-mail

Errors in QueryableState sample code?

Hi all, I was looking at the Example section of Querying State : QueryableStateCli

Re: Trying to figure out why a slot takes a long time to checkpoint

Hey guys, So, switching to Ceph/S3 didn't shine any new lights on the issue. Although the times are a bit higher, just a few slots are taking a magnitude longer to save. So I changed the logs for DEBUG. The problem is: I'm not seeing anything that seems relevant; only pings from ZooKeeper, heartb

Re: Accessing Global State when processing KeyedStreams

Hi Scott, Your use case seems to be a perfect fit for the Broadcast state pattern [1]. -- Rong [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/broadcast_state.html On Wed, Sep 19, 2018 at 7:11 AM Scott Sue wrote: > Hi, > > In our application, we receive Orde

Re: Question about Window Tigger

Hi Chang, There were some previous discussion regarding how to debug watermark and window triggers[1]. Basically if there's no data for some partitions there's no way to advance watermark. As it would not be able to determine whether this is due to network failure or actually there's no data arriv

Accessing Global State when processing KeyedStreams

Hi, In our application, we receive Orders and Prices via two KafkaSources. What I want to do is to perform calculations for a given Order against a stream of Prices for the same securityId, i.e. same identifier between the Order and stream of Prices. Naturally this is a perfect fit for a KeyedSt

Flink 1.5.2 process keeps reference to deleted blob files.

Hello, we are using YarnClusterClient for job submission. After successful/failed job execution it looks like blob file for that job is deleted, but there is still some handle from Flink process to that file. As a result the file is not removed from machine and we faced no space felt on device err

Re: How to use checkpoint in flink1.5.3

The stop order is： /home/flink-1.5.3/bin/flink cancel ff31023d23e568290d5596a34aa3cad6 -yid application_1526888270443_0336 -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

How to use checkpoint in flink1.5.3

I used flink1.3.2 before, and recently upgraded to flink1.5.3. I found that the new version has adjusted checkpoint, and there was a problem after modifying the code My code like : * RocksDBStateBackend rocksDBStateBackend = new RocksDBStateBackend("hdfs://hdfs_1/demo/demo-fs-checkpoin

Re: ***UNCHECKED*** Re: Standalone cluster instability

Hi, JobManager is not responsible and have no means to restart TaskManager in case of TaskManager process being killed (it would need to have ssh into the machine and restart it…). I don’t know, but from your description of the problem I presume that Flink’s bash startup scripts do not contain

Re: Migration to Flink 1.6.0, issues with StreamExecutionEnvironment.registerCachedFile

Hi Subramanya, I could reproduce this behavior running a job in YARN cluster. This works in standalone cluster just fine. We've changed a little bit how the cache entries are distributed in 1.6.0. I am investigating this problem right now. Would you like to create a JIRA bug for it? Best, Dawid

HA failing for 1.6.0 job cluster with docker-compose

Dear all, I am currently experimenting with a Flink 1.6.0 job cluster. The goal is to run a streaming job on K8s. Right now I am using docker-compose to experiment with the job cluster. I am trying to set-up HA with Zookeeper, but I seem to fail. I have a docker-compose file which contains the fo

Question about Window Tigger

Dear All, I have a question about the Window Trigger: let’s say i would like like use the SlidingEventTimeWindow (60 seconds window size + 1 second window shift) to count the number of records per window. And I am using Event Time with periodic watermarking with certain maxOurOfOrderness time.

Re: questions about YARN deployment and HDFS integration

Thanks for your answers :) Best regards/祝好, Chang Liu 刘畅 > On 17 Sep 2018, at 17:25, Kostas Kloudas wrote: > > Hi Chiang, > > Some of the answers you can find in line: > >> On Sep 17, 2018, at 3:47 PM, Chang Liu > > wrote: >> >> Dear All, >> >> I am helping my

***UNCHECKED*** Error while confirming Checkpoint

Hello, I have a running Flink job that reads data form one Kafka topic, applies some transformations and writes data back into another Kafka topic. The job sometimes restarts due to the following error: /java.lang.RuntimeException: Error while confirming checkpoint at org.apache.flink.run

S3 connector Hadoop class mismatch

Hi, I’m using StreamingFileSink of 1.6.0 to write logs to S3 and encounter a classloader problem. It seems that there are conflicts in flink-shaded-hadoop2-uber-1.6.0.jar and flink-s3-fs-hadoop-1.6.0.jar, and maybe related to class loading orders. Did anyone meet this problem? Thanks a lot!

Re: ***UNCHECKED*** Re: Checkpointing not working

Can you please check the following document and verify whether you have enough network bandwidth to support 30 seconds check point interval worth of the streaming data? https://data-artisans.com/blog/how-to-size-your-apache-flink-cluster-general-guidelines Regards Bhaskar On Wed, Sep 19, 2018 at