static table in flink

Hello, I have questions about static table in flink. Join the stream table with static table. I'm looking at temporal table, while the time based table would grow exponentially over period. Any suggestions? Stream tables checks the contains in static table(updates once everyday with new set of d

Re: Flink vs Kafka streams

Thanks Congxian. Yes, its been very hard to manage the cluster and thats why we are trying to evaluate alternate choices. If anyone has found better methods to deploy and scale, it would be great to know so that we can adopt the same as well. Thanks On Fri, Nov 8, 2019 at 1:56 AM Congxian Qiu wr

Re: flink on yarn-cluster kerberos authentication for hbase

Hi Can you share more information regarding how you currently setup your Kerberos that only works with Zookeeper? Does your HBase share the same KDC? -- Rong On Fri, Nov 8, 2019 at 12:33 AM venn wrote: > Thanks, I already seen, not work for me > > > > *发件人**:* Jaqie Chan > *发送时间:* Friday, No

Re: Unable to retrieve Kafka consumer group offsets

Thanks Till, The link the the thread is here: http://mail-archives.apache.org/mod_mbox/flink-user/201702.mbox/%3ccagr9p8cekzvx3fxqi363o3tbd37nuppe7l_+c7+kx_z_nqy...@mail.gmail.com%3E On Fri, Nov 8, 2019 at 5:18 AM Till Rohrmann wrote: > Hi Harrison, > > which mailing list thread are you referrin

Re: Flink Filters have state?

Thanks Till! This was helpful! Tim On Fri, Nov 8, 2019, 7:16 AM Till Rohrmann wrote: > Hi Tim, > > 1. The filter is stored within the JobGraph which is persisted to a > persistent storage if HA is enabled. Usually, this is either HDFS, S3 or > any other highly available file system. It is jus

Re: Flink SQL dynamic configuration

Thanks for your helps Till. I appreciate it. On Fri, Nov 8, 2019 at 9:02 PM Till Rohrmann wrote: > Hi Jaqie, > > not sure whether this is easily possible with Flink's SQL API but if you > used the DataStream API directly you could create a connected stream where > you have two inputs. One input

Re: Unable to retrieve Kafka consumer group offsets

Hi Harrison, which mailing list thread are you referring to exactly? I've pulled in Becket who might be able to tell you more about the Kafka connector. Cheers, Till On Thu, Nov 7, 2019 at 11:11 PM Harrison Xu wrote: > I am using Flink 1.9.0 and KafkaConsumer010 (Kafka 0.10.1.1). Attempting >

Re: Flink Filters have state?

Hi Tim, 1. The filter is stored within the JobGraph which is persisted to a persistent storage if HA is enabled. Usually, this is either HDFS, S3 or any other highly available file system. It is just a serialized POJO. If you want to store your filter's state you would need to use Flink's state AP

Re: Flink SQL dynamic configuration

Hi Jaqie, not sure whether this is easily possible with Flink's SQL API but if you used the DataStream API directly you could create a connected stream where you have two inputs. One input could be the normal message stream and the other input could be the configuration stream. So whenever there i

Re: Does Flink merge streams deterministically?

Hi Amran, in the general case, the merging of two streams is not deterministic. Cheers, Till On Thu, Nov 7, 2019 at 3:24 AM amran dean wrote: > > The Flink Kafka consumer's watermark extractor claims:. > > "The watermark extractor will run per Kafka partition, watermarks will be > merged acros

Re: Till Rohrmann - Can you please share your code for FF - SF - Flink as a lib

Thanks a lot Till !! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Limit max cpu usage per TaskManager

Hi Lu Niu, Yes, you could use `yarn.containers.vcores` to set the vcores of taskmanager. However, it could not guarantee that the application do not affect each other. By default, the yarn cluster are using cgroup share. That means a taskmanager could use more cpu than it allocated. When the machi

Re: Flink savepoint(checkpoint) recovery dev debug

Hi You can ref CheckpointMetadataLoadingTest#testLoadAndValidateSavepoint for more information. Even more, you can ref the line 136-line145 [1], and init a CompletedCheckpointStorageLocation to make it work. [1] https://github.com/apache/flink/blob/d788c6ff024fbe8a699e987d3df6a00160d22a06/flink-r

Re: Flink vs Kafka streams

Hi >From your description, seems the big problem is scale in and out, and there maybe a big downtime for trigger savepoint and restore from the savepoint. Previously, we have proposed a feature named stop-with-checkpoint[1] same as the stop-with-savepoint, but triggering a checkpoint instead of s

Re: How can I get the backpressure signals inside my function or operator?

humm, that is also another possibility. Thanks for your suggestion! *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Nov 7, 2019 at 10:41 PM Yuval Itzchakov wrote: > Hi, > > We've been dealin

回复: flink on yarn-cluster kerberos authentication for hbase

Thanks, I already seen, not work for me 发件人: Jaqie Chan 发送时间: Friday, November 8, 2019 4:14 PM 收件人: venn 抄送: user@flink.apache.org 主题: Re: flink on yarn-cluster kerberos authentication for hbase Hello, Does this have helps to you? https://stackoverflow.com/questions/34596165/how-to

Re: flink on yarn-cluster kerberos authentication for hbase

Hello, Does this have helps to you? https://stackoverflow.com/questions/34596165/how-to-do-kerberos-authentication-on-a-flink-standalone-installation Regards 嘉琪 On Fri, Nov 8, 2019 at 4:00 PM venn wrote: > HI Guys: > > Who can share some example for flink on yarn-cluster kerberos

flink on yarn-cluster kerberos authentication for hbase

HI Guys: Who can share some example for flink on yarn-cluster kerberos authentication for hbase I auth as what I do in java program, it look like just authentication zookeeper, cannot access for hbase thanks