How to emit changed data only w/ Flink trigger?

2019-11-01 Thread Qi Kang
hich changed in 1 second period? Currently we’ve got 1000+ sites, so frequently emitting all aggregation records seems somewhat expensive. BR, Qi Kang

How to use thin JAR instead of fat JAR when submitting Flink job?

2019-09-23 Thread Qi Kang
Hi, According to the documentation of Flink, it seems that fat JAR is recommended when submitting a Flink job. However, the Flink dependencies (as well as other dependencies like Hadoop) are too big in size, thus producing a fat JAR which exceeds 100MB. Is there some way to separate the 'common

Re: How can TMs distribute evenly over Flink on YARN cluster?

2019-08-26 Thread Qi Kang
jobs, so no other services are involved. > On Aug 26, 2019, at 18:21, Yang Wang wrote: > > Hi Qi Kang, > > If you means to spread out all taskmanager evenly across the yarn cluster, it > is a pity that flink could do nothing. > Each per-job flink cluster is an individu

How can TMs distribute evenly over Flink on YARN cluster?

2019-08-26 Thread Qi Kang
Hi, We got 3 Flink jobs running on a 10-node YARN cluster. The jobs were submitted in a per-job flavor, with same parallelism (10) and number of slots per TM (2). We originally assumed that TMs should automatically spread across the cluster, but what came out was just the opposite: All 5 TMs

Re: timeout error while connecting to Kafka

2019-08-22 Thread Qi Kang
The code itself is fine. Turning the app’s log level to DEBUG will give you more information. BTW, please make sure that the addresses of Kafka brokers are properly resolved. > On Aug 22, 2019, at 15:45, Eyal Pe'er wrote: > > Hi, > > I'm trying to consume events using Apache Flink. > > The