Re: Non incremental window function accumulates unbounded state with RocksDb

2019-09-01 Thread Yun Tang
Hi William I think there might be another possible cause. Since RocksDB would perform 10X less than heap state backend. Have you ever checked current watermark of the job (from web UI) to see whether window triggered as expected, and whether the rocksDB job behaves back pressured? If state have

Re: [SURVEY] Is the default restart delay of 0s causing problems?

2019-09-01 Thread Yu Li
-1 on increasing the default delay to none zero, with below reasons: a) I could see some concerns about setting the delay to zero in the very original JIRA (FLINK-2993 ) but later on in FLINK-9158

Re: [ANNOUNCE] Kinesis connector becomes part of Flink releases

2019-09-01 Thread Yu Li
Great to know, thanks for the efforts Bowen! And I believe it worth a release note in the original JIRA, wdyt? Thanks. Best Regards, Yu On Sat, 31 Aug 2019 at 11:01, Bowen Li wrote: > Hi all, > > I'm glad to announce that, as #9494 > was merged today

Re: How to handle Flink Job with 400MB+ Uberjar with 800+ containers ?

2019-09-01 Thread Zhu Zhu
Hi Elkhan, >>Regarding "One optimization that we take is letting yarn to reuse the flink-dist jar which was localized when running previous jobs." >>We are intending to use Flink Real-time pipeline for Replay from Hive/HDFS (from offline source), to have 1 single pipeline for both batch and real-t